Abstract: A system for training a robotic device to perform a task may include a glove to be worn by a user; a sensor array coupled to the glove, a camera to generate images of an object; and a processor configured to execute instructions stored in memory. The processor can receive contact measurements from the sensor array, including tactile signals determined from tactile sensors and positions determined from motion sensors, and an object pose of the object based on an image from the camera, a fusion of images, motion sensing, and/or the contact measurements. The processor can train a machine learning model, based on training data including the contact measurements and the object pose, to generate a prediction of future contact measurements and a future object pose. Other aspects are also described and claimed.

Abstract: A method for controlling a robotic device to perform a task may include determining a force measurement corresponding to a section of sensors coupled with a digit of a plurality of digits of a robotic device. The force measurement may include a magnitude of a force and a position of a centroid of the force in the section that are determined based on contact with an object corresponding to a time. The method may further include determining a digit position of the digit corresponding to the time, and moving the digit, based on a prediction, to stabilize the object to perform a task. A machine learning model can generate the prediction based on the force measurement and the digit position. Other aspects are also described and claimed.

Abstract: Embodiments of a learning-based excavation planning method are disclosed for excavating rigid objects in clutter, which is challenging due to high variance of geometric and physical properties of objects, and large resistive force during the excavation. A convolutional neural network is utilized to predict a probability of excavation success. Embodiments of a sampling-based optimization method are disclosed for planning high-quality excavation trajectories by leveraging the learned prediction model. To reduce simulation-to-real gap for excavation learning, voxel-based representations of an excavation scene are used. Excavation experiments were performed in both simulation and real world to evaluate the learning-based excavation planners. Experimental results show that embodiments of the disclosed method may plan high-quality excavations for rigid objects in clutter and outperform baseline methods by large margins.

Abstract: Embodiments of a learning-based excavation planning method are disclosed for excavating rigid objects in clutter, which is challenging due to high variance of geometric and physical properties of objects, and large resistive force during the excavation. A convolutional neural network is utilized to predict a probability of excavation success. Embodiments of a sampling-based optimization method are disclosed for planning high-quality excavation trajectories by leveraging the learned prediction model. To reduce simulation-to-real gap for excavation learning, voxel-based representations of an excavation scene are used. Excavation experiments were performed in both simulation and real world to evaluate the learning-based excavation planners. Experimental results show that embodiments of the disclosed method may plan high-quality excavations for rigid objects in clutter and outperform baseline methods by large margins.