Abstract: A pose controller is provided for controlling a pose of an object to assemble with a mating object by a gripper of a robot arm. The pose controller includes an interface configured to receive tactile signals from the tactile sensors and transmit a control signal to the actuators, a processor, and a memory, in association with the processor, configured to store a precomputed set of tactile depth images and instructions of computer-implemented method. The instructions cause the processor to perform steps of computing measured tactile depth images from the received tactile signals, refining the pose of the object by matching between the precomputed set of tactile depth images and the measured tactile depth images by, generating a gripper trajectory command based on the refined pose of the object and a target pose of the object, and controlling the actuators of the robot arm according to the gripper trajectory.

Abstract: A method, a system and a computer program product are provided for applying a neural network including an action sequence decoder for generating an action sequence for a robot to perform a task. The neural network is applied to generate the action sequence based on recordings demonstrating humans performing tasks. In an example, the method comprises collecting a recording and a sequence of captions describing scenes in the recording; extracting feature data from the recording; encoding the extracted feature data to produce a sequence of encoded features; and applying the action sequence decoder to produce a sequence of actions for the robot based on the sequence of encoded features having a semantic meaning corresponding to a semantic meaning of the sequence of captions. The feature data includes features of a video signal, an audio signal, and/or text transcription capturing a performance of the task.

Abstract: A pose controller is provided for controlling a pose of an object to assemble with a mating object by a gripper of a robot arm. The pose controller includes an interface configured to receive tactile signals from the tactile sensors and transmit a control signal to the actuators, a processor, and a memory, in association with the processor, configured to store a precomputed set of tactile depth images and instructions of computer-implemented method.

Abstract: A robotic controller for controlling a robot according to a sequence of robotic actions. comprises an input interface configured to receive a plurality of multimodal inputs each specifying instructions for performing a task in a different modality including audio, video, and a text modality. The controller also comprises a multimodal large language model, an action sequence decoder, and a controller. The multimodal LLM includes a multimodal LLM encoder and an LLM decoder. The multimodal LLM encoder is trained with machine learning to transform the multimodal instructions into encodings and the LLM decoder is configured to decode the encodings into a sequence of robotic instructions. The action sequence decoder is trained with machine learning to transform the sequence of robotic instructions into a sequence of actions using a library of robotic skills. The controller is configured to control a robot according to the sequence of actions.

Abstract: A robotic controller for controlling a robot according to a sequence of robotic actions. comprises an input interface to receive multimodal inputs specifying instructions for performing a task in audio, video, and a text modality. The controller transforms the multimodal instructions into encodings using a large language model (LLM) encoder and decodes the encodings into a first sequence of robotic instructions and a robot action description of the actions using an LLM decoder. Human feedback input is received corresponding to at least one action in the first sequence of actions and the controller encodes the feedback input with the robot action description. The controller feeds the encoded data along with multimodal features generated from the encodings into the LLM decoder to generate a corrected sequence of actions. The controller is configured to control a robot according to the corrected sequence of actions.

Abstract: A method for controlling a robotic manipulator according to a task comprises accepting a feedback signal including a sequence of multi-modal observations of a state of execution of the task. The multi-modal observations are processed with a neural network having a self-attention module with a hierarchically conditioned output to produce a skill of the robotic manipulator and an action conditioned on the skill. The neural network is trained in a supervised manner with demonstration data to produce a sequence of skills and a corresponding sequence of actions for the actuators of the robotic manipulator to perform the task. The method further comprises determining one or more control commands for the one or more actuators based on the produced action and submitting the one or more control commands to the one or more actuators causing a change of the state of execution of the task.

Abstract: A controller for controlling an operation of a robot to execute a task is provided. The controller comprises a memory configured to store a set of dynamic movement primitives (DMPs) associated with the task. The set of DMPs comprise a set of at least two dynamical systems: a function representing point attractor dynamics and a forcing function corresponding to a learned demonstration of the task. The controller comprises a processor configured to transform the set of DMPs to a set of constrained DMPs (CDMPs) by determining a perturbation function associated with the forcing function. The perturbation function is associated with a set of operational constraints. The processor is further configured to solve, a non-linear optimization problem for the set of CDMPs based on the set of operational constraints and generate, a control input for controlling the robot for executing the task, based on the solution.

Abstract: A method, a system and a computer program product are provided for applying a neural network including an action sequence decoder for generating an action sequence for a robot to perform a task. The neural network is applied to generate the action sequence based on recordings demonstrating humans performing tasks. In an example, the method comprises collecting a recording and a sequence of captions describing scenes in the recording; extracting feature data from the recording; encoding the extracted feature data to produce a sequence of encoded features; and applying the action sequence decoder to produce a sequence of actions for the robot based on the sequence of encoded features having a semantic meaning corresponding to a semantic meaning of the sequence of captions. The feature data includes features of a video signal, an audio signal, and/or text transcription capturing a performance of the task.

Abstract: A system for parameter tuning for robotic manipulators is provided. The system includes an interface configured to receive a task specification, a plurality of physical parameters, and a plurality of control parameters, wherein the interface is configured to communicate with a real-world robot via a robot controller. The system further includes a memory to store computer-executable programs including a robot simulation module, a robot controller, and an auto-tuning module a processor, in connection with the memory. In this case, the processor is configured to acquire, in communication with the real-world robot, state values of the real-world robot, state values of the robot simulation module, simultaneously update, by use of a predetermined optimization algorithm with the auto-tuning module, an estimate of one or more of the physical, and said control parameters, and store the updated parameters.

Abstract: A controller for controlling an operation of a robot to execute a task is provided. The controller comprises a memory configured to store a set of dynamic movement primitives (DMPs) associated with the task. The set of DMPs comprise a set of at least two dynamical systems: a function representing point attractor dynamics and a forcing function corresponding to a learned demonstration of the task. The controller comprises a processor configured to transform the set of DMPs to a set of constrained DMPs (CDMPs) by determining a perturbation function associated with the forcing function. The perturbation function is associated with a set of operational constraints. The processor is further configured to solve, a non-linear optimization problem for the set of CDMPs based on the set of operational constraints and generate, a control input for controlling the robot for executing the task, based on the solution.

Abstract: A controller is provided for interactive classification and recognition of an object in a scene using tactile feedback. The controller includes an interface configured to transmit and receive the control, sensor signals from a robot arm, gripper signals from a gripper attached to the robot arm, tactile signals from sensors attached to the gripper and at least one vision sensor, a memory module to store robot control programs, and a classifier and recognition model, and a processor to generate control signals based on the control program and a grasp pose on the object, configured to control the robot arm to grasp the object with the gripper.

Abstract: A system for parameter tuning for robotic manipulators is provided. The system includes an interface configured to receive a task specification, a plurality of physical parameters, and a plurality of control parameters, wherein the interface is configured to communicate with a real-world robot via a robot controller. The system further includes a memory to store computer-executable programs including a robot simulation module, a robot controller, and an auto-tuning module a processor, in connection with the memory. In this case, the processor is configured to acquire, in communication with the real-world robot, state values of the real-world robot, state values of the robot simulation module, simultaneously update, by use of a predetermined optimization algorithm with the auto-tuning module, an estimate of one or more of the physical, and said control parameters, and store the updated parameters.

Abstract: A controller is provided for interactive classification and recognition of an object in a scene using tactile feedback. The controller includes an interface configured to transmit and receive the control, sensor signals from a robot arm, gripper signals from a gripper attached to the robot arm, tactile signals from sensors attached to the gripper and at least one vision sensor, a memory module to store robot control programs, and a classifier and recognition model, and a processor to generate control signals based on the control program and a grasp pose on the object, configured to control the robot arm to grasp the object with the gripper.

Abstract: An object detection system for detecting and manipulating objects on a workspace includes a three dimensional (3D) sensor configured to acquire and transmit point clouds of a scene, each point cloud including one or more objects in the workspace, manipulator configured to move or grip each of the one or more objects, a memory to store the images and a computer executable program including an object detection-localization program, a segmentation program, a gripping-moving program and a geometry reconstruction program, a processor to perform the computer executable program using the images in connection with the 3D sensor, the memory and the manipulator.

Abstract: A system for active-training a neural network includes an input interface to receive a set of images, a memory to store an active sampler, a feature generator and a subset of the images selected from the set of the images, the subset of the images having similarity values based on a predetermined feature domain information, at least one processor to train the feature generator by use of the active sampler. The active sampler is configured to perform first-selecting, from the subset of the images, a pair of images as matching pair images according to a threshold of similarity values, second-selecting a pair of images from another subset of the images, feeding the matching pair images and the unmatched pair images to the feature generator, updating weighting parameters and increasing the threshold according to preset values an output interface to render the weighting parameters of the feature generator.

Abstract: An object detection system for detecting and manipulating objects on a workspace includes a three dimensional (3D) sensor configured to acquire and transmit point clouds of a scene, each point cloud including one or more objects in the workspace, manipulator configured to move or grip each of the one or more objects, a memory to store the images and a computer executable program including an object detection-localization program, a segmentation program, a gripping-moving program and a geometry reconstruction program, a processor to perform the computer executable program using the images in connection with the 3D sensor, the memory and the manipulator.

Abstract: A system for active-training a neural network includes an input interface to receive a set of images, a memory to store an active sampler, a feature generator and a subset of the images selected from the set of the images, the subset of the images having similarity values based on a predetermined feature domain information, at least one processor to train the feature generator by use of the active sampler. The active sampler is configured to perform first-selecting, from the subset of the images, a pair of images as matching pair images according to a threshold of similarity values, second-selecting a pair of images from another subset of the images, feeding the matching pair images and the unmatched pair images to the feature generator, updating weighting parameters and increasing the threshold according to preset values an output interface to render the weighting parameters of the feature generator.

Abstract: A method of depositing a material with an extrusion tip of an additive manufacturing machine includes translating the extrusion tip relative to a build platform and rotating the extrusion tip about an axis of the extrusion tip relative to the build platform.