Patents by Inventor Diego Romeres

Diego Romeres has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20240300097
    Abstract: A controller for controlling robotic manipulator according to a task is provided. The controller is to collect data relating to a state and an object property of an object, and execute a state adapter model to produce a state correction to state of the object having the object property different from a unitary property of a unitary object. The controller is to execute a control policy using the state correction to produce an action for the unitary object, and execute an action adapter model to produce an action correction to the action produced by the control policy. The state correction and action correction are produced based on difference between object property and unitary property. The control policy is to map a state of the unitary object to the action of the robotic manipulator to manipulate the unitary object according to the task.
    Type: Application
    Filed: March 6, 2023
    Publication date: September 12, 2024
    Inventors: Diego Romeres, Xiang Zhang
  • Publication number: 20240288870
    Abstract: A method, a system and a computer program product are provided for applying a neural network including an action sequence decoder for generating an action sequence for a robot to perform a task. The neural network is applied to generate the action sequence based on recordings demonstrating humans performing tasks. In an example, the method comprises collecting a recording and a sequence of captions describing scenes in the recording; extracting feature data from the recording; encoding the extracted feature data to produce a sequence of encoded features; and applying the action sequence decoder to produce a sequence of actions for the robot based on the sequence of encoded features having a semantic meaning corresponding to a semantic meaning of the sequence of captions. The feature data includes features of a video signal, an audio signal, and/or text transcription capturing a performance of the task.
    Type: Application
    Filed: September 27, 2023
    Publication date: August 29, 2024
    Applicant: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Chiori Hori, Jonathan Le Roux, Devesh Jha, Siddarth Jain, Radu Ioan Corcodel, Diego Romeres, Puyuang Peng, Xinyu Liu, David Harwath
  • Patent number: 12049007
    Abstract: A robotic system for manipulating an object with a robotic manipulator is provided. The robotic system is configured to collect a digital representation of a task for manipulating the object; solve a robust control problem to optimize a sequence of control forces to be applied by the robotic manipulator to change a state of the object, where an evolution of the state of the object is governed by a stochastic complementarity system modeling the task with a predefined probability. The robust control problem optimizes a cost function to generate the sequence of control forces performing the task subject to joint chance constraints including a first chance constraint on the state of the object being manipulated and a second chance constraint on stochastic complementarily constraints modeling manipulation of the object. The robotic system is further configured to control the manipulation of the object based on the sequence of control forces.
    Type: Grant
    Filed: March 1, 2022
    Date of Patent: July 30, 2024
    Assignee: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Devesh Jha, Arvind Raghunathan, Yuki Shirai, Diego Romeres
  • Publication number: 20240227179
    Abstract: A robotic controller is provided for generating sequences of movement primitives for sequential tasks of a robot having a manipulator. The controller includes at least one control processor, and a memory circuitry storing a dictionary including the movement primitives, a pretrained learning module, and a graph-search based planning module having instructions stored thereon.
    Type: Application
    Filed: October 20, 2022
    Publication date: July 11, 2024
    Inventors: Devesh Jha, Diego Romeres, Daniel Nikovski
  • Publication number: 20240131698
    Abstract: A robotic controller is provided for generating sequences of movement primitives for sequential tasks of a robot having a manipulator. The controller includes at least one control processor, and a memory circuitry storing a dictionary including the movement primitives, a pretrained learning module, and a graph-search based planning module having instructions stored thereon.
    Type: Application
    Filed: October 19, 2022
    Publication date: April 25, 2024
    Inventors: Devesh Jha, Diego Romeres, Daniel Nikovski
  • Publication number: 20240083029
    Abstract: A controller for controlling an operation of a robot to execute a task is provided. The controller comprises a memory configured to store a set of dynamic movement primitives (DMPs) associated with the task. The set of DMPs comprise a set of at least two dynamical systems: a function representing point attractor dynamics and a forcing function corresponding to a learned demonstration of the task. The controller comprises a processor configured to transform the set of DMPs to a set of constrained DMPs (CDMPs) by determining a perturbation function associated with the forcing function. The perturbation function is associated with a set of operational constraints. The processor is further configured to solve, a non-linear optimization problem for the set of CDMPs based on the set of operational constraints and generate, a control input for controlling the robot for executing the task, based on the solution.
    Type: Application
    Filed: September 14, 2022
    Publication date: March 14, 2024
    Inventors: Devesh Jha, Seiji Shaw, Arvind Raghunathan, Radu Ioan Corcodel, Diego Romeres, Daniel Nikovski
  • Patent number: 11883962
    Abstract: A controller controls a motion of an object performing a task for changing a state of the object from a start state to an end state while avoiding collision of the object with an obstacle according to an optimal trajectory determined by solving an optimization problem of the dynamics of the object producing an optimal trajectory for performing the task subject to constraints on a solution of first-order stationary conditions modeling a minimum distance between the convex hull of the object and the convex hull of the obstacle using complementarity constraints.
    Type: Grant
    Filed: May 28, 2021
    Date of Patent: January 30, 2024
    Assignee: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Arvind Raghunathan, Devesh Jha, Diego Romeres
  • Publication number: 20230294283
    Abstract: A manipulation controller is provided for reorienting an object by a manipulator of a robotic system. The manipulation controller includes an interface controller configured to acquire measurement data from sensors arranged on the robotic system, at least one processor, and a memory configured to store a computer-implemented method.
    Type: Application
    Filed: March 18, 2022
    Publication date: September 21, 2023
    Applicant: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Devesh Jha, Yuki Shirai, Arvind Raghunathan, Diego Romeres
  • Publication number: 20230278197
    Abstract: A robotic system for manipulating an object with a robotic manipulator is provided. The robotic system is configured to collect a digital representation of a task for manipulating the object; solve a robust control problem to optimize a sequence of control forces to be applied by the robotic manipulator to change a state of the object, where an evolution of the state of the object is governed by a stochastic complementarity system modeling the task with a predefined probability. The robust control problem optimizes a cost function to generate the sequence of control forces performing the task subject to joint chance constraints including a first chance constraint on the state of the object being manipulated and a second chance constraint on stochastic complementarily constraints modeling manipulation of the object. The robotic system is further configured to control the manipulation of the object based on the sequence of control forces.
    Type: Application
    Filed: March 1, 2022
    Publication date: September 7, 2023
    Applicant: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Devesh Jha, Arvind Raghunathan, Yuki Shirai, Diego Romeres
  • Patent number: 11673264
    Abstract: A robot for performing an assembly operation is provided. The robot comprises a processor configured to determine a control law for controlling a plurality of motors of the robot to move a robotic arm according to an original trajectory, execute a self-exploration program to produce training data indicative of a space of the original trajectory, and learn, using the training data, a non-linear compliant control law including a non-linear mapping that maps measurements of a force sensor of the robot to a direction of corrections to the original trajectory defining the control law. The processor transforms the original trajectory according to a new goal pose to produce a transformed trajectory, update the control law according to the transformed trajectory to produce the updated control law, and command the plurality of motors to control the robotic arm according to the updated control law corrected with the compliance control law.
    Type: Grant
    Filed: March 25, 2021
    Date of Patent: June 13, 2023
    Assignee: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Daniel Nikolaev Nikovski, Diego Romeres, Devesh Jha, William Yerazunis
  • Patent number: 11650551
    Abstract: A computer-implemented learning method for optimizing a control policy controlling a system is provided. The method includes receiving states of the system being operated for a specific task, initializing the control policy as a function approximator including neural networks, collecting state transition and reward data using a current control policy, estimating an advantage function and a state visitation frequency based on the current control policy, updating the current control policy using the second-order approximation of the objective function, a second-order approximation of the KL-divergence constraint on the permissible change in the policy using a quasi-newton trust region policy optimization, and determining an optimal control policy, for controlling the system, based on the average reward accumulated using the updated current control policy.
    Type: Grant
    Filed: October 4, 2019
    Date of Patent: May 16, 2023
    Assignee: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Devesh Jha, Arvind Raghunathan, Diego Romeres
  • Publication number: 20230119664
    Abstract: A method for controlling a system by a controller comprises accepting a current state of the system and selecting, using a trained function of the current state, a solver from a set of solvers. The method further comprises solving an optimal control optimization problem using the selected solver to produce a current control input, such that for at least some different control steps, the predictive controller solves a formulation of the optimal control optimization problem with different solvers having different accuracies, requiring different computational resources, or both and submitting the current control input to the system thereby changing the current state of the system.
    Type: Application
    Filed: October 19, 2021
    Publication date: April 20, 2023
    Applicant: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Ankush Chakrabarty, Rien Quirynen, Diego Romeres, Stefano Di Cairano
  • Publication number: 20220379478
    Abstract: A controller controls a motion of an object performing a task for changing a state of the object from a start state to an end state while avoiding collision of the object with an obstacle according to an optimal trajectory determined by solving an optimization problem of the dynamics of the object producing an optimal trajectory for performing the task subject to constraints on a solution of first-order stationary conditions modeling a minimum distance between the convex hull of the object and the convex hull of the obstacle using complementarity constraints.
    Type: Application
    Filed: May 28, 2021
    Publication date: December 1, 2022
    Applicant: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Arvind Raghunathan, Devesh Jha, Diego Romeres
  • Patent number: 11472028
    Abstract: A system for detecting an anomaly in an execution of a task in mixed human-robot processes. Receiving human worker (HW) signals and robot signals. A processor to extract from the HW signals, task information, measurements relating to a state of the HW, and input into a Human Performance (HP) model, to obtain a state of the HW based on previously learned boundaries of the state of the HW, the state of the HW is then inputted into a Human-Robot Interaction (HRI) model, to determine a classification of an anomaly or no anomaly. Update HRI model with robot operation signals, HW signals and classified anomaly, determine a control action of a robot interacting with the HW or a type of an anomaly alarm using the updated HRI model and classified anomaly. Output the control action of the robot to change a robot action or output the type of the anomaly alarm.
    Type: Grant
    Filed: December 6, 2019
    Date of Patent: October 18, 2022
    Assignee: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Emil Laftchiev, Diego Romeres
  • Publication number: 20220305645
    Abstract: A robot for performing an assembly operation is provided. The robot comprises a processor configured to determine a control law for controlling a plurality of motors of the robot to move a robotic arm according to an original trajectory, execute a self-exploration program to produce training data indicative of a space of the original trajectory, and learn, using the training data, a non-linear compliant control law including a non-linear mapping that maps measurements of a force sensor of the robot to a direction of corrections to the original trajectory defining the control law. The processor transforms the original trajectory according to a new goal pose to produce a transformed trajectory, update the control law according to the transformed trajectory to produce the updated control law, and command the plurality of motors to control the robotic arm according to the updated control law corrected with the compliance control law.
    Type: Application
    Filed: March 25, 2021
    Publication date: September 29, 2022
    Applicant: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Daniel Nikolaev Nikovski, Diego Romeres, Devesh Jha, William Yerazunis
  • Patent number: 11442429
    Abstract: A system for detection of an anomaly in a discrete manufacturing process (DMP) with human-robot teams executing a task. Receive signals including robot, worker and DMP signals. Predict a sequence of events (SOEs) from DMP signals. Determine whether the predicted SOEs in the DMP signals is inconsistent with a behavior of operation of the DMP described in a DMP model, and if the predicted SOEs from DMP signals is inconsistent with the behavior, then an alarm is to be signaled. Input worker data into a Human Performance (HP) model, to obtain a state of the worker based on previously learned boundaries of human state. The state of the HW is then input into the HRI model and the DMP model to determine a classification of anomaly or no anomaly. Update a Human-Robot Interaction (HRI) model to obtain a control action of a robot or a type of an anomaly alarm.
    Type: Grant
    Filed: December 6, 2019
    Date of Patent: September 13, 2022
    Assignee: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Emil Laftchiev, Diego Romeres
  • Patent number: 11389957
    Abstract: A manipulator learning-control apparatus for controlling a manipulating system that includes an interface configured to receive manipulator state signals of the manipulating system and object state signals with respect to an object to be manipulated by the manipulating system in a workspace, wherein the object state signals are detected by at least one object detector, an output interface configured to transmit initial and updated policy programs to the manipulating system, a memory to store computer-executable programs including a data preprocess program, object state history data, manipulator state history data, a Derivative-Free Semi-parametric Gaussian Process (DF-SPGP) kernel learning program, a Derivative-Free Semi-parametric Gaussian Process (DF-SPGP) model learning program, an update-policy program and an initial policy program, and a processor, in connection with the memory, configured to transmit the initial policy program to the manipulating system for initiating a learning process that operates th
    Type: Grant
    Filed: September 30, 2019
    Date of Patent: July 19, 2022
    Assignee: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Diego Romeres, Alberto Dalla Libera, Devesh Jha, Daniel Nikolaev Nikovski
  • Publication number: 20220179419
    Abstract: A controller for controlling a system that includes a policy configured to control the system is provided. The controller includes an interface connected to the system, the interface being configured to acquire an action state and a measurement state via sensors measuring the system, a memory to store computer-executable program modules including a model learning module and a policy learning module, a processor configured to perform steps of the program modules. The steps include offline-modeling to generate offline-learning states based on the action state and measurement state using the model learning program, providing the offline states to the policy learning program to generate policy parameters, and updating the policy of the system to operate the system based on the policy parameters.
    Type: Application
    Filed: December 4, 2020
    Publication date: June 9, 2022
    Applicant: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Diego Romeres, Fabio Amadio, Alberto Dalla Libera, Riccardo Antonello, Ruggero Carli, Daniel Nikovski
  • Patent number: 11280514
    Abstract: A controller for controlling a heating, ventilating, and air-conditioning (HVAC) system arranged to condition an environment according to HVAC setpoints is provided. The controller is configured to accept target values of thermal states at predetermined locations in the conditioned environment, current values of the thermal states at the predetermined locations in the conditioned environment, and current values of the HVAC setpoints. The controller is further configured to determine, using a neural network, target HVAC setpoints such that a difference in an operation of the HVAC system according to the target HVAC points with respect to the operation of the HVAC system according to the current HVAC setpoints changes thermal states in the predetermined locations in the conditioned environment from the current values of the thermal state to the target values of the thermal state.
    Type: Grant
    Filed: November 15, 2020
    Date of Patent: March 22, 2022
    Assignee: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Emil Laftchiev, Daniel Nikovski, Diego Romeres
  • Patent number: 11161244
    Abstract: A system for controlling a robotic arm performing insertion of a component along an insertion line accepts measurements of force experienced by the wrist of robotic arm at current position along insertion line and determines probability of value of the force conditioned on the current value of the position according to a probabilistic relationship for the force experienced by the wrist of the robotic arm along the insertion line as a probabilistic function of the positions of the wrist of the robotic arm along the line of insertion. The probabilistic function is learned from measurements of the operation repeatedly performed by one or multiple robotic arms having the configuration of the robotic arm under the control. The system determines a result of anomaly detection based on the probability of the current value of the force and controls the robotic arm based on the result of anomaly detection.
    Type: Grant
    Filed: January 22, 2019
    Date of Patent: November 2, 2021
    Assignee: Mitsubishi Electric Research Laboratories, Inc.
    Inventors: Daniel Nikolaev Nikovski, Devesh Jha, Diego Romeres