Abstract: Techniques for training a vision-based robot control model include generating, based on scene data, a plurality of scenes, generating, based on the plurality of scenes, one or more goal specifications, determining, based on the one or more goal specifications and a robot model, one or more robot plans, generating, based on the one or more robot plans and the plurality of scenes, simulated sensor data, and performing one or more training operations to generate a trained vision-based robot control model based on the one or more goal specifications, the one or more robot plans, and the simulated sensor data.

Abstract: Techniques for controlling a robot include receiving sensor data and one or more goal specifications, processing the sensor data, a robot size, and the one or more goal specifications using one or more trained encoders to generate a plurality context tokens, processing the plurality of context tokens using one or more trained decoders to generate a robot plan, and controlling a robot based on the robot plan.

Abstract: Techniques for robot control using multi-modal user inputs include receiving one or more multi-modal inputs from a user, extracting a motion hint from the one or more multi-modal inputs, generating estimated noise based on a current motion scene for the robot, generating a plurality of candidate motion plans, iteratively denoising the plurality of candidate motion plans based on the estimated noise and the motion hint to generate a plurality of revised robot motion plans, selecting a robot motion plan from the plurality of revised robot motion plans, generating a robot trajectory from the selected robot motion plan; and commanding the robot to perform a first step of the robot trajectory.

Abstract: Described herein is a framework for efficient task-agnostic, user-independent adaptive teleoperation of mobile robots and remotely operated vehicles (ROV), including ground vehicles (including legged systems), aircraft, watercraft and spacecraft. The efficiency of a human operator is improved by minimizing the entropy of the control inputs, thereby minimizing operator energy and achieving higher performance in the form of smoother trajectories by concurrently estimating the user intent online and adaptively updating the action set available to the human operator.

Abstract: Described herein is a framework for efficient task-agnostic, user-independent adaptive teleoperation of mobile robots and remotely operated vehicles (ROV), including ground vehicles (including legged systems), aircraft, watercraft and spacecraft. The efficiency of a human operator is improved by minimizing the entropy of the control inputs, thereby minimizing operator energy and achieving higher performance in the form of smoother trajectories by concurrently estimating the user intent online and adaptively updating the action set available to the human operator.