Abstract: A robotic device includes a large-language model that uses in-context learning to predict robot actions without additional training. The robotic device may include one or more processors configured to execute code that causes the one or more processors to use in-context learning. A method of using a large-language model in a robotic device to predict robot actions includes using in-context learning without additional training.

Abstract: A robotic device includes a robot having an end-effector, and a large modality model (LMM) pre-trained on vision-language tasks and fine-tuned on image-visual trace pairs. A method of predicting a next sequence of actions for a robot using a large modality model (LMM) includes receiving, at the LMM, an image input and a language input, using the LMM to predict a next action sequence for a robot having an end-effector, and using the LMM to produce predicted visual traces of the end-effector.

Abstract: A robotic system includes a robot, and one or more processors configured to execute code that cause the one or more processors to: estimate four-dimensional representations from videos, operate a machine learning transformer model to use information from the four-dimensional representation for training and testing a policy for predicting a next action for the robot, and use the machine learning transformer model with the robot.