Abstract: The present disclosure provides a humanoid robot system comprising a mechanical structure including a torso, two arms, and two legs providing at least 30 degrees of freedom, actuators coupled to the degrees of freedom, a sensor suite comprising at least one camera and proprioceptive sensors including joint encoders and an inertial measurement unit, a computing system comprising at least one processor and memory storing instructions which, when executed, implement a hierarchical bipedal action model including a Beta model configured to receive multimodal input data and generate a token sequence indicative of task intent and environmental state, and an Alpha model configured to condition on the token sequence and current robot pose data to output continuous action chunks comprising sequences of future target joint states over a finite horizon, and a low-level controller configured to convert the continuous action chunks into actuator control signals for execution.

Abstract: The present disclosure provides a method for controlling a humanoid robot using a hierarchical bipedal action model (BAM), the method comprising obtaining a base controller by training in simulation with reinforcement learning, instantiating an initial BAM including a Gamma model configured to generate intermediate goals, a Beta model configured to translate the intermediate goals into task-space actions, and an Alpha model configured to translate the task-space actions and robot state into motor commands, deploying the initial BAM such that at least the Alpha model executes on-board the humanoid robot, causing the humanoid robot to perform an initial task and logging sensor and control data to form a first dataset, based on the first dataset, training at least one policy of the BAM to generate a refined BAM, and deploying the refined BAM to control the humanoid robot autonomously.

Abstract: The present disclosure provides a humanoid robot system comprising a mechanical structure with at least 30 degrees of freedom across torso, arms, and legs, actuators driving the degrees of freedom, sensors including cameras and proprioceptive sensors, and a computing system implementing a hierarchical bipedal action model (BAM). The BAM includes: a Delta model processing sensor data and user input to generate latent representations at a first frequency; a Gamma model receiving latent representations to generate human task actions at a higher second frequency; a Beta model translating task actions into joint configurations at a higher third frequency; and an Alpha model converting joint configurations into actuator control signals at a higher fourth frequency.

Abstract: The present disclosure provides a humanoid robot system comprising a mechanical structure including a torso, two arms, and two legs providing at least 30 degrees of freedom, actuators coupled to the degrees of freedom, a sensor suite comprising at least one camera and proprioceptive sensors including joint encoders and an inertial measurement unit, a computing system comprising at least one processor and memory storing instructions which, when executed, implement a hierarchical bipedal action model including a Beta model configured to receive multimodal input data and generate a token sequence indicative of task intent and environmental state, and an Alpha model configured to condition on the token sequence and current robot pose data to output continuous action chunks comprising sequences of future target joint states over a finite horizon, and a low-level controller configured to convert the continuous action chunks into actuator control signals for execution.

Abstract: The present disclosure provides a control system for a humanoid robot comprising a bipedal action model (BAM) with hierarchical architecture including a beta model executing cognitive tasks at lower frequency, ingesting multimodal sensory inputs including visual data and natural language instructions, and an alpha model executing reactive tasks at higher frequency, communicatively coupled to the beta model. The BAM is trained on retargeted robot training data derived from robot-free training data. At runtime, the BAM outputs continuous control commands as parallel-generated action chunks controlling at least 18 degrees of freedom. The system includes a wearable collection apparatus capturing movement data from a human operator without physical connection to the robot, and a retargeting module translating robot-free training data into robot training data by solving embodiment mismatches between human and robot kinematic structures.

Abstract: The present disclosure provides a humanoid robot system comprising a mechanical structure with at least 30 degrees of freedom across torso, arms, and legs, actuators driving the degrees of freedom, sensors including cameras and proprioceptive sensors, and a computing system implementing a hierarchical bipedal action model (BAM). The BAM includes: a Delta model processing sensor data and user input to generate latent representations at a first frequency; a Gamma model receiving latent representations to generate human task actions at a higher second frequency; a Beta model translating task actions into joint configurations at a higher third frequency; and an Alpha model converting joint configurations into actuator control signals at a higher fourth frequency.

Abstract: A robot comprising a sensor configured to obtain data, an alpha model trained on video data and configured to generate data based upon both a spoken command from a human and data from the sensor, and a beta model configured to generate output data used to control an extent of the robot based in part upon the data generated by the alpha model and the data from the sensor.

Abstract: A robot comprising a sensor configured to obtain data, an alpha model configured to generate data based upon both a spoken command from a human and data from the sensor, a retrieval-augmented generation module configured to obtain additional real-time knowledge from external sources, and a beta model configured to generate output data used to control an extent of the robot based in part upon the data generated by the alpha model, the additional real-time knowledge obtained by the retrieval-augmented generation module, and the data from the sensor.

Abstract: A humanoid robot includes a torso, a left arm assembly coupled to the torso and having a first reference line, a left wrist coupled to the left arm assembly and including at least a rotational axis, and a left end effector coupled to the left wrist. The left end effector is configured to move about the rotational axis and includes a finger assembly with a second reference line and at least two degrees of freedom, and a thumb assembly with at least three degrees of freedom. A first angle is formed between the first and second reference lines when the left wrist is in a first configuration, and a second angle is formed when the left wrist is in a second configuration. Both the first and second angles are greater than 70 degrees, and the difference between the first and second angles is greater than 150 degrees.

Abstract: The present disclosure provides a method for coordinating task execution among multiple humanoid robots, comprising receiving a high-level task command, decomposing it into sub-tasks, determining a cost-optimized assignment using a cost-optimized bipedal action model (CoBAM) based on energy consumption, time to completion, and robot capabilities, and transmitting the assignment to assigned robots. The CoBAM comprises a hierarchical architecture including an L2 beta model operating at 1-20 Hz for high-level planning and an L1 alpha model operating at 100-10,000 Hz for continuous control commands. The cost function considers battery levels, physical distances between robot and sub-task locations, and mechanical wear factors associated with specific joint movements.

Abstract: The present disclosure provides a method for generating annotation data for robotic training using a hierarchical transformer-based model with multiple layers. The transformer-based model includes Alpha models generating low-level control outputs and Beta models generating high-level control outputs. The method receives multimodal input data comprising visual sensor data and natural language instructions, processes this data through the hierarchical transformer-based model to generate annotations at different abstraction levels, wherein Beta models create semantic annotations describing task objectives and Alpha models generate motor command annotations specifying robotic actions, and stores these annotations with the input data to create annotated training data for robotic control systems.

Abstract: The humanoid robot data collection system may include a wearable data collection apparatus, a computer, and a humanoid robot in data communication via a network. The wearable data collection apparatus worn by a human operator to control a robot for the purpose of collecting robot data. The wearable data collection apparatus generates pilot movement data based on various sensors contain in the wearable data collection apparatus, where the pilot movement data is used to generate robot control data and sends this control data to the robot in real-time. The wearable data collection apparatus includes a base mount, articulated arms extending from the base mount, sensors coupled to articulated arms, gloves with sensors coupled to the articulated arms, and a piloting control system operatively connected to the plurality of sensors.

Abstract: Techniques are disclosed that enable training a goal-conditioned policy based on multiple data sets, where each of the data sets describes a robot task in a different way. For example, the multiple data sets can include: a goal image data set, where the task is captured in the goal image; a natural language instruction data set, where the task is described in the natural language instruction; a task ID data set, where the task is described by the task ID, etc. In various implementations, each of the multiple data sets has a corresponding encoder, where the encoders are trained to generate a shared latent space representation of the corresponding task description. Additional or alternative techniques are disclosed that enable control of a robot using a goal-conditioned policy network. For example, the robot can be controlled, using the goal-conditioned policy network, based on free-form natural language input describing robot task(s).

Abstract: The present disclosure provides a humanoid robot comprising a torso having an alpha model deployed on a first GPU, and wherein said alpha model includes a first number of parameters and is configured to receive a natural language command from a human and generate processed data, a beta model deployed on a second GPU, and wherein said beta model includes a second number of parameters and is configured to receive the processed data from the alpha model and provide output data used to control an extent of the left wrist, and wherein the first number of parameters is larger than the second number of parameters, and a unified training framework is used to jointly train the alpha model and the beta model.

Abstract: The present disclosure provides a system for generating motor control commands for a humanoid robot, comprising an alpha model with over 1 billion parameters that processes visual observations and language instructions at a first frequency to generate contextual embeddings, and a beta model operating at a higher second frequency. The beta model includes an embodiment-specific state encoder projecting robot state information into a shared embedding space, a diffusion transformer module generating denoised action sequences through iterative flow-matching that cross-attends to the alpha model's contextual embeddings, and an embodiment-specific action decoder converting denoised sequences into motor control commands. The beta model generates action chunks comprising future action sequences over a predetermined time horizon in a single inference step, with the complete system having less than 5 billion parameters.

Abstract: The humanoid robot data collection system may include a wearable data collection apparatus, a computer, and a humanoid robot in data communication via a network. The wearable data collection apparatus worn by a human operator to control a robot for the purpose of collecting robot data. The wearable data collection apparatus generates pilot movement data based on various sensors contain in the wearable data collection apparatus, where the pilot movement data is used to generate robot control data and sends this control data to the robot in real-time. The wearable data collection apparatus includes a base frame, articulated arms extending from the base frame, sensors coupled to articulated arms, gloves with sensors coupled to the articulated arms, and a piloting control system operatively connected to the plurality of sensors.

Abstract: The present disclosure provides a humanoid robot comprising a torso having an alpha model deployed on a first GPU, and wherein said alpha model includes a first number of parameters and is configured to receive a natural language command from a human and generate processed data, a beta model deployed on a second GPU, and wherein said beta model includes a second number of parameters and is configured to receive the processed data from the alpha model and provide output data used to control an extent of the left wrist, and wherein the first number of parameters is larger than the second number of parameters, and a unified training framework is used to jointly train the alpha model and the beta model.

Abstract: Techniques are disclosed that enable training a goal-conditioned policy based on multiple data sets, where each of the data sets describes a robot task in a different way. For example, the multiple data sets can include: a goal image data set, where the task is captured in the goal image; a natural language instruction data set, where the task is described in the natural language instruction; a task ID data set, where the task is described by the task ID, etc. In various implementations, each of the multiple data sets has a corresponding encoder, where the encoders are trained to generate a shared latent space representation of the corresponding task description. Additional or alternative techniques are disclosed that enable control of a robot using a goal-conditioned policy network. For example, the robot can be controlled, using the goal-conditioned policy network, based on free-form natural language input describing robot task(s).

Abstract: A humanoid robot includes a torso, a left arm assembly coupled to the torso and having a first reference line, a left wrist coupled to the left arm assembly and including at least a rotational axis, and a left end effector coupled to the left wrist. The left end effector is configured to move about the rotational axis and includes a finger assembly with a second reference line and at least two degrees of freedom, and a thumb assembly with at least three degrees of freedom. A first angle is formed between the first and second reference lines when the left wrist is in a first configuration, and a second angle is formed when the left wrist is in a second configuration. Both the first and second angles are greater than 70 degrees, and the difference between the first and second angles is greater than 150 degrees.

Abstract: Techniques are disclosed that enable training a goal-conditioned policy based on multiple data sets, where each of the data sets describes a robot task in a different way. For example, the multiple data sets can include: a goal image data set, where the task is captured in the goal image; a natural language instruction data set, where the task is described in the natural language instruction; a task ID data set, where the task is described by the task ID, etc. In various implementations, each of the multiple data sets has a corresponding encoder, where the encoders are trained to generate a shared latent space representation of the corresponding task description. Additional or alternative techniques are disclosed that enable control of a robot using a goal-conditioned policy network. For example, the robot can be controlled, using the goal-conditioned policy network, based on free-form natural language input describing robot task(s).