Abstract: Systems, methods, and control modules for operating robot systems are described. A touch heatmap is generated for an object, indicating at least one touch region for the object. Touch regions indicate regions of an object which are most suitable or most prone to touching when the object is grasped. At least one end effector is controlled in accordance with at least one grasp primitive, to grasp the object in accordance with at least one touch region specified in the touch heatmap.

Abstract: Systems, methods, and computer program products for managing and populating environment models are described. An environment model is accessed which represents an environment, and the environment model is populated with instances of object models. Locations where the instances of object models should be positioned in the environment model are identified, by determining where in the environment model a respective size of each instance when viewed from a vantage point at the environment model matches a size of the object represented by the respective instance when viewed from a corresponding vantage point at the environment.

Abstract: Systems, methods, and computer program products for managing and populating environment models are described. An environment model is accessed which represents an environment, and the environment model is populated with instances of object models. Locations where the instances of object models should be positioned in the environment model are identified, by determining where in the environment model a respective size of each instance when viewed from a vantage point at the environment model matches a size of the object represented by the respective instance when viewed from a corresponding vantage point at the environment.

Abstract: Systems, methods, and computer program products for managing and populating environment models are described. An environment model is accessed which represents an environment, and the environment model is populated with instances of object models. Locations where the instances of object models should be positioned in the environment model are identified, by determining where in the environment model a respective size of each instance when viewed from a vantage point at the environment model matches a size of the object represented by the respective instance when viewed from a corresponding vantage point at the environment.

Abstract: Robots, systems, methods, and computer program products for training and operating (semi-)autonomous robots to complete work objectives are described. A robot accesses a library of reusable work primitives from a catalog of libraries of reusable work primitives, each reusable work primitive corresponding to a respective basic sub-action that the robot is trained to autonomously perform. A work objective is analyzed to determine a sequence of reusable work primitives that complete the work objective, and the robot executes the sequence to complete the work objective. A robot can be deployed with access to an appropriate library of reusable work primitives, based on expectations for the robot. The robot is trained to perform reusable work primitives in multiple libraries, by generating control instructions which cause the robot to perform each reusable work primitive.

Abstract: Robots, systems, methods, and computer program products for training and operating (semi-)autonomous robots to complete work objectives are described. A robot accesses a library of reusable work primitives from a catalog of libraries of reusable work primitives, each reusable work primitive corresponding to a respective basic sub-action that the robot is trained to autonomously perform. A work objective is analyzed to determine a sequence of reusable work primitives that complete the work objective, and the robot executes the sequence to complete the work objective. A robot can be deployed with access to an appropriate library of reusable work primitives, based on expectations for the robot. The robot is trained to perform reusable work primitives in multiple libraries, by generating control instructions which cause the robot to perform each reusable work primitive.

Abstract: Provided herein is a control system and methods thereof for controlling operation of a robot. The control system comprises: a robot, including a plurality of sensors wherein each generating a stream of raw sensor data having a data type, size, and frequency, and a plurality of actuators cause movement of the robot; an autonomous control subsystem configured to receive the sensor data and output autonomous actuator data; and a teleoperation control subsystem configured to receive the sensor data, transmit the sensor data to a human operator, and output teleoperation actuator control signals, wherein the autonomous control subsystem and the teleoperation control subsystem receive, from the robot, sensor data having the same data type, size, and frequency, and wherein the autonomous actuator data and the teleoperation actuator data have the same data type, size, and frequency; and a control-determining subsystem configured to switch between autonomous and teleoperation control.

Abstract: Robot control systems, methods, control modules and computer program products that leverage one or more large language model(s) (LLMs) in order to achieve at least some degree of autonomy are described. Robot control parameters and/or instructions may advantageously be specified in natural language (NL) and communicated with the LLM via a recursive sequence of NL prompts or queries. Corresponding NL responses from the LLM may then be converted into robot control parameters and/or instructions. In this way, an LLM may be leveraged by the robot control system to enhance the autonomy of various operations and/or functions, including without limitation task planning, motion planning, human interaction, and/or reasoning about the environment.

Abstract: Robot control systems, methods, control modules and computer program products that leverage one or more large language model(s) (LLMs) in order to achieve at least some degree of autonomy are described. Robot control parameters and/or instructions may advantageously be specified in natural language (NL) and communicated with the LLM via an NL prompt or query. The LLM module provides a task plan in NL, which can be evaluated for at least one fault or error. If at least one fault or error is identified, the LLM module can be queried to provide a resolution.

Abstract: Provided herein is a control system and methods thereof for controlling operation of a robot. The control system comprises: a robot, including a plurality of sensors wherein each generating a stream of raw sensor data having a data type, size, and frequency, and a plurality of actuators cause movement of the robot; an autonomous control subsystem configured to receive the sensor data and output autonomous actuator data; and a teleoperation control subsystem configured to receive the sensor data, transmit the sensor data to a human operator, and output teleoperation actuator control signals, wherein the autonomous control subsystem and the teleoperation control subsystem receive, from the robot, sensor data having the same data type, size, and frequency, and wherein the autonomous actuator data and the teleoperation actuator data have the same data type, size, and frequency; and a control-determining subsystem configured to switch between autonomous and teleoperation control.

Abstract: Control systems for controlling operation of a robot, as well as computer program products and methods thereof are provided herein, The control system comprises a robot, including a plurality of sensors configured to convert information from the environment and the robot into sensor data and a plurality of actuators; a cognitive architecture control system, communicatively coupled to the robot, configured to control the robot, to receive the sensor data, to generate a robot-egocentric model of the environment from the sensor data, and to output autonomous actuator data to the plurality of actuators of the robot, based on at least one instruction; and a graphical user interface configured to display a graphical representation of the robot-egocentric model to a human operator and enable the human operator to select the at least one instruction from a pre-determined instruction set, based on the robot-egocentric model, to control the robot semi-autonomously.

Abstract: Control systems for controlling operation of a robot, as well as computer program products and methods thereof are provided herein, The control system comprises a robot, including a plurality of sensors configured to convert information from the environment and the robot into sensor data and a plurality of actuators; a cognitive architecture control system, communicatively coupled to the robot, configured to control the robot, to receive the sensor data, to generate a robot-egocentric model of the environment from the sensor data, and to output autonomous actuator data to the plurality of actuators of the robot, based on at least one instruction; and a graphical user interface configured to display a graphical representation of the robot-egocentric model to a human operator and enable the human operator to select the at least one instruction from a pre-determined instruction set, based on the robot-egocentric model, to control the robot semi-autonomously.

Abstract: Robots, methods, and computer program products for training and operating (semi-) autonomous robots to complete multiple different work objectives are described. A robot accesses a library of reusable work primitives from a catalog of libraries of reusable work primitives, each reusable work primitive corresponding to a respective basic sub-task or sub-action that the robot is operative to autonomously perform. A work objective is analyzed to determine a sequence (i.e., a combination and/or permutation) of reusable work primitives that, when executed by the robot, will complete the work objective. The robot executes the sequence of reusable work primitives to complete the work objective. A robot can be deployed with an appropriate stored library (or access to an appropriate library) of reusable work primitives, based on what the robot is expected to do, or what service category or role the robot will operate in.

Abstract: Systems, devices, and methods for training and operating (semi-)autonomous robots to complete multiple different work objectives are described. A robot control system stores a library of reusable work primitives each corresponding to a respective basic sub-task or sub-action that the robot is operative to autonomously perform. A work objective is analyzed to determine a sequence (i.e., a combination and/or permutation) of reusable work primitives that, when executed by the robot, will complete the work objective. The robot executes the sequence of reusable work primitives to complete the work objective. The reusable work primitives may include one or more reusable grasp primitives that enable(s) a robot's end effector to grasp objects. Simulated instances of real physical robots may be trained in simulated environments to develop control instructions that, once uploaded to the real physical robots, enable such real physical robots to autonomously perform reusable work primitives.

Abstract: The present disclosure describes robots, tele-operation systems, methods, and computer program products where a robot is selectively operable in a plurality of control modes. Based on identification of a fault condition (when the robot fails to act in a suitable or sufficient manner), a control mode of the robot can be changed to provide a human operator with more explicit control over the robot. In this way, the fault condition can be resolved by human operator input, and the control modes, AI, or control paradigm for the robot can be trained to perform better in the future.

Abstract: The present disclosure describes systems, robots, and methods for organizing and selecting classifiers of a library of classifiers. The classifiers of the library of classifiers can be organized in a relational model, such as a hierarchy or probability model. Instead of storing, activating, or executing the entire library of classifiers at once by a robot system, computational resource demand is reduced by executing subset of classifiers to determine context, and the determined context is used as a basis for selection of another subset of classifiers. This process can be repeated, to iteratively refine context and select more specific subsets of classifiers. A selected subset of classifiers can eventually be specific to a task to be performed by the robot system, such that the robot system can take action based on output from executing such specific classifiers.

Abstract: Robot control systems, methods, control modules and computer program products that leverage one or more large language model(s) (LLMs) in order to achieve at least some degree of autonomy are described. Robot control parameters, environment details, and/or instructions may advantageously be specified in natural language (NL) and communicated with the LLM via an NL prompt or query. The NL query may include a request for one or more work objectives from the LLM, such as “What can I do here?”, thereby establishing a form of agency by which the robot system may identify activities to perform without operator intervention. The LLM may also be queried to convert each work objective into a task plan providing a sequence of steps that the robot system may execute to complete the work objective. Optionally, the robot system may communicate with an operator to determine whether or not to execute a task plan.

Abstract: The present disclosure describes systems, robots, and methods for organizing and selecting classifiers of a library of classifiers. The classifiers of the library of classifiers can be organized in a relational model, such as a hierarchy or probability model. Instead of storing, activating, or executing the entire library of classifiers at once by a robot system, computational resource demand is reduced by executing subset of classifiers to determine context, and the determined context is used as a basis for selection of another subset of classifiers. This process can be repeated, to iteratively refine context and select more specific subsets of classifiers. A selected subset of classifiers can eventually be specific to a task to be performed by the robot system, such that the robot system can take action based on output from executing such specific classifiers.

Abstract: Robot control systems, methods, control modules, and computer program products that leverage one or more large language model(s) (LLMs) in order to achieve at least some degree of autonomy are described. Robot control parameters, environment details, and/or instruction sets may advantageously be specified in natural language (NL) and communicated with the LLM via an NL prompt or query. An NL response from the LLM may then be converted into a task plan. A task plan that successfully completes a first instance of a work objective may be parameterized and re-used to complete a second instance of the work objective. Parameterization of a task plan may include replacing one or more nouns/objects in the NL task plan with variables, while optionally preserving one or more verbs/actions in the NL task plan.

Abstract: The present disclosure relates to utilizing idle processing resources of a robot to reduce future burden on such processing resources. In particular, idle processing resources are utilized to identify future scenarios, and generate reactions to such future scenarios. The generated reactions are stored, and quickly retrieved as needed if corresponding identified future scenarios occur.