Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling a robot to perform a custom real-time action. One of the methods comprises receiving, by a real-time robotics control framework, a definition of a custom real-time control function, wherein the definition specifies a plurality of actions and one or more custom reactions; repeatedly executing, by the real-time robotics control framework, the custom real-time control function at each tick of a real-time robotics system driving one or more physical robots, including: obtaining current values of one or more state variables, evaluating the one or more custom reactions specified by the custom real-time control function according to the current values of the one or more state variables, and whenever a custom reaction is satisfied, updating a current action in real time according to the custom reaction that is satisfied, and executing a next tick of the current action.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling a robot to perform a custom real-time action. One of the methods comprises receiving, by a real-time robotics control framework, a definition of a custom real-time control function, wherein the definition specifies a plurality of actions and one or more custom reactions; repeatedly executing, by the real-time robotics control framework, the custom real-time control function at each tick of a real-time robotics system driving one or more physical robots, including: obtaining current values of one or more state variables, evaluating the one or more custom reactions specified by the custom real-time control function according to the current values of the one or more state variables, and whenever a custom reaction is satisfied, updating a current action in real time according to the custom reaction that is satisfied, and executing a next tick of the current action.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for handling errors in a real-time robotic control system. One of the methods include: receiving a definition of a custom real-time action to be performed in real-time by the real-time robotic control system to control a robot; executing the custom real-time action according to the definition at each tick of a real-time control cycle to effectuate real-time operation of the robot, including: determining, by the custom real-time action, that a first error condition is satisfied, in response, returning a first error object from the custom real-time action to a real-time session which invoked the custom real-time action, receiving, by the real-time session, the first error object generated by the custom real-time action, and in response, executing a first recovery process.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using feature interfaces for real-time robotics control. One of the methods includes receiving a definition of a real-time action to be executed by the real-time robotics control system to control a robot using one or more underlying hardware subsystems. The real-time action is invoked including providing a respective part identifier of a part for each of the one or more feature interfaces called by the real-time action.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling a robot to perform a custom real-time action that uses a callback function. One of the methods comprises receiving a definition of a custom real-time control function that specifies a custom callback function, an action, and a custom reaction that references the custom callback function; providing a command to initiate the action; repeatedly executing, by the control layer of the real-time robotics control framework, the custom real-time control function at each tick of a real-time robotics system driving one or more physical robots, including: obtaining current values of one or more state variables, evaluating the custom reaction specified by the custom real-time control function according to the current values of the one or more state variables, and whenever the one or more conditions of the custom reaction are satisfied, invoking the custom callback function.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling a robot to perform a custom real-time action that uses streaming inputs. One of the methods comprises receiving a definition of a custom real-time streaming control function that defines a custom streaming action, wherein the custom streaming action specifies a goal state for a robot in an operating environment; providing a command to initiate the custom streaming action; and repeatedly providing updated goal states for the custom streaming action, wherein the control layer of the framework is configured to execute the custom streaming action including driving the robot toward a most recent goal state at each tick of a real-time robotics control cycle.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling a robot. One of the methods includes: receiving user input that specifies: (i) a real-time session with a robot having multiple parts, (ii) multiple actions, and (iii) at least one custom reaction that represents a condition under which a first action triggers a real-time change in behavior involving a second action; generating control parameters from the user input and providing the control parameters to a real-time robotics control layer; and executing, by the real-time robotics control layer, the control parameters including, at each tick of a real-time control cycle: executing one or more respective commands for each action, determining whether the condition of the first action is satisfied, and in response to determining that the condition of the first action is satisfied, triggering the real-time change in behavior involving the second action.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling a robot to perform a custom real-time action. One of the methods comprises receiving, by a real-time robotics control framework, a definition of a custom real-time control function, wherein the definition specifies a plurality of actions and one or more custom reactions; repeatedly executing, by the real-time robotics control framework, the custom real-time control function at each tick of a real-time robotics system driving one or more physical robots, including: obtaining current values of one or more state variables, evaluating the one or more custom reactions specified by the custom real-time control function according to the current values of the one or more state variables, and whenever a custom reaction is satisfied, updating a current action in real time according to the custom reaction that is satisfied, and executing a next tick of the current action.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling a robot to perform a custom real-time action that uses a callback function. One of the methods comprises receiving a definition of a custom real-time control function that specifies a custom callback function, an action, and a custom reaction that references the custom callback function; providing a command to initiate the action; repeatedly executing, by the control layer of the real-time robotics control framework, the custom real-time control function at each tick of a real-time robotics system driving one or more physical robots, including: obtaining current values of one or more state variables, evaluating the custom reaction specified by the custom real-time control function according to the current values of the one or more state variables, and whenever the one or more conditions of the custom reaction are satisfied, invoking the custom callback function.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling a robot to perform a custom real-time action that uses streaming inputs. One of the methods comprises receiving a definition of a custom real-time streaming control function that defines a custom streaming action, wherein the custom streaming action specifies a goal state for a robot in an operating environment; providing a command to initiate the custom streaming action; and repeatedly providing updated goal states for the custom streaming action, wherein the control layer of the framework is configured to execute the custom streaming action including driving the robot toward a most recent goal state at each tick of a real-time robotics control cycle.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for deblurring an image captured by a robot-mounted camera. One of the methods comprises capturing, using a camera that is attached to an arm of a robot, an image at a first time, wherein the image exhibits motion blur, and wherein the exhibited blur was caused by movement of the arm of the robot at the first time; receiving, from a robot control system of the robot, motion data characterizing the movement of the arm of the robot at the first time; generating a motion kernel using the received motion data; and generating a deblurred image by processing the image using the motion kernel.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for deblurring an image captured by a robot-mounted camera. One of the methods comprises capturing, using a camera that is attached to an arm of a robot, an image at a first time, wherein the image exhibits motion blur, and wherein the exhibited blur was caused by movement of the arm of the robot at the first time; receiving, from a robot control system of the robot, motion data characterizing the movement of the arm of the robot at the first time; generating a motion kernel using the received motion data; and generating a deblurred image by processing the image using the motion kernel.