Abstract: Implementations set forth herein relate to generating training data, such that each instance of training data includes a corresponding instance of vision data and drivability label(s) for the instance of vision data. A drivability label can be determined using first vision data from a first vision component that is connected to the robot. The drivability label(s) can be generated by processing the first vision data using geometric and/or heuristic methods. Second vision data can be generated using a second vision component of the robot, such as a camera that is connected to the robot. The drivability labels can be correlated to the second vision data and thereafter used to train one or more machine learning models. The trained models can be shared with a robot(s) in furtherance of enabling the robot(s) to determine drivability of areas captured in vision data, which is being collected in real-time using one or more vision components.

Abstract: Implementations set forth herein relate to generating training data, such that each instance of training data includes a corresponding instance of vision data and drivability label(s) for the instance of vision data. A drivability label can be determined using first vision data from a first vision component that is connected to the robot. The drivability label(s) can be generated by processing the first vision data using geometric and/or heuristic methods. Second vision data can be generated using a second vision component of the robot, such as a camera that is connected to the robot. The drivability labels can be correlated to the second vision data and thereafter used to train one or more machine learning models. The trained models can be shared with a robot(s) in furtherance of enabling the robot(s) to determine drivability of areas captured in vision data, which is being collected in real-time using one or more vision components.

Abstract: A method includes receiving one or more past trajectories navigated by a robotic device in an environment, wherein the one or more past trajectories are associated with initial environmental sensor data and one or more obstacle detection heuristics. The method also includes determining, based at least on subsequent environmental sensor data, one or more updated obstacle detection heuristics. The method further includes determining, based on the one or more updated obstacle detection heuristics and the initial environmental sensor data, one or more predicted drivable areas in the environment. The method additionally includes, based on the one or more predicted drivable areas including the one or more past trajectories, using the one or more updated obstacle detection heuristics to determine future navigation of the robotic device.

Abstract: A method includes receiving, from a sensor on a robotic device, a captured image representative of an environment of the robotic device when the robotic device is at a location in the environment. The method also includes determining, based at least on the location of the robotic device, a rendered image representative of the environment of the robotic device. The method further includes determining, by applying at least one pre-trained machine learning model to at least the captured image and the rendered image, a property of one or more portions of the captured image.

Abstract: A method includes receiving one or more past trajectories navigated by a robotic device in an environment, wherein the one or more past trajectories are associated with initial environmental sensor data and one or more obstacle detection heuristics. The method also includes determining, based at least on subsequent environmental sensor data, one or more updated obstacle detection heuristics. The method further includes determining, based on the one or more updated obstacle detection heuristics and the initial environmental sensor data, one or more predicted drivable areas in the environment. The method additionally includes, based on the one or more predicted drivable areas including the one or more past trajectories, using the one or more updated obstacle detection heuristics to determine future navigation of the robotic device.

Abstract: Systems and methods are provided for improved generation and selection of robot sensor data for manual annotation and/or use in training machine learning models used to operate robots. An on-robot controller can operate to determine a cross-modal inconsistency, that a temporally proximate target task was failed, and/or that a confidence in a model output indicate that particular sensor data should be transmitted to a remote system for human annotation and/or use in updating the machine learning model(s) of the robot. Embedding vector(s) representing such selected sensor data (e.g., representing common aspects across a population of sets of sensor data) could also be determined and transmitted to the robot. The robot could then determine embeddings for sensor data and, if the embeddings are similar enough to the transmitted embedding(s), the sensor data could be transmitted to the remote system for annotation and/or model updating.

Abstract: Implementations set forth herein relate to generating training data, such that each instance of training data includes a corresponding instance of vision data and drivability label(s) for the instance of vision data. A drivability label can be determined using first vision data from a first vision component that is connected to the robot. The drivability label(s) can be generated by processing the first vision data using geometric and/or heuristic methods. Second vision data can be generated using a second vision component of the robot, such as a camera that is connected to the robot. The drivability labels can be correlated to the second vision data and thereafter used to train one or more machine learning models. The trained models can be shared with a robot(s) in furtherance of enabling the robot(s) to determine drivability of areas captured in vision data, which is being collected in real-time using one or more vision components.

Abstract: Implementations set forth herein relate to generating training data, such that each instance of training data includes a corresponding instance of vision data and drivability label(s) for the instance of vision data. A drivability label can be determined using first vision data from a first vision component that is connected to the robot. The drivability label(s) can be generated by processing the first vision data using geometric and/or heuristic methods. Second vision data can be generated using a second vision component of the robot, such as a camera that is connected to the robot. The drivability labels can be correlated to the second vision data and thereafter used to train one or more machine learning models. The trained models can be shared with a robot(s) in furtherance of enabling the robot(s) to determine drivability of areas captured in vision data, which is being collected in real-time using one or more vision components.

Abstract: A method includes determining, for a robotic device that comprises a perception system, a robot planner state representing at least one future path for the robotic device in an environment. The method also includes determining a perception system trajectory by inputting at least the robot planner state into a machine learning model trained based on training data comprising at least a plurality of robot planner states corresponding to a plurality of operator-directed perception system trajectories. The method further includes controlling, by the robotic device, the perception system to move through the determined perception system trajectory.

Abstract: A method includes receiving, from a sensor on a robotic device, a captured image representative of an environment of the robotic device when the robotic device is at a location in the environment. The method also includes determining, based at least on the location of the robotic device, a rendered image representative of the environment of the robotic device. The method further includes determining, by applying at least one pre-trained machine learning model to at least the captured image and the rendered image, a property of one or more portions of the captured image.

Abstract: Implementations set forth herein relate to generating training data, such that each instance of training data includes a corresponding instance of vision data and drivability label(s) for the instance of vision data. A drivability label can be determined using first vision data from a first vision component that is connected to the robot. The drivability label(s) can be generated by processing the first vision data using geometric and/or heuristic methods. Second vision data can be generated using a second vision component of the robot, such as a camera that is connected to the robot. The drivability labels can be correlated to the second vision data and thereafter used to train one or more machine learning models. The trained models can be shared with a robot(s) in furtherance of enabling the robot(s) to determine drivability of areas captured in vision data, which is being collected in real-time using one or more vision components.

Abstract: Implementations set forth herein relate to a robot that employs a stereo camera and LIDAR for generating point cloud data while the robot is traversing an area. The point cloud data can characterize spaces within the area as occupied, unoccupied, or uncategorized. For instance, an uncategorized space can refer to a point in three-dimensional (3D) space where occupancy of the space is unknown and/or where no observation has been made by the robot—such as in circumstances where a blind spot is located at or near a base of the robot. In order to efficiently traverse certain areas, the robot can estimate resource costs of either sweeping the stereo camera indiscriminately between spaces and/or specifically focusing the stereo camera on uncategorized space(s) during the route. Based on such resource cost estimations, the robot can adaptively maneuver the stereo camera during routes while also minimizing resource consumption by the robot.