Abstract: Techniques for updating data operations in a perception system are discussed herein. A vehicle may use a perception system to capture data about an environment proximate to the vehicle. The perception system may receive image data, lidar data, and/or radar data to determine information about an object in the environment. As different sensors may be associated with different time periods for capturing and/or processing operations, the techniques include updating object data with data from sensors associated with a shorter time period to generate intermediate object data.

Abstract: A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest (“ROI”) in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.

Abstract: A machine-learning architecture may be trained to determine point cloud data associated with different types of sensors with an object detected in an image and/or generate a three-dimensional region of interest (ROI) associated with the object. In some examples, the point cloud data may be associated with sensors such as, for example, a lidar device, radar device, etc.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for rendering a new image that depicts a scene from a perspective of a camera at a new camera viewpoint.

Abstract: Ground truth data may be too sparse to supervise training of a machine-learned (ML) model enough to achieve an ML model with sufficient accuracy/recall. For example, in some cases, ground truth data may only be available for every third, tenth, or hundredth frame of raw data. Training an ML model to detect a velocity of an object when ground truth data for training is sparse may comprise training the ML model to predict a future position of the object based at least in part on image, radar, and/or lidar data (e.g., for which no ground truth may be available). The ML model may be altered based at least in part on a difference between ground truth data associated with a future time and the future position.

Abstract: A machine-learning architecture may be trained to determine point cloud data associated with different types of sensors with an object detected in an image and/or generate a three-dimensional region of interest (ROI) associated with the object. In some examples, the point cloud data may be associated with sensors such as, for example, a lidar device, radar device, etc.

Abstract: A machine-learning architecture may be trained to determine point cloud data associated with different types of sensors with an object detected in an image and/or generate a three-dimensional region of interest (ROI) associated with the object. In some examples, the point cloud data may be associated with sensors such as, for example, a lidar device, radar device, etc.

Abstract: A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest (“ROI”) in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.

Abstract: A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest (“ROI”) in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.

Abstract: A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest (“ROI”) in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.