Abstract: This document describes scene understanding for cross reality systems using occupancy grids. In one aspect, a method includes recognizing one or more objects in a model of a physical environment generated using images of the physical environment. For each object, a bounding box is fit around the object. An occupancy grid that includes a multiple cells is generated within the bounding box around the object. A value is assigned to each cell of the occupancy grid based on whether the cell includes a portion of the object. An object representation that includes information describing the occupancy grid for the object is generated. The object representations are sent to one or more devices.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an object recognition neural network using multiple data sources. One of the methods includes receiving training data that includes a plurality of training images from a first source and images from a second source. A set of training images are obtained from the training data. For each training image in the set of training images, contrast equalization is applied to the training image to generate a modified image. The modified image is processed using the neural network to generate an object recognition output for the modified image. A loss is determined based on errors between, for each training image in the set, the object recognition output for the modified image generated from the training image and ground-truth annotation for the training image. Parameters of the neural network are updated based on the determined loss.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for scalable three-dimensional (3-D) object recognition in a cross reality system. One of the methods includes maintaining object data specifying objects that have been recognized in a scene. A stream of input images of the scene is received, including a stream of color images and a stream of depth images. A color image is provided as input to an object recognition system. A recognition output that identifies a respective object mask for each object in the color image is received. A synchronization system determines a corresponding depth image for the color image. A 3-D bounding box generation system determines a respective 3-D bounding box for each object that has been recognized in the color image. Data specifying one or more 3-D bounding boxes is received as output from the 3-D bounding box generation system.

Abstract: Disclosed are systems, apparatuses, methods, and computer-readable media to train a neural network model implemented into a perception stack in an autonomous vehicle (AV) for detecting objects. A method includes pretraining an uninitialized ML model to yield a first ML model; training the first ML model with a first testing dataset for a first number of iterations based on a first configuration; analyzing the first ML model based on a convergence of the first ML model and a previous iteration of training; generating a report based on the analysis of the first ML; and after generating the report, training the first ML model to yield a second ML model.

Abstract: The subject disclosure relates to techniques for tracking keypoints on an object represented in a Light Detection and Ranging (LiDAR) point cloud. A process of the disclosed technology can include receiving, for each of a plurality of frames in a series, an identification of at least one keypoint on the object represented in LiDAR point clouds and a confidence score for the respective keypoint, wherein each of the plurality of frames includes LiDAR point clouds including the object at different times represented in the series, and determining kinematics for the object from a determined movement of the keypoint across the plurality of frames.

Abstract: Disclosed are systems, apparatuses, methods, and computer-readable media to train a neural network model implemented into a perception stack in an autonomous vehicle (AV) for detecting objects. A method includes receiving a 3D light and detection ranging (LIDAR) data to train a neural network model having residual connections for detecting objects in LIDAR data; converting each frame of the LIDAR data into a voxelized frame to yield a training dataset of voxelized frames; and training the neural network model based on the training dataset of voxelized frames and a feedback control to control input from the training dataset of voxelized frames into the neural network model.

Abstract: The subject disclosure relates to techniques for identifying keypoints associated with an object based on LiDAR point cloud data. A process of the disclosed technology can include inputting the LiDAR point cloud data representing an object as perceived by a LiDAR sensor into an algorithm trained to identify the keypoints associated with the object, and identifying, by the algorithm, at least one keypoint associated with the object and a respective confidence score for the at least one keypoint.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for scalable three-dimensional (3-D) object recognition in a cross reality system. One of the methods includes maintaining object data specifying objects that have been recognized in a scene. A stream of input images of the scene is received, including a stream of color images and a stream of depth images. A color image is provided as input to an object recognition system. A recognition output that identifies a respective object mask for each object in the color image is received. A synchronization system determines a corresponding depth image for the color image. A 3-D bounding box generation system determines a respective 3-D bounding box for each object that has been recognized in the color image. Data specifying one or more 3-D bounding boxes is received as output from the 3-D bounding box generation system.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for scalable three-dimensional (3-D) object recognition in a cross reality system. One of the methods includes maintaining object data specifying objects that have been recognized in a scene. A stream of input images of the scene is received, including a stream of color images and a stream of depth images. A color image is provided as input to an object recognition system. A recognition output that identifies a respective object mask for each object in the color image is received. A synchronization system determines a corresponding depth image for the color image. A 3-D bounding box generation system determines a respective 3-D bounding box for each object that has been recognized in the color image. Data specifying one or more 3-D bounding boxes is received as output from the 3-D bounding box generation system.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for object recognition neural network for amodal center prediction. One of the methods includes receiving an image of an object captured by a camera. The image of the object is processed using an object recognition neural network that is configured to generate an object recognition output. The object recognition output includes data defining a predicted two-dimensional amodal center of the object, wherein the predicted two-dimensional amodal center of the object is a projection of a predicted three-dimensional center of the object under a camera pose of the camera that captured the image.