Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for predicting future trajectories for an agent in an environment. A system obtains scene context data characterizing the environment. The scene context data includes data that characterizes a trajectory of an agent in a vicinity of a vehicle in an. environment up to a current time point. The system identifies a plurality of initial target locations in the environment. The system further generates, for each of a plurality of target locations that each corresponds to one of the initial target locations, a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent starting from the current time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a machine learning model on training data. In one aspect, one of the methods include: obtaining a training data set comprising a plurality of training inputs; obtaining data defining an original search space of a plurality of candidate data augmentation policies; generating, from the original search space, a compact search space that has one or more global hyperparameters; and training the machine learning model on the training data using one or more final data augmentation policies generated from the compact search space.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for estimating a 3-D pose of an object of interest from image and point cloud data. In one aspect, a method includes obtaining an image of an environment; obtaining a point cloud of a three-dimensional region of the environment; generating a fused representation of the image and the point cloud; and processing the fused representation using a pose estimation neural network and in accordance with current values of a plurality of pose estimation network parameters to generate a pose estimation network output that specifies, for each of multiple keypoints, a respective estimated position in the three-dimensional region of the environment.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for processing a perspective view range image generated from sensor measurements of an environment. The perspective view range image includes a plurality of pixels arranged in a two-dimensional grid and including, for each pixel, (i) features of one or more sensor measurements at a location in the environment corresponding to the pixel and (ii) geometry information comprising range features characterizing a range of the location in the environment corresponding to the pixel relative to the one or more sensors. The system processes the perspective view range image using a first neural network to generate an output feature representation. The first neural network comprises a first perspective point-set aggregation layer comprising a geometry-dependent kernel.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a semantic segmentation neural network for point clouds. One of the methods includes: obtaining a plurality of training points divided into a respective plurality of components; obtaining, for each of the respective plurality of components, data identifying a ground truth category for one or more labeled point; processing each training points using a semantic segmentation neural network to generate a semantic segmentation that includes a respective score for each of the plurality of categories; determining a gradient of a loss function that penalizes the semantic segmentation neural network for generating, for points in the component, non-zero scores for categories that are not the ground truth category for any labeled point in the component; and updating, using the gradient, the parameters of the semantic segmentation neural network.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for performing a conditional behavior prediction for one or more agents. The system obtains context data characterizing an environment. The context data includes data characterizing a plurality of agents, including a query agent and one or more target agents, in the environment at a current time point. The system further obtains data identifying a planned future trajectory for the query agent after the current time point, and for each target agent in the set, processes the context data and the data identifying the planned future trajectory using a first neural network to generate a conditional trajectory prediction output that defines a conditional probability distribution over possible future trajectories of the target agent after the current time point given that the query agent follows the planned future trajectory for the query agent after the current time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating realistic full-scene point clouds. One of the methods includes obtaining an initial scene point cloud characterizing an initial scene in an environment; obtaining, for each of one or more objects, an object point cloud that characterizes the object; and processing a first input comprising the initial scene point cloud and the one or more object point clouds using a first neural network that is configured to process the first input to generate a final scene point cloud that characterizes a transformed scene that has the one or more objects added to the initial scene.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for the generation and use of a surfel map with semantic labels. One of the methods includes receiving a surfel map that includes a plurality of surfels, wherein each surfel has associated data that includes one or more semantic labels; obtaining sensor data for one or more locations in the environment, the sensor data having been captured by one or more sensors of a first vehicle; determining one or more surfels corresponding to the one or more locations of the obtained sensor data; identifying one or more semantic labels for the one or more surfels corresponding to the one or more locations of the obtained sensor data; and performing, for each surfel corresponding to the one or more locations of the obtained sensor data, a label-specific detection process for the surfel.

Abstract: Methods, systems, and apparatus for generating trajectory predictions for one or more agents. In one aspect, a system comprises one or more computers configured to obtain scene context data characterizing a scene in an environment at a current time point, where the scene includes multiple agents. The one or more computers process the scene context data using a marginal trajectory prediction neural network to generate a respective marginal trajectory prediction for each of the plurality of agents that defines multiple possible trajectories for the agent after the current time point and a respective likelihood score for each of the multiple possible future trajectories. The one or more computers can generate graph data based on the respective marginal trajectory predictions, and the one or more computers can process the graph data using a graph neural network to generate a joint trajectory prediction output for the multiple agents in the scene.

Abstract: Methods, systems, and apparatus for processing point clouds using neural networks to perform a machine learning task. In one aspect, a system comprises one or more computers configured to obtain a set of point clouds captured by one or more sensors. Each point cloud includes a respective plurality of three-dimensional points. The one or more computers assign the three-dimensional points to respective voxels in a voxel grid, where the grid of voxels includes non-empty voxels to which one or more points are assigned and empty voxels to which no points are assigned. For each non-empty voxel, the one or more computers generate initial features based on the points that are assigned to the non-empty voxel. The one or more computers generate multi-scale features of the voxel grid, and the one or more computers generate an output for a point cloud processing task using the multi-scale features of the voxel grid.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting occupancies of agents. One of the methods includes obtaining scene data characterizing a current scene in an environment; and processing a neural network input comprising the scene data using a neural network to generate a neural network output, wherein: the neural network output comprises respective occupancy outputs corresponding to a plurality of agent types at one or more future time points; the occupancy output for each agent type at a first future time point comprises respective occupancy probabilities for a plurality of locations in the environment; and in the occupancy output for each agent type at the first future time point, the respective occupancy probability for each location characterizes a likelihood that an agent of the agent type will occupy the location at the first future time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting occupancies of agents. One of the methods includes obtaining scene data characterizing a current scene in an environment; and processing a neural network input comprising the scene data using a neural network to generate a neural network output, wherein: the neural network output comprises respective occupancy outputs corresponding to a plurality of agent types at one or more future time points; the occupancy output for each agent type at a first future time point comprises respective occupancy probabilities for a plurality of locations in the environment; and in the occupancy output for each agent type at the first future time point, the respective occupancy probability for each location characterizes a likelihood that an agent of the agent type will occupy the location at the first future time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for processing sensor data. In one aspect, a method includes obtaining image data representing a camera sensor measurement of a scene; obtaining radar data representing a radar sensor measurement of the scene; generating a feature representation of the image data; generating a respective initial depth estimate for each of a subset of the plurality of pixels; generating a feature representation of the radar data; for each of the subset of the plurality of pixels, generating a respective adjusted depth estimate for the pixel using the initial depth estimate for the pixel and the radar feature vectors for a corresponding subset of the plurality of radar reflection points; generating a fused point cloud that includes a plurality of three-dimensional data points; and processing the fused point cloud to generate an output that characterizes the scene.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing point cloud data using dynamic voxelization. When deployed within an on-board system of a vehicle, processing the point cloud data using dynamic voxelization can be used to make autonomous driving decisions for the vehicle with enhanced accuracy, for example by combining representations of point cloud data characterizing a scene from multiple views of the scene.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing data compression and data decompression on lidar range images.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing point cloud data using spatio-temporal-interactive networks.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating simulated trajectories using parallel beam search.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for the generation and use of a surfel map with semantic labels. One of the methods includes receiving a surfel map that includes a plurality of surfels, wherein each surfel has associated data that includes one or more semantic labels; obtaining sensor data for one or more locations in the environment, the sensor data having been captured by one or more sensors of a first vehicle; determining one or more surfels corresponding to the one or more locations of the obtained sensor data; identifying one or more semantic labels for the one or more surfels corresponding to the one or more locations of the obtained sensor data; and performing, for each surfel corresponding to the one or more locations of the obtained sensor data, a label-specific detection process for the surfel.