Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generated simulated sensor data. One of the methods includes obtaining a surfel map generated from sensor observations of a real-world environment and generating, for each surfel in the surfel map, a respective grid having a plurality of grid cells, wherein each grid has an orientation matching an orientation of a corresponding surfel, and wherein each grid cell within each grid is assigned a respective color value. For a simulated location within a simulated representation of the real-world environment, a textured surfel rendering is generated, including combining color information from grid cells visible from the simulated location within the simulated representation of the real-world environment.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for performing object detection. The system obtains a respective range image corresponding to each point cloud in a set of point clouds captured by one or more sensors. The system processes each range image using a segmentation neural network to generate range image features and a segmentation output. The system generates a feature representation of the set of point clouds from only the feature representations of the foreground points. The system processes the feature representation of the set of point clouds using a prediction neural network to generate a prediction characterizing the set of point clouds.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Abstract: The present disclosure provides solutions for generating synthetic three-dimensional (3D) objects for testing autonomous vehicle (AV) object detectors, such as AV Light Detection and Ranging (LiDAR) detectors. A process of the disclosed technology can include steps for receiving a first set of object shape data corresponding to a baseline object, permuting the first set of object shape data to generate a second set of object shape data, and providing the second set of object shape data to a differentiable pipeline to generate an output scene, wherein the output scene contains an optimized object that is based on the second set of object shape data. The process can further include steps for determining, using a detector, if the optimized object can be identified in the output scene. Systems and machine-readable media are also provided.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for predicting future trajectories for an agent in an environment. A system obtains scene context data characterizing the environment. The scene context data includes data that characterizes a trajectory of an agent in a vicinity of a vehicle in an environment up to a current time point. The system identifies a plurality of initial target locations in the environment. The system further generates, for each of a plurality of target locations that each corresponds to one of the initial target locations, a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent starting from the current time point.

Abstract: A system obtains scene context data characterizing the environment. The scene context data includes data that characterizes a trajectory of an agent in a vicinity of a vehicle up to a current time point. The system identifies a plurality of initial target locations, and generates, for each of a plurality of target locations that each corresponds to one of the initial target locations, a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent. For each target location in a first subset of the target locations, the system generates a predicted future trajectory for the agent given that the target location is the intended final location for the future trajectory. The system further selects, as likely future trajectories of the agent, one or more of the predicted future trajectories.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for predicting future trajectories for an agent in an environment. A system obtains scene context data characterizing the environment. The scene context data includes data that characterizes a trajectory of an agent in a vicinity of a vehicle in an. environment up to a current time point. The system identifies a plurality of initial target locations in the environment. The system further generates, for each of a plurality of target locations that each corresponds to one of the initial target locations, a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent starting from the current time point.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for processing a perspective view range image generated from sensor measurements of an environment. The perspective view range image includes a plurality of pixels arranged in a two-dimensional grid and including, for each pixel, (i) features of one or more sensor measurements at a location in the environment corresponding to the pixel and (ii) geometry information comprising range features characterizing a range of the location in the environment corresponding to the pixel relative to the one or more sensors. The system processes the perspective view range image using a first neural network to generate an output feature representation. The first neural network comprises a first perspective point-set aggregation layer comprising a geometry-dependent kernel.

Abstract: The disclosed technology provides solutions for generating synthetic driving scenes and in particular, for generating driving scenes with reduced safety metrics for use in testing and/or training various systems of an autonomous vehicle (AV). A method of the disclosed technology can include steps for receiving, at an encoding model, driving data representative of a first driving scene, wherein the encoding model is configured to generate a first set of feature vectors based on the driving data, processing the first set of feature vectors to generate a second set of feature vectors, and processing the second set of feature vectors to generate a second driving scene. Systems and machine-readable media are also provided.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting the future movement of agents in an environment. In particular, the future movement is predicted through occupancy flow fields that specify, for each future time point in a sequence of future time points and for each agent type in a set of one or more agent types: an occupancy prediction for the future time step that specifies, for each grid cell, an occupancy likelihood that any agent of the agent type will occupy the grid cell at the future time point, and a motion flow prediction that specifies, for each grid cell, a motion vector that represents predicted motion of agents of the agent type within the grid cell at the future time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing point cloud data representing a sensor measurement of a scene captured by one or more sensors to generate an object detection output that identifies locations of one or more objects in the scene. When deployed within an on-board system of a vehicle, the object detection output that is generated can be used to make autonomous driving decisions for the vehicle with enhanced accuracy.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for performing object detection. The system obtains a respective range image corresponding to each point cloud in a set of point clouds captured by one or more sensors. The system processes each range image using a segmentation neural network to generate range image features and a segmentation output. The system generates a feature representation of the set of point clouds from only the feature representations of the foreground points. The system processes the feature representation of the set of point clouds using a prediction neural network to generate a prediction characterizing the set of point clouds.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for processing a perspective view range image generated from sensor measurements of an environment. The perspective view range image includes a plurality of pixels arranged in a two-dimensional grid and including, for each pixel, (i) features of one or more sensor measurements at a location in the environment corresponding to the pixel and (ii) geometry information comprising range features characterizing a range of the location in the environment corresponding to the pixel relative to the one or more sensors. The system processes the perspective view range image using a first neural network to generate an output feature representation. The first neural network comprises a first perspective point-set aggregation layer comprising a geometry-dependent kernel.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for training a perspective computer vision model. The model is configured to receive input data characterizing an input scene in an environment from an input viewpoint and to process the input data in accordance with a set of model parameters to generate an output perspective representation of the scene from the input viewpoint. The system trains the model based on first data characterizing a scene in the environment from a first viewpoint and second data characterizing the scene in the environment from a second, different viewpoint.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generated simulated sensor data. One of the methods includes obtaining a surfel map generated from sensor observations of a real-world environment and generating, for each surfel in the surfel map, a respective grid having a plurality of grid cells, wherein each grid has an orientation matching an orientation of a corresponding surfel, and wherein each grid cell within each grid is assigned a respective color value. For a simulated location within a simulated representation of the real-world environment, a textured surfel rendering is generated, including combining color information from grid cells visible from the simulated location within the simulated representation of the real-world environment.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for multi object tracking using memory attention.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing point cloud data representing a sensor measurement of a scene captured by one or more sensors to generate an object detection output that identifies locations of one or more objects in the scene. When deployed within an on-board system of a vehicle, the object detection output that is generated can be used to make autonomous driving decisions for the vehicle with enhanced accuracy.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.