Abstract: A machine-learned image compression model includes a first encoder configured to generate a first image code based at least in part on first image data. The first encoder includes a first series of convolutional layers configured to generate a first series of respective feature maps based at least in part on the first image. A second encoder is configured to generate a second image code based at least in part on second image data and includes a second series of convolutional layers configured to generate a second series of respective feature maps based at least in part on the second image and disparity-warped feature data. Respective parametric skip functions associated convolutional layers of the second series are configured to generate disparity-warped feature data based at least in part on disparity associated with the first series of respective feature maps and the second series of respective feature maps.

Abstract: The present disclosure is directed to video compression using conditional entropy coding. An ordered sequence of image frames can be transformed to produce an entropy coding for each image frame. Each of the entropy codings provide a compressed form of image information based on a prior image frame and a current image frame (the current image frame occurring after the prior image frame). In this manner, the compression model can capture temporal relationships between image frames or encoded representations of the image frames using a conditional entropy encoder trained to approximate the joint entropy between frames in the image frame sequence.

Abstract: A computing system can input first relative location embedding data into an interaction transformer model and receive, as an output of the interaction transformer model, motion forecast data for actors relative to a vehicle. The computing system can input the motion forecast data into a prediction model to receive respective trajectories for the actors for a current time step and respective projected trajectories for the actors for a subsequent time step. The computing system can generate second relative location embedding data based on the respective projected trajectories from the second time step. The computing system can produce second motion forecast data using the interaction transformer model based on the second relative location embedding. The computing system can determine second respective trajectories for the actors using the prediction model based on the second forecast data.

Abstract: Disclosed herein are methods and systems for performing instance segmentation that can provide improved estimation of object boundaries. Implementations can include a machine-learned segmentation model trained to estimate an initial object boundary based on a truncated signed distance function (TSDF) generated by the model. The model can also generate outputs for optimizing the TSDF over a series of iterations to produce a final TSDF that can be used to determine the segmentation mask.

Abstract: Systems and methods for vehicle-to-vehicle communications are provided. An example computer-implemented method includes obtaining, by a computing system onboard a first autonomous vehicle, sensor data associated with an environment of the first autonomous vehicle. The method includes determining, by the computing system, an intermediate environmental representation of at least a portion of the environment of the first autonomous vehicle based at least in part on the sensor data. The method includes generating, by the computing system, a compressed intermediate environmental representation by compressing the intermediate environmental representation of at least the portion of the environment of the first autonomous vehicle. The method includes communicating, by the computing system, the compressed intermediate environmental representation to a second autonomous vehicle.

Abstract: A computer-implemented method for localizing a vehicle can include accessing, by a computing system comprising one or more computing devices, a machine-learned retrieval model that has been trained using a ground truth dataset comprising a plurality of pre-localized sensor observations. Each of the plurality of pre-localized sensor observations has a predetermined pose value associated with a previously obtained sensor reading representation. The method also includes obtaining, by the computing system, a current sensor reading representation obtained by one or more sensors located at the vehicle. The method also includes inputting, by the computing system, the current sensor reading representation into the machine-learned retrieval model.

Abstract: The present disclosure is directed encoding LIDAR point cloud data. In particular, a computing system can receive point cloud data for a three-dimensional space. The computing system can generate a tree-based data structure from the point cloud data, the tree-based data structure comprising a plurality of nodes. The computing system can generate a serial representation of the tree-based data structure. The computing system can, for each respective node represented by a symbol in the serial representation: determine contextual information for the respective node, generate, using the contextual information as input to a machine-learned model, a statistical distribution associated with the respective node, and generate a compressed representation of the symbol associated with the respective node by encoding the symbol using the statistical distribution for the respective node.

Abstract: Systems and methods for vehicle-to-vehicle communications are provided. An example computer-implemented method includes obtaining from a first autonomous vehicle, by a second autonomous vehicle, a first compressed intermediate environmental representation. The first compressed intermediate environmental representation is indicative of at least a portion of an environment of the second autonomous vehicle. The method includes generating a first decompressed intermediate environmental representation by decompressing the first compressed intermediate environmental representation. The method includes determining, using one or more machine-learned models, an updated intermediate environmental representation based at least in part on the first decompressed intermediate environmental representation and a second intermediate environmental representation generated by the second autonomous vehicle.

Abstract: The present disclosure is directed to generating trajectories using a structured machine-learned model. In particular, a computing system can obtain sensor data for an area around an autonomous vehicle. The computing system can detect one or more objects based on the sensor data. The computing system can determine a plurality of candidate object trajectories for each object in the one or more objects. The computing system can generate, using the plurality of candidate object trajectories as input to one or more machine-learned models, likelihood data for the plurality of candidate object trajectories. The computing system can update the likelihood values for each of the plurality of candidate object trajectories for each respective object in the one or more objects based on the likelihood values associated with candidate object trajectories for other objects in the one or more objects. The computing system can determine a motion plan for the autonomous vehicle.

Abstract: Systems and methods for vehicle-to-vehicle communications are provided. An example computer-implemented method includes obtaining from a first autonomous vehicle, by a computing system onboard a second autonomous vehicle, a first compressed intermediate environmental representation. The first compressed intermediate environmental representation is indicative of at least a portion of an environment of the second autonomous vehicle and is based at least in part on sensor data acquired by the first autonomous vehicle at a first time. The method includes generating, by the computing system, a first decompressed intermediate environmental representation by decompressing the first compressed intermediate environmental representation. The method includes determining, by the computing system, a first time-corrected intermediate environmental representation based at least in part on the first decompressed intermediate environmental representation.

Abstract: Systems and methods for predicting instance geometry are provided. A method includes obtaining an input image depicting at least one object. The method includes determining an instance mask for the object by inputting the input image into a machine-learned instance segmentation model. The method includes determining an initial polygon with a number of initial vertices outlining the border of the object within the input image. The method includes obtaining a feature embedding for one or more pixels of the input image and determining a vertex embedding including a feature embedding for each pixel corresponding an initial vertex of the initial polygon. The method includes determining a vertex offset for each initial vertex of the initial polygon based on the vertex embedding and applying the vertex offset to the initial polygon to obtain one or more enhanced polygons.

Abstract: Systems and methods for answering region specific questions are provided. A method includes obtaining a regional scene question including an attribute query and a spatial region of interest for a training scene depicting a surrounding environment of a vehicle. The method includes obtaining a universal embedding for the training scene and an attribute embedding for the attribute query of the scene question. The universal embedding can identify sensory data corresponding to the training scene that can be used to answer questions concerning a number of different attributes in the training scene. The attribute embedding can identify aspects of an attribute that can be used to answer questions specific to the attribute. The method includes determining an answer embedding based on the universal embedding and the attribute embedding and determining a regional scene answer to the regional scene question based on the spatial region of interest and the answer embedding.

Abstract: Systems and methods for performing semantic segmentation of three-dimensional data are provided. In one example embodiment, a computing system can be configured to obtain sensor data including three-dimensional data associated with an environment. The three-dimensional data can include a plurality of points and can be associated with one or more times. The computing system can be configured to determine data indicative of a two-dimensional voxel representation associated with the environment based at least in part on the three-dimensional data. The computing system can be configured to determine a classification for each point of the plurality of points within the three-dimensional data based at least in part on the two-dimensional voxel representation associated with the environment and a machine-learned semantic segmentation model. The computing system can be configured to initiate one or more actions based at least in part on the per-point classifications.

Abstract: A computing system can be configured to input data that describes sensor data into an object detection model and receive, as an output of the object detection model, object detection data describing features of the plurality of the actors relative to the autonomous vehicle. The computing system can generate an input sequence that describes the object detection data. The computing system can analyze the input sequence using an interaction model to produce, as an output of the interaction model, an attention embedding with respect to the plurality of actors. The computing system can be configured to input the attention embedding into a recurrent model and determine respective trajectories for the plurality of actors based on motion forecast data received as an output of the recurrent model.

Abstract: Systems and methods of the present disclosure provide an improved approach for open-set instance segmentation by identifying both known and unknown instances in an environment. For example, a method can include receiving sensor point cloud input data including a plurality of three-dimensional points. The method can include determining a feature embedding and at least one of an instance embedding, class embedding, and/or background embedding for each of the plurality of three-dimensional points. The method can include determining a first subset of points associated with one or more known instances within the environment based on the class embedding and the background embedding associated with each point in the plurality of points. The method can include determining a second subset of points associated with one or more unknown instances within the environment based on the first subset of points. The method can include segmenting the input data into known and unknown instances.

Abstract: Systems and methods for generating motion forecast data for actors with respect to an autonomous vehicle and training a machine learned model for the same are disclosed. The computing system can include an object detection model and a graph neural network including a plurality of nodes and a plurality of edges. The computing system can be configured to input sensor data into the object detection model; receive object detection data describing the location of the plurality of the actors relative to the autonomous vehicle as an output of the object detection model; input the object detection data into the graph neural network; iteratively update a plurality of node states respectively associated with the plurality of nodes; and receive, as an output of the graph neural network, the motion forecast data with respect to the plurality of actors.

Abstract: Systems and methods for autonomous vehicle localization are provided. In one example embodiment, a computer-implemented method includes obtaining, by a computing system that includes one or more computing devices onboard an autonomous vehicle, sensor data indicative of one or more geographic cues within the surrounding environment of the autonomous vehicle. The method includes obtaining, by the computing system, sparse geographic data associated with the surrounding environment of the autonomous vehicle. The sparse geographic data is indicative of the one or more geographic cues. The method includes determining, by the computing system, a location of the autonomous vehicle within the surrounding environment based at least in part on the sensor data indicative of the one or more geographic cues and the sparse geographic data. The method includes outputting, by the computing system, data indicative of the location of the autonomous vehicle within the surrounding environment.

Abstract: Systems and methods for detecting objects are provided. In one example, a computer-implemented method includes receiving sensor data from one or more sensors configured to generate sensor data. The method includes inputting the sensor data to a machine-learned model that generates a class prediction and an instance prediction for each of a plurality of portions of the sensor data. The instance prediction includes an energy value based on a distance to at least one object boundary. The machine learned model can be trained to generate a common energy value to represent the at least one object boundary. The method includes generating as outputs of the machine-learned model, an instance prediction and a class prediction corresponding to each of the plurality of portions of the sensor data. The method includes generating one or more object segments based at least in part on the instance predictions and the class predictions.

Abstract: Systems and methods for facilitating communication with autonomous vehicles are provided. In one example embodiment, a computing system can obtain rasterized LIDAR data associated with a surrounding environment of an autonomous vehicle. The rasterized LIDAR data can include LIDAR image data that is rasterized from a LIDAR point cloud. The computing system can access data indicative of a machine-learned lane boundary detection model. The computing system can input the rasterized LIDAR data associated with the surrounding environment of the autonomous vehicle into the machine-learned lane boundary detection model. The computing system can obtain an output from the machine-learned lane boundary detection model. The output can be indicative of one or more lane boundaries within the surrounding environment of the autonomous vehicle.

Abstract: A machine-learned image compression model includes a first encoder configured to generate a first image code based at least in part on first image data. The first encoder includes a first series of convolutional layers configured to generate a first series of respective feature maps based at least in part on the first image. A second encoder is configured to generate a second image code based at least in part on second image data and includes a second series of convolutional layers configured to generate a second series of respective feature maps based at least in part on the second image and disparity-warped feature data. Respective parametric skip functions associated convolutional layers of the second series are configured to generate disparity-warped feature data based at least in part on disparity associated with the first series of respective feature maps and the second series of respective feature maps.