Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for the generation and use of a surfel map with semantic labels. One of the methods includes receiving a surfel map that includes a plurality of surfels, wherein each surfel has associated data that includes one or more semantic labels; obtaining sensor data for one or more locations in the environment, the sensor data having been captured by one or more sensors of a first vehicle; determining one or more surfels corresponding to the one or more locations of the obtained sensor data; identifying one or more semantic labels for the one or more surfels corresponding to the one or more locations of the obtained sensor data; and performing, for each surfel corresponding to the one or more locations of the obtained sensor data, a label-specific detection process for the surfel.

Abstract: Aspects and implementations of the present disclosure address shortcomings of existing technology by enabling autonomous vehicle simulations based on retro-reflection optical data. The subject matter of this specification can be implemented in, among other things, a method that involves initiating a simulation of an environment of an autonomous driving vehicle, the simulation including a plurality of simulated objects, each having an identification of a material type of the respective object. The method can further involve accessing simulated reflection data based on the plurality of simulated objects and retro-reflectivity data for the material types of the simulated objects, and determining, using an autonomous vehicle control system for the autonomous vehicle, a driving path relative to the simulated objects, the driving path based on the simulated reflection data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining an alignment between cross-modal sensor data. In one aspect, a method comprises: obtaining (i) an image that characterizes a visual appearance of an environment, and (ii) a point cloud comprising a collection of data points that characterizes a three-dimensional geometry of the environment; processing each of a plurality of regions of the image using a visual embedding neural network to generate a respective embedding of each of the image regions; processing each of a plurality of regions of the point cloud using a shape embedding neural network to generate a respective embedding of each of the point cloud regions; and identifying a plurality of region pairs using the embeddings of the image regions and the embeddings of the point cloud regions.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining an alignment between cross-modal sensor data. In one aspect, a method comprises: obtaining (i) an image that characterizes a visual appearance of an environment, and (ii) a point cloud comprising a collection of data points that characterizes a three-dimensional geometry of the environment; processing each of a plurality of regions of the image using a visual embedding neural network to generate a respective embedding of each of the image regions; processing each of a plurality of regions of the point cloud using a shape embedding neural network to generate a respective embedding of each of the point cloud regions; and identifying a plurality of region pairs using the embeddings of the image regions and the embeddings of the point cloud regions.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining that prediction outputs generated by a prediction system are sensitive to variations in the values of one or more of a set of target sensor calibration parameters. In one aspect, a sensitivity analysis system is configured to perform operations comprising updating the values of one or more target sensor calibration parameters of each sensor data tuple of a plurality sensor data tuples, comprising, for each sensor data tuple: providing the sensor data tuple with the current values of the target sensor calibration parameters to the prediction subsystem to generate a current prediction output; determining a gradient of a function of the current prediction output with respect to the target sensor calibration parameters; and updating the current values of the target sensor calibration parameters of the sensor data tuple using the gradient.