Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

Abstract: Three dimensional object reconstruction for sensor simulation includes performing operations that include rendering, by a differential rendering engine, an object image from a target object model, and computing, by a loss function of the differential rendering engine, a loss based on a comparison of the object image with an actual image and a comparison of the target object model with a corresponding lidar point cloud. The operations further include updating the target object model by the differential rendering engine according to the loss, and rendering, after updating the target object model, a target object in a virtual world using the target object model.

Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

Abstract: Real world object reconstruction and representation include performing operations that include sampling locations along a camera ray from a virtual camera to a target object to obtain a sample set of the locations along the camera ray. For each location of the at least a subset of the sample set, the operations include determining a position of the location with respect to the target object, executing, based on the position, a reflectance multilayer perceptron (MLP) model, to determine an albedo and material shininess for the location, and computing a radiance for the location and based on a viewing direction of the camera ray using the albedo and the material shininess. The operations further includes rendering a color value for the camera ray by compositing the radiance across the first sample set.

Abstract: Systems and methods for vehicle-to-vehicle communications are provided. An example computer-implemented method includes obtaining, by a computing system onboard a first autonomous vehicle, sensor data associated with an environment of the first autonomous vehicle. The method includes determining, by the computing system, an intermediate environmental representation of at least a portion of the environment of the first autonomous vehicle based at least in part on the sensor data. The method includes generating, by the computing system, a compressed intermediate environmental representation by compressing the intermediate environmental representation of at least the portion of the environment of the first autonomous vehicle. The method includes communicating, by the computing system, the compressed intermediate environmental representation to a second autonomous vehicle.

Abstract: Systems and methods for vehicle-to-vehicle communications are provided. An example computer-implemented method includes obtaining from a first autonomous vehicle, by a computing system onboard a second autonomous vehicle, a first compressed intermediate environmental representation. The first compressed intermediate environmental representation is indicative of at least a portion of an environment of the second autonomous vehicle and is based at least in part on sensor data acquired by the first autonomous vehicle at a first time. The method includes generating, by the computing system, a first decompressed intermediate environmental representation by decompressing the first compressed intermediate environmental representation. The method includes determining, by the computing system, a first time-corrected intermediate environmental representation based at least in part on the first decompressed intermediate environmental representation.

Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

Abstract: Systems and methods for vehicle-to-vehicle communications are provided. An example computer-implemented method includes obtaining from a first autonomous vehicle, by a second autonomous vehicle, a first compressed intermediate environmental representation. The first compressed intermediate environmental representation is indicative of at least a portion of an environment of the second autonomous vehicle. The method includes generating a first decompressed intermediate environmental representation by decompressing the first compressed intermediate environmental representation. The method includes determining, using one or more machine-learned models, an updated intermediate environmental representation based at least in part on the first decompressed intermediate environmental representation and a second intermediate environmental representation generated by the second autonomous vehicle.

Abstract: Systems and methods for training object detection models using adversarial examples are provided. A method includes obtaining a training scene and identifying a target object within the training scene. The method includes obtaining an adversarial object and generating a modified training scene based on the adversarial object, the target object, and the training scene. The modified training scene includes the training scene modified to include the adversarial object placed on the target object. The modified training scene is input to a machine-learned model configured to detect the training object. A detection score is determined based on whether the training object is detected, and the machine-learned model and the parameters of the adversarial object are trained based on the detection output. The machine-learned model is trained to maximize the detection output. The parameters of the adversarial object are trained to minimize the detection output.

Abstract: Example aspects of the present disclosure describe a scene generator for simulating scenes in an environment. For example, snapshots of simulated traffic scenes can be generated by sampling a joint probability distribution trained on real-world traffic scenes. In some implementations, samples of the joint probability distribution can be obtained by sampling a plurality of factorized probability distributions for a plurality of objects for sequential insertion into the scene.

Abstract: Example aspects of the present disclosure describe a scene generator for simulating scenes in an environment. For example, snapshots of simulated traffic scenes can be generated by sampling a joint probability distribution trained on real-world traffic scenes. In some implementations, samples of the joint probability distribution can be obtained by sampling a plurality of factorized probability distributions for a plurality of objects for sequential insertion into the scene.

Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

Abstract: The present disclosure provides systems and methods for generating photorealistic image simulation data with geometry-aware composition for testing autonomous vehicles. In particular, aspects of the present disclosure can involve the intake of data on an environment and output of augmented data on the environment with the photorealistic addition of an object. As one example, data on the driving experiences of a self-driving vehicle can be augmented to add another vehicle into the collected environment data. The augmented data may then be used to test safety features of software for a self-driving vehicle.

Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

Abstract: The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

Abstract: The present disclosure provides systems and methods for generating photorealistic image simulation data with geometry-aware composition for testing autonomous vehicles. In particular, aspects of the present disclosure can involve the intake of data on an environment and output of augmented data on the environment with the photorealistic addition of an object. As one example, data on the driving experiences of a self-driving vehicle can be augmented to add another vehicle into the collected environment data. The augmented data may then be used to test safety features of software for a self-driving vehicle.

Abstract: Example aspects of the present disclosure describe a scene generator for simulating scenes in an environment. For example, snapshots of simulated traffic scenes can be generated by sampling a joint probability distribution trained on real-world traffic scenes. In some implementations, samples of the joint probability distribution can be obtained by sampling a plurality of factorized probability distributions for a plurality of objects for sequential insertion into the scene.

Abstract: Techniques for generating testing data for an autonomous vehicle (AV) are described herein. A system can obtain sensor data descriptive of a traffic scenario. The traffic scenario can include a subject vehicle and actors in an environment. Additionally, the system can generate a perturbed trajectory for a first actor in the environment based on perturbation values. Moreover, the system can generate simulated sensor data. The simulated sensor data can include data descriptive of the perturbed trajectory for the first actor in the environment. Furthermore, the system can provide the simulated sensor data as input to an AV control system. The AV control system can be configured to process the simulated sensor data to generate an updated trajectory for the subject vehicle in the environment. Subsequently, the system can evaluate an adversarial loss function based on the updated trajectory for the subject vehicle to generate an adversarial loss value.