Abstract: Systems and methods for generating motion plans for autonomous vehicles are provided. An autonomous vehicle can include a machine-learned motion planning system including one or more machine-learned models configured to generate target trajectories for the autonomous vehicle. The model(s) include a behavioral planning stage configured to receive situational data based at least in part on the one or more outputs of the set of sensors and to generate behavioral planning data based at least in part on the situational data and a unified cost function. The model(s) includes a trajectory planning stage configured to receive the behavioral planning data from the behavioral planning stage and to generate target trajectory data for the autonomous vehicle based at least in part on the behavioral planning data and the unified cost function.

Abstract: The present disclosure provides systems and methods for generating photorealistic image simulation data with geometry-aware composition for testing autonomous vehicles. In particular, aspects of the present disclosure can involve the intake of data on an environment and output of augmented data on the environment with the photorealistic addition of an object. As one example, data on the driving experiences of a self-driving vehicle can be augmented to add another vehicle into the collected environment data. The augmented data may then be used to test safety features of software for a self-driving vehicle.

Abstract: Systems and methods for generating motion plans for autonomous vehicles are provided. An autonomous vehicle can include a machine-learned motion planning system including one or more machine-learned models configured to generate target trajectories for the autonomous vehicle. The model(s) include a behavioral planning stage configured to receive situational data based at least in part on the one or more outputs of the set of sensors and to generate behavioral planning data based at least in part on the situational data and a unified cost function. The model(s) includes a trajectory planning stage configured to receive the behavioral planning data from the behavioral planning stage and to generate target trajectory data for the autonomous vehicle based at least in part on the behavioral planning data and the unified cost function.

Abstract: Systems and methods for generating motion plans for autonomous vehicles are provided. An autonomous vehicle can include a machine-learned motion planning system including one or more machine-learned models configured to generate target trajectories for the autonomous vehicle. The model(s) include a behavioral planning stage configured to receive situational data based at least in part on the one or more outputs of the set of sensors and to generate behavioral planning data based at least in part on the situational data and a unified cost function. The model(s) includes a trajectory planning stage configured to receive the behavioral planning data from the behavioral planning stage and to generate target trajectory data for the autonomous vehicle based at least in part on the behavioral planning data and the unified cost function.

Abstract: Methods and systems are disclosed for correlating synthetic LiDAR data to a real-world domain for use in training an model for use by autonomous vehicle when operating in an environment. To do this, the system will obtain a data set of synthetic LiDAR data, along with images of a real-world environment. The system will transfer the synthetic LiDAR data to a two-dimensional representation, use the two-dimensional representation and the images to train a model that a vehicle can use to operate in a real-world environment.

Abstract: Systems and methods for answering region specific questions are provided. A method includes obtaining a regional scene question including an attribute query and a spatial region of interest for a training scene depicting a surrounding environment of a vehicle. The method includes obtaining a universal embedding for the training scene and an attribute embedding for the attribute query of the scene question. The universal embedding can identify sensory data corresponding to the training scene that can be used to answer questions concerning a number of different attributes in the training scene. The attribute embedding can identify aspects of an attribute that can be used to answer questions specific to the attribute. The method includes determining an answer embedding based on the universal embedding and the attribute embedding and determining a regional scene answer to the regional scene question based on the spatial region of interest and the answer embedding.

Abstract: The present disclosure provides systems and methods for generating photorealistic image simulation data with geometry-aware composition for testing autonomous vehicles. In particular, aspects of the present disclosure can involve the intake of data on an environment and output of augmented data on the environment with the photorealistic addition of an object. As one example, data on the driving experiences of a self-driving vehicle can be augmented to add another vehicle into the collected environment data. The augmented data may then be used to test safety features of software for a self-driving vehicle.

Abstract: Systems and methods are provided for representing a traffic signal device. The method includes receiving a digital image of a traffic signal device that includes one or more traffic signal elements, representing the traffic signal device as a raster image, each traffic signal element of the traffic signal device being represented by a mask corresponding to a location of the traffic signal element on the traffic signal device, representing each mask in a channel in the raster image, providing the raster image as an input to a neural network to classify a state for each of the one or more traffic signal elements, and receiving, from the neural network, a classified raster image, in which the classified raster image includes a plurality of masks, each mask representing a state of one of the one or more traffic signal elements.

Abstract: The present disclosure provides systems and methods for generating photorealistic image simulation data with geometry-aware composition for testing autonomous vehicles. In particular, aspects of the present disclosure can involve the intake of data on an environment and output of augmented data on the environment with the photorealistic addition of an object. As one example, data on the driving experiences of a self-driving vehicle can be augmented to add another vehicle into the collected environment data. The augmented data may then be used to test safety features of software for a self-driving vehicle.

Abstract: The present disclosure provides systems and methods for generating photorealistic image simulation data with geometry-aware composition for testing autonomous vehicles. In particular, aspects of the present disclosure can involve the intake of data on an environment and output of augmented data on the environment with the photorealistic addition of an object. As one example, data on the driving experiences of a self-driving vehicle can be augmented to add another vehicle into the collected environment data. The augmented data may then be used to test safety features of software for a self-driving vehicle.

Abstract: Systems and methods are provided for representing a traffic signal device. The method includes receiving a digital image of a traffic signal device that includes one or more traffic signal elements, representing the traffic signal device as a raster image, each traffic signal element of the traffic signal device being represented by a mask corresponding to a location of the traffic signal element on the traffic signal device, representing each mask in a channel in the raster image, providing the raster image as an input to a neural network to classify a state for each of the one or more traffic signal elements, and receiving, from the neural network, a classified raster image, in which the classified raster image includes a plurality of masks, each mask representing a state of one of the one or more traffic signal elements.

Abstract: Methods and systems are disclosed for correlating synthetic LiDAR data to a real-world domain for use in training an model for use by autonomous vehicle when operating in an environment. To do this, the system will obtain a data set of synthetic LiDAR data, along with images of a real-world environment. The system will transfer the synthetic LiDAR data to a two-dimensional representation, use the two-dimensional representation and the images to train a model that a vehicle can use to operate in a real-world environment.

Abstract: Systems and methods for generating motion plans for autonomous vehicles are provided. An autonomous vehicle can include a machine-learned motion planning system including one or more machine-learned models configured to generate target trajectories for the autonomous vehicle. The model(s) include a behavioral planning stage configured to receive situational data based at least in part on the one or more outputs of the set of sensors and to generate behavioral planning data based at least in part on the situational data and a unified cost function. The model(s) includes a trajectory planning stage configured to receive the behavioral planning data from the behavioral planning stage and to generate target trajectory data for the autonomous vehicle based at least in part on the behavioral planning data and the unified cost function.

Abstract: Methods and systems are disclosed for correlating synthetic LiDAR data to a real-world domain for use in training an autonomous vehicle in how to operate in an environment. To do this, the system will obtain a data set of synthetic LiDAR data, transfer the synthetic LiDAR data to a two-dimensional representation, use the two-dimensional representation to train a model of a real-world environment, and use the trained model of the real-world environment to train an autonomous vehicle.

Abstract: Systems and methods for answering region specific questions are provided. A method includes obtaining a regional scene question including an attribute query and a spatial region of interest for a training scene depicting a surrounding environment of a vehicle. The method includes obtaining a universal embedding for the training scene and an attribute embedding for the attribute query of the scene question. The universal embedding can identify sensory data corresponding to the training scene that can be used to answer questions concerning a number of different attributes in the training scene. The attribute embedding can identify aspects of an attribute that can be used to answer questions specific to the attribute. The method includes determining an answer embedding based on the universal embedding and the attribute embedding and determining a regional scene answer to the regional scene question based on the spatial region of interest and the answer embedding.

Abstract: Methods and systems are disclosed for correlating synthetic LiDAR data to a real-world domain for use in training an autonomous vehicle in how to operate in an environment. To do this, the system will obtain a data set of synthetic LiDAR data, transfer the synthetic LiDAR data to a two-dimensional representation, use the two-dimensional representation to train a model of a real-world environment, and use the trained model of the real-world environment to train an autonomous vehicle.