Abstract: Provided are methods for motion prediction in an autonomous vehicle using fused synthetic and camera images. The method can include obtaining data pairs, each of which reflects data corresponding to a synthetic image representing a birds-eye-view of an area around a vehicle and identifying an object, and data corresponding to a camera image depicting the object. A machine learning model can be trained based on the data pairs to result in a trained model that predicts motion of the object within the data pair based on the synthetic image and camera image in the data pair. Systems and computer program products are also provided.

Abstract: Enclosed are embodiments for scoring one or more trajectories of a vehicle through a given traffic scenario using a machine learning model that predicts reasonableness scores for the trajectories. In an embodiment, human annotators, referred to as a “reasonable crowd,” are presented with renderings of two or more vehicle trajectories traversing through the same or different traffic scenarios. The annotators are asked to indicate their preference for one trajectory over the other(s). Inputs collected from the human annotators are used to train the machine learning model to predict reasonableness scores for one or more trajectories for a given traffic scenario. These predicted trajectories can be used to rank trajectories generated by a route planner based on their scores, compare AV software stacks, or used by any other application that could benefit from a machine learning model that scores vehicle trajectories.

Abstract: Techniques are provided for improving a perception processing pipeline for object detection that fuses image segmentation data (e.g., segmentation scores) with LiDAR points. The disclosed techniques are implemented using an architecture that accepts point clouds and images as input and estimates oriented 3D bounding boxes for all relevant object classes. In an embodiment, a method comprises: matching temporally, using one or more processors of a vehicle, points in a three-dimensional (3D) point cloud with an image; generating, using an image-based neural network, semantic data for the image; decorating, using the one or more processors, the points in the 3D point cloud with the semantic data; and estimating, using a 3D object detector with the decorated points as input, oriented 3D bounding boxes for the one or more objects.

Abstract: Enclosed are embodiments for scoring one or more trajectories of a vehicle through a given traffic scenario using a machine learning model that predicts reasonableness scores for the trajectories. In an embodiment, human annotators, referred to as a “reasonable crowd,” are presented with renderings of two or more vehicle trajectories traversing through the same or different traffic scenarios. The annotators are asked to indicate their preference for one trajectory over the other(s). Inputs collected from the human annotators are used to train the machine learning model to predict reasonableness scores for one or more trajectories for a given traffic scenario. These predicted trajectories can be used to rank trajectories generated by a route planner based on their scores, compare AV software stacks, or used by any other application that could benefit from a machine learning model that scores vehicle trajectories.

Abstract: Provided are systems, methods and computer program products for evaluating subsystem performance. In some embodiments, a method comprises perturbing a first attribute of a first subsystem of a system that includes a plurality of subsystems, determining a change in a second attribute of a second subsystem of the system in response to the perturbing of the first attribute, where at least one output of the first subsystem is passed to the second subsystem, and determining a value for a performance metric of the system based on a correlation of the performance metric with the first and second attributes. In some embodiments, the system is a software stack of an autonomous vehicle (AV) and the performance metric is an objective function output that measures a quality of the AV's driving behavior.

Abstract: Provided are systems, methods and computer program products for evaluating subsystem performance. In some embodiments, a method comprises perturbing a first attribute of a first subsystem of a system that includes a plurality of subsystems, determining a change in a second attribute of a second subsystem of the system in response to the perturbing of the first attribute, where at least one output of the first subsystem is passed to the second subsystem, and determining a value for a performance metric of the system based on a correlation of the performance metric with the first and second attributes. In some embodiments, the system is a software stack of an autonomous vehicle (AV) and the performance metric is an objective function output that measures a quality of the AV's driving behavior.

Abstract: Techniques are provided for improving a perception processing pipeline for object detection that fuses image segmentation data (e.g., segmentation scores) with LiDAR points. The disclosed techniques are implemented using an architecture that accepts point clouds and images as input and estimates oriented 3D bounding boxes for all relevant object classes. In an embodiment, a method comprises: matching temporally, using one or more processors of a vehicle, points in a three-dimensional (3D) point cloud with an image; generating, using an image-based neural network, semantic data for the image; decorating, using the one or more processors, the points in the 3D point cloud with the semantic data; and estimating, using a 3D object detector with the decorated points as input, oriented 3D bounding boxes for the one or more objects.

Abstract: Enclosed are embodiments for scoring one or more trajectories of a vehicle through a given traffic scenario using a machine learning model that predicts reasonableness scores for the trajectories. In an embodiment, human annotators, referred to as a “reasonable crowd,” are presented with renderings of two or more vehicle trajectories traversing through the same or different traffic scenarios. The annotators are asked to indicate their preference for one trajectory over the other(s). Inputs collected from the human annotators are used to train the machine learning model to predict reasonableness scores for one or more trajectories for a given traffic scenario. These predicted trajectories can be used to rank trajectories generated by a route planner based on their scores, compare AV software stacks, or used by any other application that could benefit from a machine learning model that scores vehicle trajectories.

Abstract: Techniques are provided for improving a perception processing pipeline for object detection that fuses image segmentation data (e.g., segmentation scores) with LiDAR points. The disclosed techniques are implemented using an architecture that accepts point clouds and images as input and estimates oriented 3D bounding boxes for all relevant object classes. In an embodiment, a method comprises: matching temporally, using one or more processors of a vehicle, points in a three-dimensional (3D) point cloud with an image; generating, using an image-based neural network, semantic data for the image; decorating, using the one or more processors, the points in the 3D point cloud with the semantic data; and estimating, using a 3D object detector with the decorated points as input, oriented 3D bounding boxes for the one or more objects.

Abstract: Enclosed are embodiments for scoring one or more trajectories of a vehicle through a given traffic scenario using a machine learning model that predicts reasonableness scores for the trajectories. In an embodiment, human annotators, referred to as a “reasonable crowd,” are presented with renderings of two or more vehicle trajectories traversing through the same or different traffic scenarios. The annotators are asked to indicate their preference for one trajectory over the other(s). Inputs collected from the human annotators are used to train the machine learning model to predict reasonableness scores for one or more trajectories for a given traffic scenario. These predicted trajectories can be used to rank trajectories generated by a route planner based on their scores, compare AV software stacks, or used by any other application that could benefit from a machine learning model that scores vehicle trajectories.

Abstract: Techniques are provided for improving a perception processing pipeline for object detection that fuses image segmentation data (e.g., segmentation scores) with LiDAR points. The disclosed techniques are implemented using an architecture that accepts point clouds and images as input and estimates oriented 3D bounding boxes for all relevant object classes. In an embodiment, a method comprises: matching temporally, using one or more processors of a vehicle, points in a three-dimensional (3D) point cloud with an image; generating, using an image-based neural network, semantic data for the image; decorating, using the one or more processors, the points in the 3D point cloud with the semantic data; and estimating, using a 3D object detector with the decorated points as input, oriented 3D bounding boxes for the one or more objects.