Abstract: In an example method, a computer system receives first data representing a plurality of scenarios for estimating a performance of a vehicle in conducting autonomous operations. Further, the computer system determines, for each of the scenarios: (i) a first metric indicating an observed performance of the vehicle in that scenario, where the first metric is determined based on at least one rule, and (ii) a second metric indicating a degree of information gain associated with that scenario. The computer system selects a subset of the scenarios based on the first metrics and the second metrics, and outputs second data indicative of the subset of the scenarios.

Abstract: Provided are methods for testing of a control system of a vehicle using generated rulebook based scenarios, which can include determining a simulated environment, receiving a hierarchical plurality of autonomous vehicle rules, determining a trajectory of a simulated vehicle within the simulated environment, generating a plurality of simulated scenarios for the simulated vehicle, identifying at least one violation of at least one autonomous vehicle rule by the simulated vehicle in a set of the simulated scenarios, determining a scenario score for each simulated scenario based on the violations, and identifying at least one simulated scenario for a trained neural network of a vehicle based on the scenario scores.

Abstract: Provided are methods for trajectory generation based on a hierarchical plurality of rules using diverse trajectories, which can include generating a first set of trajectories for a vehicle from a first pose, identifying a first trajectory and a second trajectory from the first set of trajectories, generating a second set of trajectories for the vehicle from a second pose and a third set of trajectories for the vehicle from a third pose, identifying a third trajectory based at least in part on the second set of trajectories and the third set of trajectories, the third trajectory violating a first behavioral rule associated with a first priority that is less than a priority of behavioral rules violated by other trajectories, and determining a path for the vehicle based at least in part on the third trajectory. Systems and computer program products are also provided.

Abstract: Provided are methods for autonomous vehicle yielding, which can include obtaining sensor data associated with an environment and obtaining a rule indicative of a target expressive operation. Some methods described also include determining whether the sensor data meets a first criterion, applying the rule in response to the sensor data meeting the first criterion, evaluating a first trajectory of the autonomous vehicle, and selecting a second trajectory. Systems and computer program products are also provided.

Abstract: Provided are methods, systems, and computer program products for autonomous vehicle validation using real-world adversarial events are described herein. The method includes obtaining a record of human driver data and extracting at least one safety critical scenario from the record to create a test data set associated with operation of vehicles by human drivers during safety critical scenarios. The method includes simulating operation of a vehicle under evaluation during safety critical scenarios and comparing a response of the vehicle under evaluation to the operation of vehicles by human drivers. The method further includes validating a response of the vehicle under evaluation during the simulated operation according to the test data set by transforming the vehicle response into a safety indicator.

Abstract: A rule violation leading to a traffic conflict involving a vehicle and at least one agent in real-time driving scenarios, simulation, re-simulation or other application is determined to be the fault of the vehicle by determining whether the situation was “reasonably foreseeable,” determining whether the at least one agent is a vulnerable road user (VRU) and determining whether the at least one agent violated a higher priority rule than the rule violated by the vehicle. In an embodiment, on-vehicle decisions are based on a rulebook with a priority structure that accounts for responsibility and determines an “initiator” of the traffic conflict based on whether the vehicle or the at least one agent was the first to violate a rule and the first to violate a higher priority rule.

Abstract: Enclosed are embodiments for scoring one or more trajectories of a vehicle through a given traffic scenario using a machine learning model that predicts reasonableness scores for the trajectories. In an embodiment, human annotators, referred to as a “reasonable crowd,” are presented with renderings of two or more vehicle trajectories traversing through the same or different traffic scenarios. The annotators are asked to indicate their preference for one trajectory over the other(s). Inputs collected from the human annotators are used to train the machine learning model to predict reasonableness scores for one or more trajectories for a given traffic scenario. These predicted trajectories can be used to rank trajectories generated by a route planner based on their scores, compare AV software stacks, or used by any other application that could benefit from a machine learning model that scores vehicle trajectories.

Abstract: Provided are methods for graph exploration for rulebook trajectory generation. Some methods described include generating a next set of alternative trajectories for the vehicle from a next pose, the next set of alternative trajectories representing operation of the vehicle from the next pose, wherein the next pose is located at an end of an identified trajectory. Next trajectories are iteratively identified from corresponding next sets of alternative trajectories, wherein a next trajectory violates a lowest behavioral rule of the hierarchical plurality of rules, the lowest behavioral rule having a priority less than a priority of behavioral rules associated with other trajectories in a corresponding next set of alternative trajectories until a goal pose or timeout is reached to generate a graph. Systems and computer program products are also provided.

Abstract: Enclosed are embodiments for scenario-based behavior specification and validation.

Abstract: In an embodiment, a method comprises: selecting a scenario for simulating a vehicle and agent(s) in a virtual world environment; selecting a set of subsystem component models for the vehicle; simulating the scenario in the virtual world environment using the selected subsystem component models, wherein the simulating comprises: estimating a pose of the vehicle and the agent(s); determining a probability of detection of the agent(s) by sensor(s) of the vehicle based on a perception subsystem component model, the estimated pose of the vehicle/agent(s) and a latency model modeling latency of a message queuing network for communicating data between the subsystems; generating a set of candidate trajectories for the vehicle; evaluating each trajectory based on a number of rule violations associated with the trajectory; selecting one trajectory from the set of trajectories based on the number of rule violations; and analyzing outputs of the subsystem component models to determine their interdependencies.

Abstract: In an example method, a computer system receives first data representing a plurality of scenarios for estimating a performance of a vehicle in conducting autonomous operations. Further, the computer system determines, for each of the scenarios: (i) a first metric indicating an observed performance of the vehicle in that scenario, where the first metric is determined based on at least one rule, and (ii) a second metric indicating a degree of information gain associated with that scenario. The computer system selects a subset of the scenarios based on the first metrics and the second metrics, and outputs second data indicative of the subset of the scenarios.

Abstract: Enclosed are embodiments for scoring one or more trajectories of a vehicle through a given traffic scenario using a machine learning model that predicts reasonableness scores for the trajectories. In an embodiment, human annotators, referred to as a “reasonable crowd,” are presented with renderings of two or more vehicle trajectories traversing through the same or different traffic scenarios. The annotators are asked to indicate their preference for one trajectory over the other(s). Inputs collected from the human annotators are used to train the machine learning model to predict reasonableness scores for one or more trajectories for a given traffic scenario. These predicted trajectories can be used to rank trajectories generated by a route planner based on their scores, compare AV software stacks, or used by any other application that could benefit from a machine learning model that scores vehicle trajectories.

Abstract: Enclosed are embodiments for scenario-based behavior specification and validation.

Abstract: Methods for vehicle operation using behavioral rule checks include receiving first sensor data from first sensors and second sensor data from second sensors of the vehicle. The first sensor data represents operation of the vehicle in accordance with a first trajectory. The second sensor data represents at least one object. It is determined that the first trajectory violates a first behavioral rule of operation based on the first sensor data and the second sensor data. The first behavioral rule has a first priority. Multiple alternative trajectories are generated using control barrier functions. A second trajectory is identified that violates a second behavioral rule having a second priority less than the first priority. Responsive to identifying the second trajectory, a message is transmitted to a control circuit of the vehicle to operate the vehicle based on the second trajectory.

Abstract: Methods for vehicle operation using a behavioral rule model include receiving sensor data from a first set of sensors and a second set of sensors. The sensor data represents operation of the vehicle with respect to one or more objects. Violations of a behavioral model of the operation of the vehicle are determined based on the sensor data. A first risk level of the one or more violations is determined based on a distribution of events of the operation of the vehicle with respect to the one or more objects. Responsive to the first risk level being greater than a threshold risk level, a trajectory is generated. The trajectory has a second risk level lower than the threshold risk level. The vehicle is operated based on the trajectory to avoid a collision of the vehicle and the one or more objects.

Abstract: Enclosed are embodiments for scoring one or more trajectories of a vehicle through a given traffic scenario using a machine learning model that predicts reasonableness scores for the trajectories. In an embodiment, human annotators, referred to as a “reasonable crowd,” are presented with renderings of two or more vehicle trajectories traversing through the same or different traffic scenarios. The annotators are asked to indicate their preference for one trajectory over the other(s). Inputs collected from the human annotators are used to train the machine learning model to predict reasonableness scores for one or more trajectories for a given traffic scenario. These predicted trajectories can be used to rank trajectories generated by a route planner based on their scores, compare AV software stacks, or used by any other application that could benefit from a machine learning model that scores vehicle trajectories.

Abstract: Enclosed are embodiments for scoring one or more trajectories of a vehicle through a given traffic scenario using a machine learning model that predicts reasonableness scores for the trajectories. In an embodiment, human annotators, referred to as a “reasonable crowd,” are presented with renderings of two or more vehicle trajectories traversing through the same or different traffic scenarios. The annotators are asked to indicate their preference for one trajectory over the other(s). Inputs collected from the human annotators are used to train the machine learning model to predict reasonableness scores for one or more trajectories for a given traffic scenario. These predicted trajectories can be used to rank trajectories generated by a route planner based on their scores, compare AV software stacks, or used by any other application that could benefit from a machine learning model that scores vehicle trajectories.

Abstract: Techniques are provided for operation of a vehicle in the event of an emergency. The techniques include receiving, using one or more sensors of a vehicle operating within an environment, sensor data representing an object located within the environment. The sensor data is used to identify whether the object is an emergency vehicle. Responsive to identifying that the object is an emergency vehicle, the sensor data is used to determine whether the emergency vehicle is operating in an emergency mode. Responsive to determining that the emergency vehicle is operating in the emergency mode, instructions representing an emergency operation for the vehicle are transmitted to a control module of the vehicle. The control module of the vehicle operates the vehicle in accordance with the emergency operation.