Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that obtain scene features in an environment that includes an autonomous vehicle, a first target agent, and a second target agent, and determines whether the first target agent is an emergency vehicle that is active at a current time point. In response to determining that the first target agent is an emergency vehicle that is active at the current time point, an input is generated from the scene features. The input can characterize the scene and indicate that the first target agent is an emergency vehicle that is active at the current time point. Also in response, the input can be processed using a machine learning model that is configured to generate a trajectory prediction output for the second target agent that characterizes predicted future behavior of the second target agent after the current time point.

Abstract: Aspects of the disclosure relate to detecting and responding to objects in a vehicle's environment. For example, an object may be identified in a vehicle's environment, the object having a heading and location. A set of possible actions for the object may be generated using map information describing the vehicle's environment and the heading and location of the object. A set of possible future trajectories of the object may be generated based on the set of possible actions. A likelihood value of each trajectory of the set of possible future trajectories may be determined based on contextual information including a status of the detected object. A final future trajectory is determined based on the determined likelihood value for each trajectory of the set of possible future trajectories. The vehicle is then maneuvered in order to avoid the final future trajectory and the object.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for intervention behavior prediction. One of the methods includes receiving data characterizing a scene that includes a first agent and a second agent in an environment. A confounder prediction input generated from the data is processed using a confounder prediction model. A plurality of predicted conditional probability distributions is generated, wherein each predicted conditional probability distribution is conditioned on: (i) a planned intervention by the second agent, and (ii) the confounder variable belonging to a corresponding confounder class.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. The high-priority agents can be identified based on a set of mutual importance scores in which each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent from a set of agents on planning decisions of the other. The mutual importance scores can be calculated based on importance scores assessed from the perspectives of both the vehicle and the agents.

Abstract: To operate an autonomous vehicle, a rail agent is detected in a vicinity of the autonomous vehicle using a detection system. One or more tracks are determined on which the detected rail agent is possibly traveling, and possible paths for the rail agent are predicted based on the determined one or more tracks. One or more motion paths are determined for one or more probable paths from the possible paths, and a likelihood for each of the one or more probable paths is determined based on each motion plan. A path for the autonomous vehicle is then determined based on a most probable path associated with a highest likelihood for the rail agent, and the autonomous vehicle is operated using the determined path.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that obtain scene features in an environment that includes an autonomous vehicle and a target agent at a current time point; determine whether the target agent is an emergency vehicle that is active at the current time point; generate from the scene features, an input (i) that includes the scene features and (ii) that indicates whether the target vehicle is an emergency vehicle that is active at the current time point; and process the input using a machine learning model that is configured to generate a behavior prediction output for the target agent that characterizes predicted future behavior of the target agent after the current time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using slice-based dynamic neural networks. One of the methods includes receiving a new input for processing by a neural network that includes a first conditional neural network layer that has a set of shared parameters and a respective set of slice parameters for each of a plurality of slices. One or more slices to which the new input belongs are identified. The new input is processed to generate a network output, including: receiving a layer input to the first conditional neural network layer; and processing the layer input using the set of shared parameters, the respective one or more sets of slice parameters for the identified one or more slices, but not the respective sets of slice parameters for any other slices to which the new input does not belong.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating training data for training a machine learning model to perform trajectory prediction. One of the methods includes: obtaining a training input, the training input including (i) data characterizing an agent in an environment as of a first time and (ii) data characterizing a candidate trajectory of the agent in the environment for a first time period that is after the first time. A long-term label for the candidate trajectory that indicates whether the agent actually followed the candidate trajectory for the first time period is determined. A short-term label for the candidate trajectory that indicates whether the agent intended to follow the candidate trajectory is determined. A ground-truth probability for the candidate trajectory is determined. The training input is associated with the ground-truth probability for the candidate trajectory in the training data.

Abstract: To operate an autonomous vehicle, a rail agent is detected in a vicinity of the autonomous vehicle using a detection system. One or more tracks are determined on which the detected rail agent is possibly traveling, and possible paths for the rail agent are predicted based on the determined one or more tracks. One or more motion paths are determined for one or more probable paths from the possible paths, and a likelihood for each of the one or more probable paths is determined based on each motion plan. A path for the autonomous vehicle is then determined based on a most probable path associated with a highest likelihood for the rail agent, and the autonomous vehicle is operated using the determined path.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining respective importance scores for a plurality of agents in a vicinity of an autonomous vehicle navigating through an environment. The respective importance scores characterize a relative impact of each agent on planned trajectories generated by a planning subsystem of the autonomous vehicle. In one aspect, a method comprises providing different states of an environment as input to the planning subsystem and obtaining as output from the planning subsystem corresponding planned trajectories. Importance scores for the one or more agents that are in one state but not in the other are determined based on a measure of difference between the planned trajectories.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle and, for only those agents which are high priority agents, generating data characterizing the agents using a first prediction model. In a first aspect, a system identifies multiple agents in an environment in a vicinity of a vehicle. The system generates a respective importance score for each of the agents by processing a feature representation of each agent using an importance scoring model. The importance score for an agent characterizes an estimated impact of the agent on planning decisions generated by a planning system of the vehicle which plans a future trajectory of the vehicle. The system identifies, as high-priority agents, a proper subset of the plurality of agents with the highest importance scores.

Abstract: Aspects of the disclosure provide methods of modeling wrong-way driving of road users. For instance, log data including an observed trajectory of a first road user may be accessed. A first set of candidate lane segments for wrong-way driving may be identified from map information. A second set of candidate lane segments for not wrong-way driving may be identified from the map information. For each candidate lane segment in the first set and in the second set, a distance cost between the candidate lane segment and the observed trajectory may be determined. A candidate lane segment may be selected from at least one of the first set or the second set based on the determined distance costs. The selected candidate lane segment may be used to train a model to provide a likelihood of a second road user being engaged in wrong-way driving in a lane.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. The high-priority agents can be identified based on a set of mutual importance scores in which each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent from a set of agents on planning decisions of the other. The mutual importance scores can be calculated based on importance scores assessed from the perspectives of both the vehicle and the agents.

Abstract: As an example, data identifying characteristics of a road user as well as contextual information about the vehicle's environment is received from the vehicle's perception system. A prediction of the intent of the object including an action of a predetermined list of actions to be initiated by the road user and a point in time for initiation of the action is generated using the data. A prediction of the behavior of the road user for a predetermined period of time into the future indicating that the road user is not going to initiate the action during the predetermined period of time is generated using the data. When the prediction of the behavior indicates that the road user is not going to initiate the action during the predetermined period of time, the vehicle is maneuvered according to the prediction of the intent prior to the vehicle passing the object.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. The high-priority agents can be identified based on a set of mutual importance scores in which each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent from a set of agents on planning decisions of the other. The mutual importance scores can be calculated based on importance scores assessed from the perspectives of both the vehicle and the agents.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. In one aspect, a method comprises processing an input that characterizes a trajectory of the vehicle in an environment using an importance scoring model to generate an output that defines a respective importance score for each of a plurality of agents in the environment in the vicinity of the vehicle. The importance score for an agent characterizes an estimated impact of the agent on planning decisions generated by a planning system of the vehicle which plans a future trajectory of the vehicle. The high-priority agents are identified as a proper subset of the plurality of agents with the highest importance scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that obtain scene features in an environment that includes an autonomous vehicle, a first target agent, and a second target agent, and determines whether the first target agent is an emergency vehicle that is active at a current time point. In response to determining that the first target agent is an emergency vehicle that is active at the current time point, an input is generated from the scene features. The input can characterize the scene and indicate that the first target agent is an emergency vehicle that is active at the current time point. Also in response, the input can be processed using a machine learning model that is configured to generate a trajectory prediction output for the second target agent that characterizes predicted future behavior of the second target agent after the current time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that obtain scene features in an environment that includes an autonomous vehicle and a target agent at a current time point; determine whether the target agent is an emergency vehicle that is active at the current time point; generate from the scene features, an input (i) that includes the scene features and (ii) that indicates whether the target vehicle is an emergency vehicle that is active at the current time point; and process the input using a machine learning model that is configured to generate a behavior prediction output for the target agent that characterizes predicted future behavior of the target agent after the current time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for predicting how likely it is that a target agent in an environment will yield to another agent when the pair of agents are predicted to have overlapping future paths. In one aspect, a method comprises obtaining a first trajectory prediction specifying a predicted future path for a target agent in an environment; obtaining a second trajectory prediction specifying a predicted future path for another agent in the environment; determining that, at an overlapping region, the predicted future path for the target agent overlaps with the predicted future path for the other agent; and in response: providing as input to a machine learning model respective features for the target agent and the other agent; and obtaining the likelihood score as output from the machine learning model.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network having a plurality of sub neural networks to assign respective confidence scores to one or more candidate future trajectories for an agent. Each confidence score indicates a predicted likelihood that the agent will move along the corresponding candidate future trajectory in the future. In one aspect, a method includes using the first sub neural network to generate a training intermediate representation; using the second sub neural network to generate respective training confidence scores; using a trajectory generation neural network to generate a training trajectory generation output; computing a first loss and a second loss; and determining an update to the current values of the parameters of the first and second sub neural networks.