Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. The high-priority agents can be identified based on a set of mutual importance scores in which each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent from a set of agents on planning decisions of the other. The mutual importance scores can be calculated based on importance scores assessed from the perspectives of both the vehicle and the agents.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining occupancies of surrounding agents. One of the methods includes obtaining scene data characterizing an environment at a current time point; processing a first network input generated from the scene data using a first neural network to generate an intermediate output; obtaining an identification of a future time point that is after the current time point; and generating, from the intermediate output and the future time point, an occupancy output, wherein the occupancy output comprises respective occupancy probabilities for each of a plurality of locations in the environment, wherein the respective occupancy probability for each location characterizes a likelihood that one or more agents will occupy the location at the future time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using slice-based dynamic neural networks. One of the methods includes receiving a new input for processing by a neural network that includes a first conditional neural network layer that has a set of shared parameters and a respective set of slice parameters for each of a plurality of slices. One or more slices to which the new input belongs are identified. The new input is processed to generate a network output, including: receiving a layer input to the first conditional neural network layer; and processing the layer input using the set of shared parameters, the respective one or more sets of slice parameters for the identified one or more slices, but not the respective sets of slice parameters for any other slices to which the new input does not belong.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating training data for training a machine learning model to perform trajectory prediction. One of the methods includes: obtaining a training input, the training input including (i) data characterizing an agent in an environment as of a first time and (ii) data characterizing a candidate trajectory of the agent in the environment for a first time period that is after the first time. A long-term label for the candidate trajectory that indicates whether the agent actually followed the candidate trajectory for the first time period is determined. A short-term label for the candidate trajectory that indicates whether the agent intended to follow the candidate trajectory is determined. A ground-truth probability for the candidate trajectory is determined. The training input is associated with the ground-truth probability for the candidate trajectory in the training data.

Abstract: An autonomous vehicle includes sensor subsystem(s) that output a sensor signal. A perception subsystem (i) detects an agent in a vicinity of the autonomous vehicle and (ii) generates a motion signal that describes at least one of a past motion or a present motion of the agent. An intention prediction subsystem processes the sensor signal to generate an intention signal that describes at least one intended action of the agent. A behavior prediction subsystem processes the motion signal and the intention signal to generate a behavior prediction signal that describes at least one predicted behavior of the agent. A planner subsystem processes the behavior prediction signal to plan a driving decision for the autonomous vehicle.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating candidate future trajectories for agents. One of the methods includes obtaining scene data characterizing a scene in an environment at a current time point; for each of a plurality of lane segments, processing a model input comprising (i) features of the lane segment and (ii) features of the target agent using a machine learning model that is configured to process the model input to generate a respective score for the lane segment that represents a likelihood that the lane segment will be a first lane segment traversed by the target agent after the current time point; selecting, as a set of seed lane segments, a proper subset of the plurality of lane segments based on the respective scores; and generating a plurality of candidate future trajectories for the target agent.

Abstract: Data representing a set of predicted trajectories and a planned trajectory for an autonomous vehicle is obtained. A predictability score for the planned trajectory can be determined based on a comparison of the planned trajectory to the set of predicted trajectories for the autonomous vehicle. The predictability score indicates a level of predictability of the planned trajectory. A determination can be made, based at least on the predictability score, whether to initiate travel with the autonomous vehicle along the planned trajectory. In response to determining to initiate travel with the autonomous vehicle along the planned trajectory, a control system can be directed to maneuver the autonomous vehicle along the planned trajectory.

Abstract: Methods, systems, and apparatus for generating trajectory predictions for one or more target agents. In one aspect, a system comprises one or more computers configured to obtain scene context data characterizing a scene in an environment at a current time point, where the scene includes multiple agents that include a target agent and one or more context agents, and the scene context data includes respective context data for each of multiple different modalities of context data. The one or more computers then generate an encoded representation of the scene in the environment that includes one or more embeddings and process the encoded representation of the scene context data using a decoder neural network to generate a trajectory prediction output for the target agent that predicts a future trajectory of the target after the current time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining respective importance scores for a plurality of agents in a vicinity of an autonomous vehicle navigating through an environment. The respective importance scores characterize a relative impact of each agent on planned trajectories generated by a planning subsystem of the autonomous vehicle. In one aspect, a method comprises providing different states of an environment as input to the planning subsystem and obtaining as output from the planning subsystem corresponding planned trajectories. Importance scores for the one or more agents that are in one state but not in the other are determined based on a measure of difference between the planned trajectories.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle and, for only those agents which are high priority agents, generating data characterizing the agents using a first prediction model. In a first aspect, a system identifies multiple agents in an environment in a vicinity of a vehicle. The system generates a respective importance score for each of the agents by processing a feature representation of each agent using an importance scoring model. The importance score for an agent characterizes an estimated impact of the agent on planning decisions generated by a planning system of the vehicle which plans a future trajectory of the vehicle. The system identifies, as high-priority agents, a proper subset of the plurality of agents with the highest importance scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. The high-priority agents can be identified based on a set of mutual importance scores in which each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent from a set of agents on planning decisions of the other. The mutual importance scores can be calculated based on importance scores assessed from the perspectives of both the vehicle and the agents.

Abstract: An autonomous vehicle includes sensor subsystem(s) that output a sensor signal. A perception subsystem (i) detects an agent in a vicinity of the autonomous vehicle and (ii) generates a motion signal that describes at least one of a past motion or a present motion of the agent. An intention prediction subsystem processes the sensor signal to generate an intention signal that describes at least one intended action of the agent. A behavior prediction subsystem processes the motion signal and the intention signal to generate a behavior prediction signal that describes at least one predicted behavior of the agent. A planner subsystem processes the behavior prediction signal to plan a driving decision for the autonomous vehicle.

Abstract: Methods, systems, and apparatus for predicting future trajectories of agents in an environment. In one aspect, a system comprises one or more computers configured to receive a data set comprising multiple training examples. The training examples include scene data comprising respective agent data for multiple agents and a ground truth trajectory for a target agent that represents ground truth motion of the target agent after a corresponding time point. The one or more computers obtain data identifying one or more of the multiple agents as non-causal agents for each training example. A non-causal agent is an agent whose states do not cause the ground truth trajectory for the target agent to change. The one or more computers generate a respective modified training example from each of the multiple training examples.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating cut-in probabilities of agents surrounding a vehicle. One of the methods includes obtaining agent trajectory data for one or more agents in an environment; obtaining vehicle trajectory data of a vehicle in the environment; and processing a network input generated from the agent trajectory data and vehicle trajectory data using a neural network to generate a cut-in output, wherein the cut-in output comprises respective cut-in probabilities for each of a plurality of locations in the environment, wherein the respective cut-in probability for each location that is a current location of one of the one or more agents characterizes a likelihood that the agent in the current location will intersect with a planned future location of the vehicle within a predetermined amount of time.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. The high-priority agents can be identified based on a set of mutual importance scores in which each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent from a set of agents on planning decisions of the other. The mutual importance scores can be calculated based on importance scores assessed from the perspectives of both the vehicle and the agents.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. In one aspect, a method comprises processing an input that characterizes a trajectory of the vehicle in an environment using an importance scoring model to generate an output that defines a respective importance score for each of a plurality of agents in the environment in the vicinity of the vehicle. The importance score for an agent characterizes an estimated impact of the agent on planning decisions generated by a planning system of the vehicle which plans a future trajectory of the vehicle. The high-priority agents are identified as a proper subset of the plurality of agents with the highest importance scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network configured to receive a network input and to assign a respective score to each of a plurality of locations in the network input. In one aspect, a method includes obtaining a training input and a corresponding ground truth output; processing the training input to generate a training output; computing a loss for the training input, comprising: selecting a plurality of candidate locations; setting to zero the training scores for any location in the selected candidate locations that has a ground truth score below a threshold value; for each of a plurality of pairs of locations in the selected candidate locations: computing a pair-wise loss for the pair; and combining the pair-wise losses to compute the loss for the training input; and determining an update to the current values of the parameters.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent behavior prediction using keypoint data. One of the methods includes obtaining data characterizing a scene in an environment, the data comprising: (i) context data comprising data characterizing historical trajectories of a plurality of agents up to the current time point; and (ii) keypoint data for a target agent; processing the context data using a context data encoder neural network to generate a context embedding for the target agent; processing the keypoint data using a keypoint encoder neural network to generate a keypoint embedding for the target agent; generating a combined embedding for the target agent from the context embedding and the keypoint embedding; and processing the combined embedding using a decoder neural network to generate a behavior prediction output for the target agent that characterizes predicted behavior of the target agent after the current time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium that determine yield behavior for an autonomous vehicle, and can include identifying an agent that is in a vicinity of an autonomous vehicle navigating through a scene at a current time point. Scene features can be obtained and can include features of (i) the agent and (ii) the autonomous vehicle. An input that can include the scene features can be processed using a first machine learning model that is configured to generate (i) a crossing intent prediction that includes a crossing intent score that represents a likelihood that the agent intends to cross a roadway in a future time window after the current time, and (ii) a crossing action prediction that includes a crossing action score that represents a likelihood that the agent will cross the roadway in the future time window after the current time.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a representation of a trajectory of a target agent in an environment. In one aspect, the representation of the trajectory of the target agent in the environment is a concatenation of a plurality of channels, where each channel is represented as a two-dimensional array of data values. Each position in each channel corresponds to a respective spatial position in the environment, and corresponding positions in different channels correspond to the same spatial position in the environment. The channels include a time channel and a respective motion channel corresponding to each motion parameter in a predetermined set of motion parameters.