Abstract: Methods, systems, and apparatus for generating trajectory predictions for one or more target agents. In one aspect, a system comprises one or more computers configured to obtain scene context data characterizing a scene in an environment at a current time point, where the scene includes multiple agents that include a target agent and one or more context agents, and the scene context data includes respective context data for each of multiple different modalities of context data. The one or more computers then generate an encoded representation of the scene in the environment that includes one or more embeddings and process the encoded representation of the scene context data using a decoder neural network to generate a trajectory prediction output for the target agent that predicts a future trajectory of the target after the current time point.

Abstract: Aspects of the disclosure provide for generating distributions for hypothetical or potentially occluded objects. For instance, a location for which to generate one or more distributions may be identified. Observations of road users by perception systems of a plurality of autonomous vehicles may be accessed. Each of these observations may identify a characteristic of one of the road users. A distribution of the characteristic for the location may be determined based on the observations. The distribution may be provided to one or more autonomous vehicles in order to enable the one or more autonomous vehicles to use the distribution to generate a characteristic for a hypothetical occluded road user and to respond to the hypothetical occluded road user.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent behavior prediction using keypoint data. One of the methods includes obtaining data characterizing a scene in an environment, the data comprising: (i) context data comprising data characterizing historical trajectories of a plurality of agents up to the current time point; and (ii) keypoint data for a target agent; processing the context data using a context data encoder neural network to generate a context embedding for the target agent; processing the keypoint data using a keypoint encoder neural network to generate a keypoint embedding for the target agent; generating a combined embedding for the target agent from the context embedding and the keypoint embedding; and processing the combined embedding using a decoder neural network to generate a behavior prediction output for the target agent that characterizes predicted behavior of the target agent after the current time point.

Abstract: Methods, systems, and apparatus for generating trajectory predictions for one or more target agents. In one aspect, a system comprises one or more computers configured to obtain scene context data characterizing a scene in an environment at a current time point, where the scene includes multiple agents that include a target agent and one or more context agents, and the scene context data includes respective context data for each of multiple different modalities of context data. The one or more computers then generate an encoded representation of the scene in the environment that includes one or more embeddings and process the encoded representation of the scene context data using a decoder neural network to generate a trajectory prediction output for the target agent that predicts a future trajectory of the target after the current time point.

Abstract: Methods, systems, and apparatus for predicting future trajectories of agents in an environment. In one aspect, a system comprises one or more computers configured to receive a data set comprising multiple training examples. The training examples include scene data comprising respective agent data for multiple agents and a ground truth trajectory for a target agent that represents ground truth motion of the target agent after a corresponding time point. The one or more computers obtain data identifying one or more of the multiple agents as non-causal agents for each training example. A non-causal agent is an agent whose states do not cause the ground truth trajectory for the target agent to change. The one or more computers generate a respective modified training example from each of the multiple training examples.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for anomaly estimation for behavior predictions using a neural network. One of the methods includes receiving data characterizing a scene that includes an agent in an environment. A behavior prediction input generated from the data is processed using a behavior prediction model. The behavior prediction model is configured to process the behavior prediction input to generate a predicted probability distribution over a plurality of possible behaviors for the agent. An anomaly estimation input generated from the data is processed using an anomaly estimation model. The anomaly estimation model is configured to process the anomaly estimation input to generate a prediction error for the predicted probability distribution.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using context-sensitive fusion.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium that determine yield behavior for an autonomous vehicle, and can include identifying an agent that is in a vicinity of an autonomous vehicle navigating through a scene at a current time point. Scene features can be obtained and can include features of (i) the agent and (ii) the autonomous vehicle. An input that can include the scene features can be processed using a first machine learning model that is configured to generate (i) a crossing intent prediction that includes a crossing intent score that represents a likelihood that the agent intends to cross a roadway in a future time window after the current time, and (ii) a crossing action prediction that includes a crossing action score that represents a likelihood that the agent will cross the roadway in the future time window after the current time.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using slice-based dynamic neural networks. One of the methods includes receiving a new input for processing by a neural network that includes a first conditional neural network layer that has a set of shared parameters and a respective set of slice parameters for each of a plurality of slices. One or more slices to which the new input belongs are identified. The new input is processed to generate a network output, including: receiving a layer input to the first conditional neural network layer; and processing the layer input using the set of shared parameters, the respective one or more sets of slice parameters for the identified one or more slices, but not the respective sets of slice parameters for any other slices to which the new input does not belong.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for intervention behavior prediction. One of the methods includes receiving data characterizing a scene that includes a first agent and a second agent in an environment and receiving intervention data specifying a planned intervention to be performed by the second agent. A conditional behavior prediction output that assigns, to each of a plurality of possible future behaviors, (i) a respective conditional likelihood that the first agent performs the possible future behavior given that the second agent performs the planned intervention and (ii) a predicted value of a confounder variable for the possible future behavior is generated using a conditional behavior prediction model.

Abstract: Aspects of the disclosure provide for controlling a vehicle in an autonomous driving mode. For instance, sensor data for an object as well as a plurality of predicted trajectories may be received. Each predicted trajectory may represent a plurality of possible future locations for the object. A grid including a plurality of cells, each being associated with a geographic area, may be generated. Probabilities that the object will enter the geographic area associated with each of the plurality of cells over a period of time into the future may be determined based on the sensor data in order to generate a heat map. One or more of the plurality of predicted trajectories may be compared to the heat map. The vehicle may be controlled in the autonomous driving mode based on the comparison.

Abstract: An autonomous vehicle includes sensor subsystem(s) that output a sensor signal. A perception subsystem (i) detects an agent in a vicinity of the autonomous vehicle and (ii) generates a motion signal that describes at least one of a past motion or a present motion of the agent. An intention prediction subsystem processes the sensor signal to generate an intention signal that describes at least one intended action of the agent. A behavior prediction subsystem processes the motion signal and the intention signal to generate a behavior prediction signal that describes at least one predicted behavior of the agent. A planner subsystem processes the behavior prediction signal to plan a driving decision for the autonomous vehicle.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating trajectory predictions for one or more agents in an environment. In one aspect, a method comprises: obtaining scene context data characterizing a scene in an environment at a current time point and generating a respective predicted future trajectory for each of a plurality of agents in the scene at the current time point by sampling a sequence of discrete motion tokens that defines a joint future trajectory for the plurality of agents using a trajectory prediction neural network that is conditioned on the scene context data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for intervention behavior prediction. One of the methods includes receiving data characterizing a scene that includes a first agent and a second agent in an environment. A confounder prediction input generated from the data is processed using a confounder prediction model. A plurality of predicted conditional probability distributions is generated, wherein each predicted conditional probability distribution is conditioned on: (i) a planned intervention by the second agent, and (ii) the confounder variable belonging to a corresponding confounder class.

Abstract: Data representing a set of predicted trajectories and a planned trajectory for an autonomous vehicle is obtained. A predictability score for the planned trajectory can be determined based on a comparison of the planned trajectory to the set of predicted trajectories for the autonomous vehicle. The predictability score indicates a level of predictability of the planned trajectory. A determination can be made, based at least on the predictability score, whether to initiate travel with the autonomous vehicle along the planned trajectory. In response to determining to initiate travel with the autonomous vehicle along the planned trajectory, a control system can be directed to maneuver the autonomous vehicle along the planned trajectory.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. The high-priority agents can be identified based on a set of mutual importance scores in which each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent from a set of agents on planning decisions of the other. The mutual importance scores can be calculated based on importance scores assessed from the perspectives of both the vehicle and the agents.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining occupancies of surrounding agents. One of the methods includes obtaining scene data characterizing an environment at a current time point; processing a first network input generated from the scene data using a first neural network to generate an intermediate output; obtaining an identification of a future time point that is after the current time point; and generating, from the intermediate output and the future time point, an occupancy output, wherein the occupancy output comprises respective occupancy probabilities for each of a plurality of locations in the environment, wherein the respective occupancy probability for each location characterizes a likelihood that one or more agents will occupy the location at the future time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using slice-based dynamic neural networks. One of the methods includes receiving a new input for processing by a neural network that includes a first conditional neural network layer that has a set of shared parameters and a respective set of slice parameters for each of a plurality of slices. One or more slices to which the new input belongs are identified. The new input is processed to generate a network output, including: receiving a layer input to the first conditional neural network layer; and processing the layer input using the set of shared parameters, the respective one or more sets of slice parameters for the identified one or more slices, but not the respective sets of slice parameters for any other slices to which the new input does not belong.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating training data for training a machine learning model to perform trajectory prediction. One of the methods includes: obtaining a training input, the training input including (i) data characterizing an agent in an environment as of a first time and (ii) data characterizing a candidate trajectory of the agent in the environment for a first time period that is after the first time. A long-term label for the candidate trajectory that indicates whether the agent actually followed the candidate trajectory for the first time period is determined. A short-term label for the candidate trajectory that indicates whether the agent intended to follow the candidate trajectory is determined. A ground-truth probability for the candidate trajectory is determined. The training input is associated with the ground-truth probability for the candidate trajectory in the training data.

Abstract: An autonomous vehicle includes sensor subsystem(s) that output a sensor signal. A perception subsystem (i) detects an agent in a vicinity of the autonomous vehicle and (ii) generates a motion signal that describes at least one of a past motion or a present motion of the agent. An intention prediction subsystem processes the sensor signal to generate an intention signal that describes at least one intended action of the agent. A behavior prediction subsystem processes the motion signal and the intention signal to generate a behavior prediction signal that describes at least one predicted behavior of the agent. A planner subsystem processes the behavior prediction signal to plan a driving decision for the autonomous vehicle.