Abstract: A method performed by one or more computers, the method comprising: obtaining scene context data characterizing a scene in an environment at a current time point, wherein the scene context data includes features of the scene in a scene-centric coordinate system; generating a scene-centric encoded representation of the scene in the environment by processing the scene context data using an encoder neural network; for each target agent: obtaining agent-specific features for the target agent, processing the agent-specific features for the target agent and the scene-centric encoded representation of the scene using a fusion neural network to generate a fused scene representation for the target agent, and processing the fused scene representation for the target agent using a decoder neural network to generate a trajectory prediction output for the target agent in an agent-centric coordinate system for the target agent.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating trajectory predictions for one or more agents in an environment. In one aspect, a method comprises: obtaining scene context data characterizing a scene in an environment at a current time point and generating a respective predicted future trajectory for each of a plurality of agents in the scene at the current time point by sampling a sequence of discrete motion tokens that defines a joint future trajectory for the plurality of agents using a trajectory prediction neural network that is conditioned on the scene context data.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for predicting future trajectories for an agent in an environment. A system obtains scene context data characterizing the environment. The scene context data includes data that characterizes a trajectory of an agent in a vicinity of a vehicle in an environment up to a current time point. The system identifies a plurality of initial target locations in the environment. The system further generates, for each of a plurality of target locations that each corresponds to one of the initial target locations, a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent starting from the current time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using temporal-spatial interaction predictions.

Abstract: A system obtains scene context data characterizing the environment. The scene context data includes data that characterizes a trajectory of an agent in a vicinity of a vehicle up to a current time point. The system identifies a plurality of initial target locations, and generates, for each of a plurality of target locations that each corresponds to one of the initial target locations, a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent. For each target location in a first subset of the target locations, the system generates a predicted future trajectory for the agent given that the target location is the intended final location for the future trajectory. The system further selects, as likely future trajectories of the agent, one or more of the predicted future trajectories.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating trajectory predictions for one or more target agents, e.g., a vehicle, a cyclist, or a pedestrian, in an environment. In one aspect, one of the methods include: obtaining scene context data characterizing a scene at a current time point in an environment that includes multiple target agents; generating, from the scene context data, an encoded representation of the scene in the environment; and generating, by a diffusion model based on the encoded representation, a respective trajectory prediction output that predicts a respective future trajectory for each of the multiple target agents after the current time point.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for predicting future trajectories for an agent in an environment. A system obtains scene context data characterizing the environment. The scene context data includes data that characterizes a trajectory of an agent in a vicinity of a vehicle in an. environment up to a current time point. The system identifies a plurality of initial target locations in the environment. The system further generates, for each of a plurality of target locations that each corresponds to one of the initial target locations, a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent starting from the current time point.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for performing a conditional behavior prediction for one or more agents. The system obtains context data characterizing an environment. The context data includes data characterizing a plurality of agents, including a query agent and one or more target agents, in the environment at a current time point. The system further obtains data identifying a planned future trajectory for the query agent after the current time point, and for each target agent in the set, processes the context data and the data identifying the planned future trajectory using a first neural network to generate a conditional trajectory prediction output that defines a conditional probability distribution over possible future trajectories of the target agent after the current time point given that the query agent follows the planned future trajectory for the query agent after the current time point.

Abstract: Methods, systems, and apparatus for generating trajectory predictions for one or more agents. In one aspect, a system comprises one or more computers configured to obtain scene context data characterizing a scene in an environment at a current time point, where the scene includes multiple agents. The one or more computers process the scene context data using a marginal trajectory prediction neural network to generate a respective marginal trajectory prediction for each of the plurality of agents that defines multiple possible trajectories for the agent after the current time point and a respective likelihood score for each of the multiple possible future trajectories. The one or more computers can generate graph data based on the respective marginal trajectory predictions, and the one or more computers can process the graph data using a graph neural network to generate a joint trajectory prediction output for the multiple agents in the scene.

Abstract: Methods, systems, and apparatus for generating trajectory predictions for one or more target agents. In one aspect, a system comprises one or more computers configured to obtain scene context data characterizing a scene in an environment at a current time point, where the scene includes multiple agents that include a target agent and one or more context agents, and the scene context data includes respective context data for each of multiple different modalities of context data. The one or more computers then generate an encoded representation of the scene in the environment that includes one or more embeddings and process the encoded representation of the scene context data using a decoder neural network to generate a trajectory prediction output for the target agent that predicts a future trajectory of the target after the current time point.

Abstract: Methods, systems, and apparatus for predicting future trajectories of agents in an environment. In one aspect, a system comprises one or more computers configured to receive a data set comprising multiple training examples. The training examples include scene data comprising respective agent data for multiple agents and a ground truth trajectory for a target agent that represents ground truth motion of the target agent after a corresponding time point. The one or more computers obtain data identifying one or more of the multiple agents as non-causal agents for each training example. A non-causal agent is an agent whose states do not cause the ground truth trajectory for the target agent to change. The one or more computers generate a respective modified training example from each of the multiple training examples.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting the future movement of agents in an environment. In particular, the future movement is predicted through occupancy flow fields that specify, for each future time point in a sequence of future time points and for each agent type in a set of one or more agent types: an occupancy prediction for the future time step that specifies, for each grid cell, an occupancy likelihood that any agent of the agent type will occupy the grid cell at the future time point, and a motion flow prediction that specifies, for each grid cell, a motion vector that represents predicted motion of agents of the agent type within the grid cell at the future time point.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using context-sensitive fusion.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using temporal-spatial interaction predictions.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for generating a prediction that characterizes an environment. The system obtains an input including data characterizing observed trajectories one or more agents and data characterizing one or more map features identified in a map of the environment. The system generates, from the input, an encoder input that comprises representations for each of a plurality of points in a top-down representation of the environment. The system processes the encoder input using a point cloud encoder neural network to generate a global feature map of the environment, and processes a prediction input including the global feature map using a predictor neural network to generate a prediction output characterizing the environment.

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for performing a conditional behavior prediction for one or more agents. The system obtains context data characterizing an environment. The context data includes data characterizing a plurality of agents, including a query agent and one or more target agents, in the environment at a current time point. The system further obtains data identifying a planned future trajectory for the query agent after the current time point, and for each target agent in the set, processes the context data and the data identifying the planned future trajectory using a first neural network to generate a conditional trajectory prediction output that defines a conditional probability distribution over possible future trajectories of the target agent after the current time point given that the query agent follows the planned future trajectory for the query agent after the current time point.

Abstract: The generation and/or use of probabilistic heat maps for use in predicting the behavior of entities in an environment is described. In an example, a computing device(s) can receive sensor data from sensors of a vehicle in an environment. The computing device(s) can determine, based at least in partly on the sensor data, a location of an entity in the environment, and a first characteristic associated with the entity or the environment (type, velocity, orientation, etc.). The computing device(s) can access, from a database, a heat map generated from previously collected sensor data associated with the environment. The computing device(s) can perform a look-up using the heat map based at least partly on the location of the entity and the first characteristic, and can determine a predicted behavior of the entity at a predetermined future time based on a pattern of behavior associated with a cell in the heat map.