Abstract: A system and method for providing an agent action anticipative transformer that include receiving image data associated with a video of a surrounding environment of an ego agent. The system and method additionally include analyzing the image data and extracting short range clips from the image data. The system and method also include analyzing the short range clips and extracting clip-level features associated with each of the short range clips. The system and method further include executing self-supervision using causal masking with respect to the extracted clip-level features to output action predictions and feature predictions to enable ego-centric action anticipation with respect to at least one target agent to autonomously control the ego agent.

Abstract: A system and method for providing long term and key intentions for trajectory prediction that include receiving image data and LiDAR data associated with RGB images and LiDAR point clouds that are associated with a surrounding environment of an ego agent and processing a long term and key intentions for trajectory prediction dataset (LOKI dataset) that is utilized to complete joint trajectory and intention prediction for heterogeneous traffic agents. The system and method also include encoding a past observation history of each of the heterogeneous traffic agents and sampling a respective goal. The system and method further include decoding and predicting future trajectories associated with each of the heterogeneous traffic agents based on data included within the LOKI dataset, the encoded past observation history, and the respective goal.