Abstract: Methods and systems for training an autonomous driving, agent-centric vison-language planning (VLP) machine learning model. Image data is obtained from a vehicle-mounted camera, encompassing details about agents situated within the external environment. Via image processing, the system identifies these agents within the environment. A Bird's Eye View (BEV) representation of the surroundings is then generated, encapsulating BEV features including spatiotemporal information linked to the vehicle and the recognized agents. Executing the VLP model begins by first extracting agent-wise BEV features from the BEV, wherein the agent-wise BEV features are associated with respective agents in the environment. Agent-wise text features are extracted from natural language text prompts. A contrastive learning model derives similarities between the agent-wise BEV features and the agent-wise text features.

Abstract: A system for training a drive system for an autonomous vehicle (AV) includes one or more computing devices configured to output a first drive plan and a second drive plan using an initial autonomous drive system (ADS) and an initial behavior foundation system (BFS). The computing devices is configured to generate a system-to-system (SS) loss between the initial ADS and the initial BFS using data from the respective system, and generate a module task loss for each system using respective drive plan and respective ground truth data. The computing devices is configured to adjust tunable parameters of the initial ADS and/or tunable parameters of the initial BFS to reduce a total loss provided by the SS loss and the module task loss. The initial ADS and/or the initial BFS is outputted as a trained drive system to be employed for the AV in response to the total loss being reduced.

Abstract: A Bird's Eye View (BEV)-based object detection framework for autonomous vehicles. The disclosed embodiments pretrain a diffusion model system on BEV representations generated from sensor data, such as cameras, LiDARs, and radars. The pretrained diffusion model system may be integrated into the BEV generation network to denoise BEV features through a supervision loss mechanism during training. This approach enhances the quality of BEV representations used in downstream tasks, such as object detection and trajectory prediction, without introducing any incremental computational cost during run-time.

Abstract: Methods and systems for training an end-to-end autonomous driving system using a vision-language planning (VLP) machine learning model in a closed-loop environment. Images associated with an environment about a vehicle are generated, and a BEV model is executed to generate a BEV view based on the images. A planning model predicts navigation trajectories based on the BEV. The VLP model enhances the system by extracting vision-based planning features, generating text prompts, and employing a language encoder to create text-based expectation features. A contrastive learning model identifies similarities between vision and text features, boosting the performance of the BEV and planning models. The system undergoes closed-loop evaluation in a simulated environment, capturing metrics to refine the autonomous driving system.

Abstract: Methods and systems for training an autonomous driving system using a vision-language planning (VLP) model. Image data is obtained from a vehicle-mounted camera, encompassing details about agents situated within the external environment. Via image processing, the system identifies these agents within the environment. A Bird's Eye View (BEV) representation of the surroundings is then generated, encapsulating the spatiotemporal information linked to the vehicle and the recognized agents. Execution of the VLP machine learning model begins by extracting vision-based planning features from the BEV, and receiving or generating textual information characterizing various attributes of the vehicle within the environment. Text-based planning features are extracted from this textual information. To enhance model performance, a contrastive learning model is engaged to establish similarities between the vision-based and text-based planning features, and a predicted trajectory is output based on the similarities.

Abstract: Methods and systems for training an autonomous driving system using a vision-language planning (VLP) model. Image data is obtained from a vehicle-mounted camera, encompassing details about agents situated within the external environment. Via image processing, the system identifies these agents within the environment. A Bird's Eye View (BEV) representation of the surroundings is then generated, encapsulating the spatiotemporal information linked to the vehicle and the recognized agents. Execution of the VLP machine learning model begins by extracting vision-based planning features from the BEV, and receiving or generating textual information characterizing various attributes of the vehicle within the environment. Text-based planning features are extracted from this textual information. To enhance model performance, a contrastive learning model is engaged to establish similarities between the vision-based and text-based planning features, and a predicted trajectory is output based on the similarities.

Abstract: Methods and systems for training an autonomous driving, agent-centric vison-language planning (VLP) machine learning model. Image data is obtained from a vehicle-mounted camera, encompassing details about agents situated within the external environment. Via image processing, the system identifies these agents within the environment. A Bird's Eye View (BEV) representation of the surroundings is then generated, encapsulating BEV features including spatiotemporal information linked to the vehicle and the recognized agents. Executing the VLP model begins by first extracting agent-wise BEV features from the BEV, wherein the agent-wise BEV features are associated with respective agents in the environment. Agent-wise text features are extracted from natural language text prompts. A contrastive learning model derives similarities between the agent-wise BEV features and the agent-wise text features.

Abstract: A system and approach for automatically detecting and classifying thermal anomalies in thermal images of structures such as buildings. The system that can receive a thermal image, autonomously process the image using a machine learning algorithm specifically designed and trained for detecting and classifying thermal anomalies, and then display the classified anomalies. The machine learning algorithm may comprise a neural network and, in one example, may be a prediction-tuning capsule network employing two instance layers, namely, a fully connected PT capsule layer and a locally connected PT capsule layer. The machine learning algorithm may comprise a transformer-based segmentation method for the autonomous heat anomaly segmentation task.