Abstract: Object recognition and classification based on data from multiple sensor modalities is disclosed. A computing system can detect an event in a monitored portion of an environment coordinate field associated with a vehicle. The computing system can retrieve data associated with the detected event from a plurality of modalities. At least one of the modalities can include raw data from a sensor associated with the vehicle. The computing system can classify the detected event based at least in part on the retrieved data and one or more parameters of a classifier.

Abstract: Systems, methods, and computer-readable storage mediums which use annotated environmental models for sensor modification are disclosed. A computing system receives an environmental model for a vehicle. The environmental model can include data from a plurality of modalities. Object annotations are received from sensors which are used to modify the environmental model, creating an annotated environmental model. A classification of a current situation is generated using the annotated environmental model, and a sensor is modified using the classification.

Abstract: Determining a situational annotation of a vehicle based on an annotated environmental model is disclosed. A computing system receives an annotated environmental model for a vehicle. The annotated environmental model can include data from a plurality of modalities. The computing system uses a classifier to determine an annotation of the vehicle situation based, at least in part, on the annotated environmental model. The computing system provides the situational annotation to a driving functionality system. The driving functionality system is configured to use the situational annotation to generate signals to control operation the vehicle.

Abstract: Object recognition and classification based on data from multiple sensor modalities is disclosed. A computing system can detect an event in a monitored portion of an environment coordinate field associated with a vehicle. The computing system can retrieve data associated with the detected event from a plurality of modalities. At least one of the modalities can include raw data from a sensor associated with the vehicle. The computing system can classify the detected event based at least in part on the retrieved data and one or more parameters of a classifier.