Abstract: A system and a method are disclosed for detecting a sentiment based in part on visual data. The system receives visual data of an environment generated by one or more sensors, accesses a sequence of AI models. Outputs of earlier models in the sequence act as inputs to one or more later models in the sequence. The sequence of AI models includes one or more frame-based models and one or more temporal-based models. The frame-based model is configured to receive the visual data as input, and extract multiple sets of frame-based features associated with the person based in part on the visual data. The temporal-based model is configured to receive the multiple sets of frame-based features as input, and determine a sentiment of the one or more persons based in part on the multiple sets of frame-based features.

Abstract: A device performs operations including determining a probability that a vulnerable road user (VRU) will continue on a current path (e.g., in connection with controlling an autonomous vehicle). The device receives an image depicting a vulnerable road user (VRU). The device inputs at least a portion of the image into a model, and receives, as output from the model, a plurality of probabilities describing the VRU, each of the probabilities corresponding to a probability that the VRU is in a given state. The device determines, based on at least some of the plurality of probabilities, a probability that the VRU will exhibit a behavior, and outputs the probability that the VRU will exhibit the behavior to a control system.

Abstract: A device performs operations including determining a probability that a vulnerable road user (VRU) will continue on a current path (e.g., in connection with controlling an autonomous vehicle). The device receives an image depicting a vulnerable road user (VRU). The device inputs at least a portion of the image into a model, and receives, as output from the model, a plurality of probabilities describing the VRU, each of the probabilities corresponding to a probability that the VRU is in a given state. The device determines, based on at least some of the plurality of probabilities, a probability that the VRU will exhibit a behavior, and outputs the probability that the VRU will exhibit the behavior to a control system.

Abstract: Systems and methods are disclosed herein for tracking a vulnerable road user (VRU) regardless of occlusion. In an embodiment, the system captures a series of images including the VRU, and inputs each of the images into a detection model. The system receives a bounding box for each of the series of images of the VRU as output from the detection model. The system inputs each bounding box into a multi-task model, and receives as output from the multi-task model an embedding for each bounding box. The system determines, using the embeddings for each bounding box across the series of images, an indication of which of the embeddings correspond to the VRU.

Abstract: Systems and methods are disclosed herein for tracking a vulnerable road user (VRU) regardless of occlusion. In an embodiment, the system captures a series of images including the VRU, and inputs each of the images into a detection model. The system receives a bounding box for each of the series of images of the VRU as output from the detection model. The system inputs each bounding box into a multi-task model, and receives as output from the multi-task model an embedding for each bounding box. The system determines, using the embeddings for each bounding box across the series of images, an indication of which of the embeddings correspond to the VRU.

Abstract: A behavior prediction system predicts human behaviors based on environment-aware information such as camera movement data and geospatial data. The system receives sensor data of a vehicle reflecting a state of the vehicle at a given time and a given location. The system determines a field of concern in images of a video stream and determines one or more portions of images of the video stream that correspond to the field of concern. The system may apply different levels of processing powers to objects in the images based on whether an object is in the field of concern. The system then generates features of objects and identify VRUs from the objects of the video stream. For the identified VRUs, the system inputs a representation of the VRUs and the features into a machine learning model, and outputs from the machine learning model a behavioral risk assessment of the VRUs.

Abstract: An occlusion analysis system improves accuracy of behavior prediction models by generating occlusion parameters that may inform mathematical models to generate more accurate predictions. The occlusion analysis system trains and applies models for generating occlusion parameters, such as a manner in which a person is occluded, occlusion percentage, occlusion type. A behavior prediction system may input the occlusion parameters as well as other parameters relating to activity of the human into a second mathematical model for behavior prediction. The second machine learning model is a higher-level model trained to output a prediction that the human will exhibit a future behavior and a confidence level associated with the prediction. The confidence level is at least partially determined based on the occlusion parameters.

Abstract: Systems and methods are disclosed herein for tracking a vulnerable road user (VRU) regardless of occlusion. In an embodiment, the system captures a series of images including the VRU, and inputs each of the images into a detection model. The system receives a bounding box for each of the series of images of the VRU as output from the detection model. The system inputs each bounding box into a multi-task model, and receives as output from the multi-task model an embedding for each bounding box. The system determines, using the embeddings for each bounding box across the series of images, an indication of which of the embeddings correspond to the VRU.

Abstract: A device performs operations including determining a probability that a vulnerable road user (VRU) will continue on a current path (e.g., in connection with controlling an autonomous vehicle). The device receives an image depicting a vulnerable road user (VRU). The device inputs at least a portion of the image into a model, and receives, as output from the model, a plurality of probabilities describing the VRU, each of the probabilities corresponding to a probability that the VRU is in a given state. The device determines, based on at least some of the plurality of probabilities, a probability that the VRU will exhibit a behavior, and outputs the probability that the VRU will exhibit the behavior to a control system.