Abstract: A system and method for providing object-level driver attention reasoning with a graph convolution network that include receiving image data associated with a plurality of image clips of a surrounding environment of a vehicle and determining anchor objectness scores and anchor importance scores associated with relevant objects included within the plurality of image clips. The system and method also include analyzing the anchor objectness scores and anchor importance scores associated with relevant objects and determining top relevant objects with respect to an operation of the vehicle. The system and method further include passing object node features and edges of an interaction graph through the graph convolution network to update features of each object node through interaction with other object nodes and determining importance scores for the top relevant objects.

Abstract: A system and method for providing object-level driver attention reasoning with a graph convolution network that include receiving image data associated with a plurality of image clips of a surrounding environment of a vehicle and determining anchor objectness scores and anchor importance scores associated with relevant objects included within the plurality of image clips. The system and method also include analyzing the anchor objectness scores and anchor importance scores associated with relevant objects and determining top relevant objects with respect to an operation of the vehicle. The system and method further include passing object node features and edges of an interaction graph through the graph convolution network to update features of each object node through interaction with other object nodes and determining importance scores for the top relevant objects.

Abstract: The present disclosure provides a method and system to operationalize driver eye movement data analysis based on moving objects of interest. Correlation and/or regression analyses between indirect (e.g., eye-tracking) and/or direct measures of driver awareness may identify variables that feature spatial and/or temporal aspects of driver eye glance behavior relative to objects of interest. The proposed systems and methods may be further combined with computer-vision techniques such as object recognition to (e.g., fully) automate eye movement data processing as well as machine learning approaches to improve the accuracy of driver awareness estimation.

Abstract: A system and method for providing object-level driver attention reasoning with a graph convolution network that include receiving image data associated with a plurality of image clips of a surrounding environment of a vehicle and determining anchor object-ness scores and anchor importance scores associated with relevant objects included within the plurality of image clips. The system and method also include analyzing the anchor object-ness scores and anchor importance scores associated with relevant objects and determining top relevant objects with respect to an operation of the vehicle. The system and method further include passing object node features and edges of an interaction graph through the graph convolution network to update features of each object node through interaction with other object nodes and determining importance scores for the top relevant objects.

Abstract: Aspects of the present disclosure may include methods, apparatuses, and computer readable media for receiving one or more images having a plurality of objects, receiving a notification from an occupant of the self-driving vehicle, generating an attention map highlighting the plurality of objects based on at least one of the one or more images and the notification, and providing at least one of a steering control or a velocity control to operate the self-driving vehicle based on the attention map and the notification.

Abstract: Determining object importance in vehicle control systems can include obtaining, for a vehicle in operation, an image of a dynamic scene, identifying an object type associated with one or more objects in the image, determining, based on the object type and a goal associated with the vehicle, an importance metric associated with the one or more objects, and controlling the vehicle based at least in part on the importance metric associated with the one or more objects.

Abstract: A system and method for outputting vehicle dynamic controls using deep neural networks that include receiving environmental sensor data from at least one sensor of a vehicle of a surrounding environment of the vehicle. The system and method also include inputting the environmental sensor data to a primary deep neural network structure and inputting intermediate representation, at least one applicable traffic rule, and at least one applicable vehicle maneuver to a secondary deep neural network structure. The system and method further include outputting vehicle dynamic controls to autonomously control the vehicle to navigate within the surrounding environment of the vehicle based on the at least one applicable traffic rule and the at least one applicable vehicle maneuver.

Abstract: A system and method for providing context aware road user importance estimation that include receiving at least one image of a vicinity of an ego vehicle. The system and method also include analyzing the at least one image to determine a local context associated with at least one road user located within the vicinity of the ego vehicle. The system and method additionally include determining a global context associated with the ego vehicle. The system and method further include fusing the local context and the global context to classify at least one highly important road user that is to be accounted for with respect to operating the ego vehicle.

Abstract: Systems and techniques for autonomous vehicle decision making may include training an autonomous vehicle decision making database by capturing an image including a first training object and a second training object during a training phase. The first training object may be classified as a first class and the second training object may be classified as a second class based on a driver gaze location associated with a driver of the vehicle. The database may be built based on classification of the first training object and the second training object. The autonomous vehicle decision making database may be utilized to classify a first object as a first class and a second object as a second class during an operation phase. A processor may perform a first computation associated with the first object based on the classification of the first object and the classification of the second object.

Abstract: The present disclosure provides a method and system to operationalize driver eye movement data analysis based on moving objects of interest. Correlation and/or regression analyses between indirect (e.g., eye-tracking) and/or direct measures of driver awareness may identify variables that feature spatial and/or temporal aspects of driver eye glance behavior relative to objects of interest. The proposed systems and methods may be further combined with computer-vision techniques such as object recognition to (e.g., fully) automate eye movement data processing as well as machine learning approaches to improve the accuracy of driver awareness estimation.

Abstract: A system and method for providing object-level driver attention reasoning with a graph convolution network that include receiving image data associated with a plurality of image clips of a surrounding environment of a vehicle and determining anchor object-ness scores and anchor importance scores associated with relevant objects included within the plurality of image clips. The system and method also include analyzing the anchor object-ness scores and anchor importance scores associated with relevant objects and determining top relevant objects with respect to an operation of the vehicle. The system and method further include passing object node features and edges of an interaction graph through the graph convolution network to update features of each object node through interaction with other object nodes and determining importance scores for the top relevant objects.

Abstract: Saliency training may be provided to build a saliency database, which may be utilized to facilitate operation of an autonomous vehicle. The saliency database may be built by minimizing a loss function between a saliency prediction result and a saliency mapper result. The saliency mapper result may be obtained from a ground truth database, which includes image frames of an operation environment where objects or regions within respective image frames are associated with a positive saliency, a neutral saliency, or a negative saliency. Neutral saliency may be indicative of a detected gaze location of a driver corresponding to the object or region at a time prior to the time associated with a given image frame. The saliency prediction result may be generated based on features extracted from respective image frames, depth-wise concatenations associated with respective image frames, and a long short-term memory layer or a recurrent neural network.

Abstract: Aspects of the present disclosure may include methods, apparatuses, and computer readable media for receiving one or more images having a plurality of objects, receiving a notification from an occupant of the self-driving vehicle, generating an attention map highlighting the plurality of objects based on at least one of the one or more images and the notification, and providing at least one of a steering control or a velocity control to operate the self-driving vehicle based on the attention map and the notification.

Abstract: A trainer device trains an automated driver system. The trainer device may include a vehicle manager that manages data associated with controlling a vehicle and a simulation manager that manages data associated with simulating the vehicle. The vehicle manager may analyze vehicle data to identify an intervention event, and the simulation manager obtains a portion of the vehicle data corresponding to the intervention event to generate simulation data, obtains user data associated with the simulation data, analyzes the user data to determine whether the user data satisfies a predetermined intervention threshold, and, on condition that the user data satisfies the predetermined intervention threshold, transmits the user data to the vehicle manager for modifying the first control data.

Abstract: Systems and methods for estimating an object of fixation from a gaze of a driver. The method includes: receiving image data from a plurality of input devices; processing the image data from a first one of the plurality of input devices to identify an object track; analyzing the image data from a second one of the plurality of input devices and the image data from the first one of the plurality of input devices to determine a projected gaze of a driver; analyzing the object track and the projected gaze to identify a plurality of objects in the gaze of the driver; performing a probability analysis to estimate the object of fixation from among the plurality of objects; and generating an output image identifying the estimated object of fixation.

Abstract: Determining object importance in vehicle control systems can include obtaining, for a vehicle in operation, an image of a dynamic scene, identifying an object type associated with one or more objects in the image, determining, based on the object type and a goal associated with the vehicle, an importance metric associated with the one or more objects, and controlling the vehicle based at least in part on the importance metric associated with the one or more objects.

Abstract: A system and method for outputting vehicle dynamic controls using deep neural networks that include receiving environmental sensor data from at least one sensor of a vehicle of a surrounding environment of the vehicle. The system and method also include inputting the environmental sensor data to a primary deep neural network structure and inputting intermediate representation, at least one applicable traffic rule, and at least one applicable vehicle maneuver to a secondary deep neural network structure. The system and method further include outputting vehicle dynamic controls to autonomously control the vehicle to navigate within the surrounding environment of the vehicle based on the at least one applicable traffic rule and the at least one applicable vehicle maneuver.

Abstract: The systems and methods provided herein are directed to the uploading and transmission of vehicle data to a remote system when a physiological event for a driver has been detected using one or more sensors. Information such as the driver's heart rate, temperature, voice inflection or facial expression may be monitored to detect the physiological event. Vehicle data, such as gathering or control system data, may be sent once the event has been detected. Selected vehicle data associated with the event or all data during the time of the event may be sent. After receiving the vehicle data, the remote system may process or store it where it may be used to modify automated driving functionalities.

Abstract: A system and method for providing context aware road user importance estimation that include receiving at least one image of a vicinity of an ego vehicle. The system and method also include analyzing the at least one image to determine a local context associated with at least one road user located within the vicinity of the ego vehicle. The system and method additionally include determining a global context associated with the ego vehicle. The system and method further include fusing the local context and the global context to classify at least one highly important road user that is to be accounted for with respect to operating the ego vehicle.

Abstract: Systems and techniques for autonomous vehicle decision making may include training an autonomous vehicle decision making database by capturing an image including a first training object and a second training object during a training phase. The first training object may be classified as a first class and the second training object may be classified as a second class based on a driver gaze location associated with a driver of the vehicle. The database may be built based on classification of the first training object and the second training object. The autonomous vehicle decision making database may be utilized to classify a first object as a first class and a second object as a second class during an operation phase. A processor may perform a first computation associated with the first object based on the classification of the first object and the classification of the second object.