Abstract: In various examples, interior sensor calibration using exterior features for autonomous systems and applications is described herein. Systems and methods are disclosed that use one or more features that are located exterior to a vehicle, such as one or more tags located in the environment surrounding the vehicle, to determine one or more values for one or more calibration parameters that calibrate an interior sensor of the vehicle with respect to a reference coordinate system of the vehicle. For instance, such as when the vehicle is located at a calibration station, the sensor may generate sensor data representing a sensor representation, where at least a portion of the sensor representation is associated with a feature that is visible through a transparent component (e.g., a window) of the vehicle. The sensor data may then be used to calibrate the sensor with respect to the coordinate system.

Abstract: In various examples, interior sensor calibration for autonomous systems and applications is described herein. Systems and methods are disclosed that may recalibrate sensors of a vehicle, such as sensors located within the interior of the vehicle, using one or more techniques. For instance, if a sensor is attached to a component within the interior of the vehicle, an additional sensor associated with the component may output data indicating the location and/or orientation of the component within the vehicle. The indicated location and/or orientation of the component may then be used to recalibrate the sensor with respect to a reference coordinate system of the vehicle. For a second example, the sensor may output data representing at least a feature located within the interior of the vehicle. The sensor may then again be recalibrated based at least on a portion of the data that represents the feature.

Abstract: In various examples, infrared image data may be used to detect a subcutaneous characteristic(s) (e.g., a palm vein topology) of a person (e.g., a person requesting entry to a vehicle, a vehicle occupant) and authenticate the user based on the detected subcutaneous characteristic(s). For example, infrared image data representing one or more acquired subcutaneous characteristics (e.g., a topology of veins and/or other blood vessels in a region of the authenticating user's palm, hand, neck, forearm, face, fingertip, eye, etc.) may be generated. Hand and/or palm detection may be applied to detect a region depicting the user's hand or palm, and that region (or some subset thereof) may be segmented to generate a representation of an acquired vein topology. The acquired vein topology may be compared with one or more reference vein topologies stored in a database to determine whether the acquired vein topology matches one of the reference vein topologies.

Abstract: In various examples, image-based three-dimensional occupant assessment for in-cabin monitoring systems and applications are provided. An evaluation function may determine a 3D representation of an occupant of a machine by evaluating sensor data comprising an image frame from an optical image sensor. The 3D representation may comprise at least one characteristic representative of a size of the occupant, (e.g., a 3D pose and/or 3D shape), which may be used to derive other characteristics such as, but not limited to weight, height, and/or age. A first processing path may generate a representation of one or more features corresponding to at least a portion of the occupant based on optical image data, and a second processing path may determine a depth corresponding to the one or more features based on depth data derived from the optical image data and ground truth depth data corresponding to the interior of the machine.

Abstract: In various examples, occupant assessment using multi-modal sensor fusion for monitoring systems and applications are provided. In some embodiments, an occupant monitoring system comprises an occupant evaluation function that may predict at least one characteristic representative of a size of the occupant. The occupant evaluation function may include a first processing path that generates a representation of features corresponding to the occupant based on optical image data, and a second processing path that performs operations to determine a depth corresponding to the one or more features based on depth data derived from the optical image data and the point cloud depth data. In some embodiments, a three-dimensional pose detection model generates a three-dimensional pose estimate of the occupant using the optical image data, and the three-dimensional pose estimate is scaled to an absolute pose based on the point cloud depth data.

Abstract: In various examples, systems and methods for pose detection model training for predicting three-dimensional pose estimates using two-dimensional image data are provided. The occupant pose detection model may be trained using multi-view image sensor training data that includes image frames that capture a pose of a training subject within a machine interior using multiple synchronized optical image sensors placed around the machine interior that produce a set of captured image frames of the training subject from different viewpoints. Based on the multi-view image sensor training data, the occupant pose detection model may generate a set of individual, predicted 3D pose estimates for the training subject from a captured image frame from each of the respective optical image sensors. To adjust the occupant pose detection model during training, a loss feedback may be generated that comprises a pose alignment loss, a pose depth loss, and/or a ground truth kinematic loss.

Abstract: In various examples, sensor data (e.g., image and/or RADAR data) may be used to detect occupants and classify them (e.g., as children or adults) using one or more predictions that represent estimated age (e.g., based on detected limb length, a detected face) and/or detected child presence (e.g., based on detecting an occupied child seat). In some embodiments, multiple predictions generated using multiple machine learning models (and optionally one or more corresponding confidence values) may be combined using a state machine and/or one or more machine learning models to generate a combined assessment of occupant presence and/or age for each occupant and/or supported occupant slot. As such, the techniques described herein may be utilized to detect child presence, detect unattended child presence, determine age or size of a particular occupant, and/or take some responsive action (e.g., trigger an alarm, control temperature, unlock door(s), permit or disable airbag deployment, etc.).

Abstract: In various examples, low power proximity based threat detection using optical flow for vehicle systems and applications are provided. Some embodiments may use a tiered framework that uses sensor fusion techniques to detect and track the movement of a threat candidate, and perform a threat classification and/or intent prediction as the threat candidate approaches approach. Relative depth indications from optical flow, computed using data from image sensors, can be used to initially segment and track a moving object over a sequence of image frames. Additional sensors and processing may be brought online when a moving object becomes close enough to be considered a higher risk threat candidate. A threat response system may generate a risk score based on a predicted intent of a threat candidate, and when the risk score exceeds a certain threshold, then the threat response system may respond accordingly based on the threat classification and/or risk score.

Abstract: Systems and methods for more accurate and robust determination of subject characteristics from an image of the subject. One or more machine learning models receive as input an image of a subject, and output both facial landmarks and associated confidence values. Confidence values represent the degrees to which portions of the subject's face corresponding to those landmarks are occluded, i.e., the amount of uncertainty in the position of each landmark location. These landmark points and their associated confidence values, and/or associated information, may then be input to another set of one or more machine learning models which may output any facial analysis quantity or quantities, such as the subject's gaze direction, head pose, drowsiness state, cognitive load, or distraction state.

Abstract: In various examples, systems and methods that use dialogue systems associated with various machine systems and applications are described. For instance, the systems and methods may receive text data representing speech, such as a question associated with a vehicle or other machine type. The systems and methods then use a retrieval system(s) to retrieve a question/answer pair(s) associated with the text data and/or contextual information associated with the text data. In some examples, the contextual information is associated with a knowledge base associated with or corresponding to the vehicle. The systems and methods then generate a prompt using the text data, the question/answer pair(s), and/or the contextual information. Additionally, the systems and methods determine, using a language model(s) and based at least on the prompt, an output associated with the text data. For instance, the output may include information that answers the question associated with the vehicle.

Abstract: Systems and methods for more accurate and robust determination of subject characteristics from an image of the subject. One or more machine learning models receive as input an image of a subject, and output both facial landmarks and associated confidence values. Confidence values represent the degrees to which portions of the subject's face corresponding to those landmarks are occluded, i.e., the amount of uncertainty in the position of each landmark location. These landmark points and their associated confidence values, and/or associated information, may then be input to another set of one or more machine learning models which may output any facial analysis quantity or quantities, such as the subject's gaze direction, head pose, drowsiness state, cognitive load, or distraction state.

Abstract: In various examples, techniques for using scene-aware context for dialogue systems and applications are described herein. For instance, systems and methods are disclosed that process audio data representing speech in order to determine an intent associated with the speech. Systems and methods are also disclosed that process sensor data representing at least a user in order to determine a point of interest associated with the user. In some examples, the point of interest may include a landmark, a person, and/or any other object within an environment. The systems and methods may then generate a context associated with the point of interest. Additionally, the systems and methods may process the intent and the context using one or more language models. Based on the processing, the language model(s) may output data associated with the speech.

Abstract: In various examples, techniques are described for detecting whether spoofing attacks are occurring using multiple sensors. Systems and methods are disclosed that include at least a first sensor having a first pose to capture a first perspective view of a user and a second sensor having a second pose to capture a second perspective view of the user. The first sensor and/or the second sensor may include an image sensor, a depth sensor, and/or the like. The systems and methods include a neural network that is configured to analyze first sensor data generated by the first sensor and second sensor data generated by the second sensor to determine whether a spoofing attack is occurring. The systems and methods may also perform one or more processes, such as facial recognition, based on whether the spoofing attack is occurring.

Abstract: Systems and methods for determining the gaze direction of a subject and projecting this gaze direction onto specific regions of an arbitrary three-dimensional geometry. In an exemplary embodiment, gaze direction may be determined by a regression-based machine learning model. The determined gaze direction is then projected onto a three-dimensional map or set of surfaces that may represent any desired object or system. Maps may represent any three-dimensional layout or geometry, whether actual or virtual. Gaze vectors can thus be used to determine the object of gaze within any environment. Systems can also readily and efficiently adapt for use in different environments by retrieving a different set of surfaces or regions for each environment.

Abstract: In various examples, an adaptive eye tracking machine learning model engine (“adaptive-model engine”) for an eye tracking system is described. The adaptive-model engine may include an eye tracking or gaze tracking development pipeline (“adaptive-model training pipeline”) that supports collecting data, training, optimizing, and deploying an adaptive eye tracking model that is a customized eye tracking model based on a set of features of an identified deployment environment. The adaptive-model engine supports ensembling the adaptive eye tracking model that may be trained on gaze vector estimation in surround environments and ensemble based on a plurality of eye tracking variant models and a plurality of facial landmark neural network metrics.

Abstract: In various examples, a background of an object may be modified to generate a training image. A segmentation mask may be generated and used to generate an object image that includes image data representing the object. The object image may be integrated into a different background and used for data augmentation in training a neural network. Data augmentation may also be performed using hue adjustment (e.g., of the object image) and/or rendering three-dimensional capture data that corresponds to the object from selected views. Inference scores may be analyzed to select a background for an image to be included in a training dataset. Backgrounds may be selected and training images may be added to a training dataset iteratively during training (e.g., between epochs). Additionally, early or late fusion nay be employed that uses object mask data to improve inferencing performed by a neural network trained using object mask data.

Abstract: Systems and methods for determining the gaze direction of a subject and projecting this gaze direction onto specific regions of an arbitrary three-dimensional geometry. In an exemplary embodiment, gaze direction may be determined by a regression-based machine learning model. The determined gaze direction is then projected onto a three-dimensional map or set of surfaces that may represent any desired object or system. Maps may represent any three-dimensional layout or geometry, whether actual or virtual. Gaze vectors can thus be used to determine the object of gaze within any environment. Systems can also readily and efficiently adapt for use in different environments by retrieving a different set of surfaces or regions for each environment.

Abstract: Systems and methods for more accurate and robust determination of subject characteristics from an image of the subject. One or more machine learning models receive as input an image of a subject, and output both facial landmarks and associated confidence values. Confidence values represent the degrees to which portions of the subject's face corresponding to those landmarks are occluded, i.e., the amount of uncertainty in the position of each landmark location. These landmark points and their associated confidence values, and/or associated information, may then be input to another set of one or more machine learning models which may output any facial analysis quantity or quantities, such as the subject's gaze direction, head pose, drowsiness state, cognitive load, or distraction state.

Abstract: In various examples, an adaptive eye tracking machine learning model engine (“adaptive-model engine”) for an eye tracking system is described. The adaptive-model engine may include an eye tracking or gaze tracking development pipeline (“adaptive-model training pipeline”) that supports collecting data, training, optimizing, and deploying an adaptive eye tracking model that is a customized eye tracking model based on a set of features of an identified deployment environment. The adaptive-model engine supports ensembling the adaptive eye tracking model that may be trained on gaze vector estimation in surround environments and ensemble based on a plurality of eye tracking variant models and a plurality of facial landmark neural network metrics.

Abstract: Systems and methods for more accurate and robust determination of subject characteristics from an image of the subject. One or more machine learning models receive as input an image of a subject, and output both facial landmarks and associated confidence values. Confidence values represent the degrees to which portions of the subject's face corresponding to those landmarks are occluded, i.e., the amount of uncertainty in the position of each landmark location. These landmark points and their associated confidence values, and/or associated information, may then be input to another set of one or more machine learning models which may output any facial analysis quantity or quantities, such as the subject's gaze direction, head pose, drowsiness state, cognitive load, or distraction state.