Abstract: A computer-implemented system and method relate to gesture recognition. A machine learning system is trained using a training dataset of sensor data that include a set of gestures. The training dataset includes at least a first subset that displays a first gesture. Loss data is generated based on a first loss function that includes a first cross entropy loss and a second cross entropy loss. Parameters of the machine learning system are updated based on the loss data. The machine learning system is outputted and configured for gesture recognition of the set of gestures. The machine learning system includes (i) a first subnetwork to generate feature data based on the sensor data, (ii) a second subnetwork to extract a selected patch of the feature data, and (iii) a third subnetwork to generate gesture data based on a classification of the corresponding feature data of the selected patch. The first cross entropy loss is based on a first performance of the second subnetwork in relation to the training dataset.

Abstract: Methods and systems for generating a HD map and lane trajectory for an autonomous vehicle based on an SD map. Images from one or more image sensors mounted on a vehicle are received. Via a vehicle processor, perception data is generated based on the received images, wherein the perception data provides a representation of an environment proximate to the vehicle. A standard definition (SD) map corresponding with the environment proximate to the vehicle. The vehicle processor generates a high definition (HD) map corresponding with the environment proximate to the vehicle based on the SD map and the perception data. The vehicle processor also generates lane-level trajectory associated with a planned route for the vehicle utilizing the HD map.

Abstract: Methods and system for detecting out-of-distribution data for a neural network. A training dataset includes in-distribution data, for example image data associated with one or more images. The neural network is trained on the in-distribution data, and has a plurality of layers. A subspace of in-distribution data of the training dataset is generated based on a sample of one of the layers trained with the in-distribution data. Input image data associated with a sample image is received, and the neural network is executed on the input image data to determine a gradient associated with the sample image. The gradient is projected into the subspace to derive a projection of the gradient. The image data associated with the sample image is determined to be out of distribution based on a magnitude of the projection of the gradient.

Abstract: A computer-implemented method and system relate to generating a three-dimensional (3D) layout model. Segmentation masks are generated using a digital image. The segmentation masks identify architectural elements in the digital image. Depth data is generated for each segmentation mask. A set of planes is generated using the depth data and the segmentation masks. Boundary estimate data is generated for the set of planes using boundary data of the segmentation masks. A set of plane segments is generated by bounding the set of planes using the boundary estimate data. Boundary tolerance data is generated for each boundary estimate data. A 3D layout model is constructed by generating at least a boundary segment that connects a first bounded plane and a second bounded plane at an intersection, which is located using the boundary estimate data and the boundary tolerance data.

Abstract: A computer-implemented system and method relate an edge device with a local machine learning model, which generates local prediction data and confidence score data, in response to sensor data. Query threshold data is received from a cloud computing system. An assessment result is assessed using the confidence score data and the query threshold data. The assessment result indicates whether or not to generate a query with the sensor data for transmission to the cloud computing system. The local predication data is assigned as a prediction result when the assessment result indicates that the query is not being generated and transmitted. The cloud prediction data is assigned as the prediction result when the assessment result indicates that the query is being generated and transmitted. The cloud prediction data is received from the cloud computing system in response to the query. The edge device is controlled using the prediction result.

Abstract: A computer-implemented system and method relate to managing a cloud computing system. Queries are received from one or more edge devices of a set of edge devices. Each query includes sensor data from the respective edge device. Prediction data is generated via one or more cloud machine learning models, using the sensor data, during a current time period. System state data is generated and indicates a current state of an environment during the current time period. The environment is defined by the cloud computing system and the set of edge devices. A machine learning system generates policy data by optimizing an expected return of a reward with respect to taking a particular action given the system state data. The machine learning system is employed by the cloud computing system. The policy data indicates a recommended action from a set of actions. The cloud computing system performs the recommended action.

Abstract: Methods and systems for training an autonomous driving, agent-centric vison-language planning (VLP) machine learning model. Image data is obtained from a vehicle-mounted camera, encompassing details about agents situated within the external environment. Via image processing, the system identifies these agents within the environment. A Bird's Eye View (BEV) representation of the surroundings is then generated, encapsulating BEV features including spatiotemporal information linked to the vehicle and the recognized agents. Executing the VLP model begins by first extracting agent-wise BEV features from the BEV, wherein the agent-wise BEV features are associated with respective agents in the environment. Agent-wise text features are extracted from natural language text prompts. A contrastive learning model derives similarities between the agent-wise BEV features and the agent-wise text features.

Abstract: Methods and systems for training an autonomous driving system using a vision-language planning (VLP) model. Image data is obtained from a vehicle-mounted camera, encompassing details about agents situated within the external environment. Via image processing, the system identifies these agents within the environment. A Bird's Eye View (BEV) representation of the surroundings is then generated, encapsulating the spatiotemporal information linked to the vehicle and the recognized agents. Execution of the VLP machine learning model begins by extracting vision-based planning features from the BEV, and receiving or generating textual information characterizing various attributes of the vehicle within the environment. Text-based planning features are extracted from this textual information. To enhance model performance, a contrastive learning model is engaged to establish similarities between the vision-based and text-based planning features, and a predicted trajectory is output based on the similarities.

Abstract: A system and method are disclosed herein for developing a machine perception model in the omnidirectional image domain. The system and method utilize the knowledge distillation process to transfer and adapt knowledge from the perspective projection image domain to the omnidirectional image domain. A teacher model is pre-trained to perform the machine perception task in the perspective projection image. A student model is trained by adapting the pre-existing knowledge of the teacher model from the perspective projection image domain to the omnidirectional image domain. By way of this training, the student model learns to perform the same machine perception task, except in the omnidirectional image domain, using limited or no suitably labeled training data in the omnidirectional image domain.

Abstract: A method of performing open world object detection includes receiving object data, that includes embeddings data corresponding to a plurality of embeddings for known objects in a first input image, projecting the embeddings into a hyperbolic embedding space that includes embeddings in a plurality of categories of objects each including one or more classes of objects, regularizing the projected embeddings within the hyperbolic embedding space by moving each of the projected embeddings closer to embeddings in a same category of the plurality of categories and further away from embeddings in different categories of the plurality of categories, receiving an unmatched query corresponding to an object in a second input image, and generating, based on the hyperbolic embedding space including the regularized embeddings, an output signal that indicates whether the object in the second input image corresponds to an unknown object in one of the classes of objects.

Abstract: A method of obtaining an uncertainty attribution of a prediction of objects in an input image includes receiving an input image, generating a prediction of objects in the input image, estimating an uncertainty associated with the prediction of the objects in the input image, calculating an uncertainty attribution that represents regions of the input image that cause the estimated uncertainty, including generating a plurality of adversarial gradients each corresponding to a modification of the input image configured to change the estimated uncertainty, and generating an output indicative of the calculated uncertainty attribution.

Abstract: A method of performing data pre-selection for an object detection system includes receiving a first dataset that includes unlabeled data corresponding to one or more images, providing the first dataset and a plurality of learnable prompt vectors to a pre-training model. The learnable prompt vectors include text inputs. The method further includes generating, using the pre-training model, an unsupervised learning prompt based on the first dataset and the plurality of learnable prompt vectors. The unsupervised learning prompt corresponds to a multi-modal feature of the one or more images of the first dataset. The method further includes extracting features from either of the first dataset and a second dataset based on the unsupervised learning prompt, selecting and labeling a subset of instances of the extracted features, and generating and outputting a labeled dataset based on the labeled subset of instances.

Abstract: In some implementations, the device may receive, for a plurality of stations, processing times indicating a time required for a part to be processed by each station, and waiting times indicating how long the part waited before moving to a subsequent one of the plurality of stations. In addition, the device may determine, cycle times for a predetermined window of time, where the cycle times indicates an average number of parts processed by the plurality of stations during the predetermined window of time. The device may determine one of the stations as a potential bottleneck station. Moreover, the device may display, to a user, the potential bottleneck station as a visualization which includes the processing time, the waiting time, and the cycle time of the potential bottleneck station. Also, the device may receive, from the user, a user feedback related to the potential bottleneck station.

Abstract: Methods and systems for providing an interactive image scene graph pattern search are provided. A user is provide with an image having a plurality of selectable segmented regions therein. The user selects one or more of the segmented regions to build a query graph. Via a graph neural network, matching target graphs are retrieved that contain the query graph from a target graph database. Each matching target graph has matching target nodes that match with the query nodes of the query graph. Matching target images from an image database are associated with the matching target graphs. Embeddings of each of the query nodes and the matching target nodes are extracted. A comparison of the embeddings of each query node with the embeddings of each matching target node is performed. The user interface displays the matching target images that are associated with the matching target graphs.

Abstract: A method for performing at least one perception task associated with autonomous vehicle control includes receiving a first dataset and identifying a first object category of objects associated with the plurality of images, the first object category including a plurality of object types. The method also includes identifying a current statistical distribution of a first object type of the plurality of object types and determining a first distribution difference between the current statistical distribution of the first object type and a standard statistical distribution associated with the first object category. The method also includes, in response to a determination that the first distribution difference is greater than a threshold, generating first object type data corresponding to the first object type, configuring at least one attribute of the first object type data, and generating a second dataset by augmenting the first dataset using the first object type data.

Abstract: A computer-implemented system and method relates to language-guided self-supervised semantic segmentation. A modified image is generated by performing data augmentation on a source image. A machine learning model generates first pixel embeddings based on the modified image. First segment embeddings are generated using the first pixel embeddings. A pretrained vision-language model generates second pixel embeddings based on the source image. Second segment embeddings are generated by applying segment contour data from the first pixel embeddings to the second pixel embeddings after the data augmentation is performed on the second pixel embeddings. Embedding consistent loss data is generated by comparing the first segment embeddings in relation to the second segment embeddings. Combined loss data is generated that includes the embedding consistent loss data. Parameters of the machine learning model are updated based on the combined loss data.

Abstract: A method for training a model for determining graph similarity is disclosed. The method comprises receiving a first graph and a second graph as training inputs, the first graph and the second graph each including nodes connected by edges. The method further comprises applying a model to the first graph and the second graph to determine (i) pairs of aligned nodes between the first graph and the second graph and (ii) a first training loss. The method further comprises generating a first augmented graph by modifying the first graph depending on the pairs of aligned nodes. The method further comprises applying the model to the first graph and the first augmented graph to determine a second training loss. The method further comprises refining the model based on the first training loss and the second training loss.

Abstract: A computer-implemented system and method includes generating first pseudo segment data from a first augmented image and generating second pseudo segment data from a second augmented image. The first augmented image and the second augmented image are in a dataset along with other augmented images. A machine learning system is configured to generate pixel embeddings based on the dataset. The first pseudo segment data and the second pseudo segment data are used to identify a first set of segments that a given pixel belongs with respect to the first augmented image and the second augmented image. A second set of segments is identified across the dataset. The second set of segments do not include the given pixel. A local segmentation loss is computed for the given pixel based on the corresponding pixel embedding that involves attracting the first set of segments while repelling the second set of segments.

Abstract: A computer-implemented system and method relate to gesture recognition. A machine learning system is trained using a training dataset of sensor data that include a set of gestures. The training dataset includes at least a first subset that displays a first gesture. Loss data is generated based on a first loss function that includes a first cross entropy loss and a second cross entropy loss. Parameters of the machine learning system are updated based on the loss data. The machine learning system is outputted and configured for gesture recognition of the set of gestures. The machine learning system includes (i) a first subnetwork to generate feature data based on the sensor data, (ii) a second subnetwork to extract a selected patch of the feature data, and (iii) a third subnetwork to generate gesture data based on a classification of the corresponding feature data of the selected patch. The first cross entropy loss is based on a first performance of the second subnetwork in relation to the training dataset.

Abstract: A computer-implemented system and method relate to gesture recognition. A machine learning model includes a first subnetwork, a second subnetwork, and a third subnetwork. The first subnetwork generates feature data based on sensor data, which includes a gesture. The feature data is divided into a set of patches. The second subnetwork selects a target patch of feature data from among the set of patches. The third subnetwork generates gesture data based on the target patch of feature data. The gesture data identifies the gesture of the sensor data. Command data is generated based on the gesture data. A device is controlled based on the command data.