Abstract: According to aspects described herein, a device can extract first features from frames of first sensor data and second features from frames of second sensor data (captured after the first sensor data). The device can obtain first weighted features based on the first features and second weighted features based on the second features. The device can aggregate the first weighted features to determine a first feature vector and the second weighted features to determine a second feature vector. The device can obtain a first transformed feature vector (based on transforming the first feature vector into a coordinate space) and a second transformed feature vector (based on transforming the second feature vector into the coordinate space). The device can aggregate first transformed weighted features (based on the first transformed feature vector) and second transformed weighted features (based on the second transformed feature vector) to determine a fused feature vector.

Abstract: An example device for training a neural network includes a memory configured to store a neural network model for the neural network; and a processing system comprising one or more processors implemented in circuitry, the processing system being configured to: extract image features from an image of an area, the image features representing objects in the area; extract point cloud features from a point cloud representation of the area, the point cloud features representing the objects in the area; add Gaussian noise to a ground truth depth map for the area to generate a noisy ground truth depth map, the ground truth depth map representing accurate positions of the objects in the area; and train the neural network using the image features, the point cloud features, and the noisy ground truth depth map to generate a depth map.

Abstract: An apparatus includes a memory and processing circuitry in communication with the memory. The processing circuitry is configured to process a joint graph representation using a graph neural network (GNN) to form an enhanced graph representation. The joint graph representation includes first features from a voxelized point cloud, and second features from a plurality of camera images. The enhanced graph representation includes enhanced first features and enhanced second features. The processing circuitry is further configured to perform a diffusion processes on the enhanced first features and the enhanced second features of the enhanced graph representation to form a denoised graph representation having denoised first features and denoised second features, and fuse the denoised first features and the denoised second features of the denoised graph representation using a graph attention network (GAT) to form a fused point cloud having fused features.

Abstract: A method includes receiving a plurality of images, wherein a first image of the one or more images comprises a range image and a second image comprises a camera image and filtering the first image to generate a filtered first image. The method also includes generating a plurality of depth estimates based on the second image and generating an attention map by combining the filtered first image and the plurality of depth estimates. Additionally, the method includes generating a consistency score indicative of a consistency of depth estimates between the first image and the second image based on the attention map, modulating one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features, and generating a classification of one or more soiled regions in the second image based on the modulated one or more features.

Abstract: This disclosure provides systems, methods, and devices for processing and aligning sensor data features for navigation. In a first aspect, a method is provided that includes determining, based on received sensor data, a first set of features for an area surrounding the vehicle. A second set of features for the area surrounding the vehicle may be determined based on an occupancy map for the area surrounding the vehicle. A third set of features may be determined that align the first set of features with the second set of features. The third set of features may align each of at least a subset of the second set of features with at least one corresponding feature from the first set of features. Other aspects and features are also claimed and described.

Abstract: Systems that support graph neural network (GNN) implemented multi-modal spatiotemporal fusion are provided. Identifying and tracking an object in images captured by an imaging system is facilitated by generating a graph based on multimodal data received from a plurality of sensors. The graph encodes spatial components and spatial data associated with the images and encodes temporal data associated with the images. Pooled features are generated, through application of a first graph attention network (GAT), by pooling spatial features and temporal features. The spatial features are based on the spatial component and on the spatial relationship, and the temporal features are based on the temporal relationship. A three dimensional bounding box associated with the object is decoded by propagating the pooled features through a fully connected layer.

Abstract: A method includes receiving one or more images, wherein at least one of the one or more images depicts a water region and analyzing, by one or more processors, the one or more images using a first machine learning model to determine a depth of the water region. The method also includes analyzing, by the one or more processors, the one or more images using a second machine learning model to determine a surface normal of the water region and performing, by the one or more processors, using a third machine learning model, multi-class segmentation of the one or more images. Additionally, the method includes performing one or more fusion operations on outputs of at least two of the first machine learning model, the second machine learning model and the third machine learning model to generate a classification of the water region.

Abstract: A method of image processing includes receiving first feature data from image content captured with a sensor, the first feature data having a first set of states with values that change non-linearly over time, generating second feature data based at least in part on the first feature data, the second feature data having a second set of states with values that change approximately linearly over time relative to a linear operator, wherein the second set of states is greater than the first set of states, and predicting movement of one or more objects in the image content based at least in part on the second feature data.

Abstract: Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. A set of object detections, each respective object detection in the set of object detections corresponding to a respective object detected in an environment, is accessed. Based on the set of object detections, a graph representation comprising a plurality of nodes is generated, where each respective node in the plurality of nodes corresponds to a respective object detection in the set of object detections. A set of output features is generated based on processing the graph representation using a trained message passing network. A predicted object relationship graph is generated based on processing the set of output features using a layer of a trained machine learning model.

Abstract: A method of processing image content includes constructing a first graph representation having a first level of point sparsity from a first point cloud data, and performing diffusion-based upsampling on the first graph representation to generate a second graph representation having a second level of point sparsity. Performing diffusion-based upsampling includes inputting the first graph representation into a diffusion-based trained model to generate a first intermediate graph representation having a first intermediate level of point sparsity, inputting the first intermediate graph representation into the diffusion-based trained model to generate a second intermediate graph representation having a second intermediate level of point sparsity, and generating the second graph representation based on at least on the second intermediate graph representation.

Abstract: An apparatus includes a memory and processing circuitry in communication with the memory. The processing circuitry is configured to apply, based on a positional encoding model, a first feature conditioning module to a set of bird's eye view (BEV) position data features corresponding to position data to generate a set of conditioned BEV position data features, and apply, based on the position encoding model, a second feature conditioning module to a set of perspective image data features corresponding to image data to generate a set of conditioned perspective image data features. The processing circuitry is also configured to generate, based on the positional encoding model, the set of conditioned BEV position data features, and the set of conditioned perspective image data features, a weighted summation. Additionally, the processing circuitry is configured to generate, based on the weighted summation, a set of BEV image data features.

Abstract: A system for processing image data and position data, the system comprising: a memory for storing the image data and the position data; and processing circuitry in communication with the memory. The processing circuitry is configured to: apply a first encoder to extract, from the image data, a first set of features; apply a first decoder to determine, based on the first set of features, a first uncertainty score. Additionally, the processing circuitry is configured to apply a second encoder to extract, from the position data, a second set of features; apply a second decoder to determine, based on the second set of features, a second uncertainty score; and fuse the first set of features and the second set of features based on the first uncertainty score and the second uncertainty score.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of image processing includes receiving a plurality of image frames representative of a scene; receiving point cloud data representative of the scene; determining, using a NeRF model, a three-dimensional reconstruction of the scene based on the plurality of image frames; and outputting fused data that combines first BEV features of the three-dimensional reconstruction of the scene and second BEV features of the point cloud data. Other aspects and features are also claimed and described.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method includes receiving an image frame representing a scene; receiving point cloud data representing the scene; determining first sets of image frame features; determining second sets of point cloud data features based on a plurality of voxels representing the point cloud data; determining a third set of features of the image frame based on a first set of features of the plurality of first sets of features of the image frame and a second set of features of the plurality of second sets of features of the point cloud data; and outputting fused data that combines the third set of features of the image frame and a fourth set of features of the point cloud data. Other aspects and features are also claimed and described.

Abstract: A system for navigation store, in a time-based buffer, a first set of frames acquired by a sensor; stores, in a distance-based buffer, a second set of frames acquired by the sensor, performs moving object segmentation on the first set of frames and the second set of frames to identify at least one moving object in a scene of the frames; predicts a trajectory of the at least one moving object; and performs a navigation function based on the predicted trajectory.

Abstract: In some aspects, a device may obtain, via a camera associated with the device, an image that includes one or more objects located within an area of the device. The device may generate a first three-dimensional output based at least in part on the image. The device may obtain, via an audio component associated with the device, an audio input associated with the one or more objects. The device may generate a second three-dimensional output based at least in part on the audio input. The device may detect the one or more objects based at least in part on the first three-dimensional output and the second three-dimensional output. Numerous other aspects are described.

Abstract: An apparatus for multi-object tracking determines a current representation of a current object in a current image. The apparatus computes a joint Gaussian distribution between the current representation of the current object and a previous representation stored in one or more memory buffers, wherein the previous representation was determined from a previous image. The apparatus updates the one or more memory buffers based on the joint Gaussian distribution. For example, the apparatus determines whether to remove or replace the previous representation in the one or more memory buffers based on values of a covariance matrix of the joint Gaussian distribution.

Abstract: A method of processing image data includes receiving, with a frame correction machine-learning (ML) model executing on processing circuitry, an image frame captured from a first camera of a plurality of cameras; performing, with the frame correction ML model executing on the processing circuitry, image frame correction to generate a corrected image frame based on weights or biases of the frame correction ML model applied to two or more of: samples of the image frame, samples of previously captured image frames from the first camera, or samples from image frames from other cameras of the plurality of cameras; and performing, with the processing circuitry, post-processing based on the corrected image frame.

Abstract: This disclosure provides systems, methods, and devices that support image processing. In a first aspect, a method for multi-sensor fusion includes receiving first information indicative of a first set of BEV features of image data captured by an image sensor; receiving second information indicative of a second set of BEV features of non-image sensor data captured by a non-image sensor; and determining fused data that combines the image data and the non-image sensor data based on the first information, the second information, and third information indicative of differences between BEV features of training data and the first set of BEV features and the second set of BEV features. The BEV features of the training data include a third set of BEV features associated with the image sensor and a fourth set of BEV features associated with the non-image sensor. Other aspects and features are also claimed and described.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of image processing includes receiving an image frame from an image sensor of a camera; receiving an indicator associated with a type of lens of the camera; determining a first tensor grid associated with the indicator, the first tensor grid including a plurality of image framework positions associated with the type of lens; and determining, using a machine learning model, a BEV feature map corresponding to the image frame based on features of the image frame and the first tensor grid. Other aspects and features are also claimed and described.