Abstract: In some aspects of the disclosure, an apparatus includes a processing system that includes one or more processors and one or more memories coupled to the one or more processors. The processing system is configured to receive sensor data associated with a scene and to generate a cylindrical representation associated with the scene. The processing system is further configured to modify the cylindrical representation based on detecting a feature of the cylindrical representation being included in a first region of the cylindrical representation. Modifying the cylindrical representation includes relocating the feature from the first region to a second region that is different than the first region. The processing system is further configured to perform, based on the modified cylindrical representation, one or more three-dimensional (3D) perception operations associated with the scene.

Abstract: Aspects presented herein may improve the accuracy and reliability of object detections performed by multiple object detection models. In one aspect, a UE detects (1) a set of polylines from at least one of a set of bird's eye view (BEV) features or a set of perspective view (PV) features associated with a set of images and (2) a set of three-dimensional (3D) objects in the set of BEV features. The UE associates the set of polylines with the set of 3D objects. The UE updates the set of polylines based on a set of nearby 3D objects or updates the set of 3D objects based on a set of nearby polylines. The UE outputs an indication of the updated set of polylines or the updated set of 3D objects.

Abstract: An apparatus includes a memory and processing circuitry in communication with the memory. The processing circuitry is configured to process a joint graph representation using a graph neural network (GNN) to form an enhanced graph representation. The joint graph representation includes first features from a voxelized point cloud, and second features from a plurality of camera images. The enhanced graph representation includes enhanced first features and enhanced second features. The processing circuitry is further configured to perform a diffusion processes on the enhanced first features and the enhanced second features of the enhanced graph representation to form a denoised graph representation having denoised first features and denoised second features, and fuse the denoised first features and the denoised second features of the denoised graph representation using a graph attention network (GAT) to form a fused point cloud having fused features.

Abstract: According to aspects described herein, a device can extract first features from frames of first sensor data and second features from frames of second sensor data (captured after the first sensor data). The device can obtain first weighted features based on the first features and second weighted features based on the second features. The device can aggregate the first weighted features to determine a first feature vector and the second weighted features to determine a second feature vector. The device can obtain a first transformed feature vector (based on transforming the first feature vector into a coordinate space) and a second transformed feature vector (based on transforming the second feature vector into the coordinate space). The device can aggregate first transformed weighted features (based on the first transformed feature vector) and second transformed weighted features (based on the second transformed feature vector) to determine a fused feature vector.

Abstract: An example device for training a neural network includes a memory configured to store a neural network model for the neural network; and a processing system comprising one or more processors implemented in circuitry, the processing system being configured to: extract image features from an image of an area, the image features representing objects in the area; extract point cloud features from a point cloud representation of the area, the point cloud features representing the objects in the area; add Gaussian noise to a ground truth depth map for the area to generate a noisy ground truth depth map, the ground truth depth map representing accurate positions of the objects in the area; and train the neural network using the image features, the point cloud features, and the noisy ground truth depth map to generate a depth map.

Abstract: Systems and techniques are described herein for processing data. For instance, a method for processing data is provided. The method may include obtaining source features generated based on first sensor data captured using a first set of sensors; obtaining source semantic attributes related to the source features; obtaining target features generated based on second sensor data captured using a second set of sensors; obtaining map information; obtaining location information of a device comprising the second set of sensors; obtaining target semantic attributes from the map information based on the location information; aligning the target features with a set of the source features, based on the source semantic attributes and the target semantic attributes, to generate aligned target features; and processing the aligned target features to generate an output.

Abstract: An apparatus includes a memory for storing image data and position data, wherein the image data comprises a set of two-dimensional (2D) camera images, and wherein the position data comprises a set of three-dimensional (3D) point cloud frames. The apparatus also includes processing circuitry in communication with the memory, wherein the processing circuitry is configured to convert the set of 2D camera images into a first 3D representation of a 3D environment corresponding to the image data and the position data, wherein the set of 3D point cloud frames comprises a second 3D representation of the 3D environment. The processing circuitry is also configured to generate, based on the first 3D representation and the second 3D representation, a set of bird's eye view (BEV) feature kernels in a continuous space; and generate, based on the set of BEV feature kernels, an output.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of image processing includes receiving image BEV features and receiving first radio detection and ranging (RADAR) BEV features. The first RADAR BEV features that are received are determined based on first RADAR data associated with a first data type. First normalized RADAR BEV features are determined, which includes rescaling the first RADAR BEV features using a first attention mechanism based on the image BEV features and the first RADAR BEV features. Fused data is determined that combines the first normalized RADAR BEV features and the image BEV features. Other aspects and features are also claimed and described.

Abstract: This disclosure provides systems, methods, and devices for processing and aligning sensor data features for navigation. In a first aspect, a method is provided that includes determining, based on received sensor data, a first set of features for an area surrounding the vehicle. A second set of features for the area surrounding the vehicle may be determined based on an occupancy map for the area surrounding the vehicle. A third set of features may be determined that align the first set of features with the second set of features. The third set of features may align each of at least a subset of the second set of features with at least one corresponding feature from the first set of features. Other aspects and features are also claimed and described.

Abstract: Systems that support graph neural network (GNN) implemented multi-modal spatiotemporal fusion are provided. Identifying and tracking an object in images captured by an imaging system is facilitated by generating a graph based on multimodal data received from a plurality of sensors. The graph encodes spatial components and spatial data associated with the images and encodes temporal data associated with the images. Pooled features are generated, through application of a first graph attention network (GAT), by pooling spatial features and temporal features. The spatial features are based on the spatial component and on the spatial relationship, and the temporal features are based on the temporal relationship. A three dimensional bounding box associated with the object is decoded by propagating the pooled features through a fully connected layer.

Abstract: A method includes receiving a plurality of images, wherein a first image of the one or more images comprises a range image and a second image comprises a camera image and filtering the first image to generate a filtered first image. The method also includes generating a plurality of depth estimates based on the second image and generating an attention map by combining the filtered first image and the plurality of depth estimates. Additionally, the method includes generating a consistency score indicative of a consistency of depth estimates between the first image and the second image based on the attention map, modulating one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features, and generating a classification of one or more soiled regions in the second image based on the modulated one or more features.

Abstract: A method of image processing includes receiving first feature data from image content captured with a sensor, the first feature data having a first set of states with values that change non-linearly over time, generating second feature data based at least in part on the first feature data, the second feature data having a second set of states with values that change approximately linearly over time relative to a linear operator, wherein the second set of states is greater than the first set of states, and predicting movement of one or more objects in the image content based at least in part on the second feature data.

Abstract: A method includes receiving one or more images, wherein at least one of the one or more images depicts a water region and analyzing, by one or more processors, the one or more images using a first machine learning model to determine a depth of the water region. The method also includes analyzing, by the one or more processors, the one or more images using a second machine learning model to determine a surface normal of the water region and performing, by the one or more processors, using a third machine learning model, multi-class segmentation of the one or more images. Additionally, the method includes performing one or more fusion operations on outputs of at least two of the first machine learning model, the second machine learning model and the third machine learning model to generate a classification of the water region.

Abstract: Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. A set of object detections, each respective object detection in the set of object detections corresponding to a respective object detected in an environment, is accessed. Based on the set of object detections, a graph representation comprising a plurality of nodes is generated, where each respective node in the plurality of nodes corresponds to a respective object detection in the set of object detections. A set of output features is generated based on processing the graph representation using a trained message passing network. A predicted object relationship graph is generated based on processing the set of output features using a layer of a trained machine learning model.

Abstract: A method of processing image content includes constructing a first graph representation having a first level of point sparsity from a first point cloud data, and performing diffusion-based upsampling on the first graph representation to generate a second graph representation having a second level of point sparsity. Performing diffusion-based upsampling includes inputting the first graph representation into a diffusion-based trained model to generate a first intermediate graph representation having a first intermediate level of point sparsity, inputting the first intermediate graph representation into the diffusion-based trained model to generate a second intermediate graph representation having a second intermediate level of point sparsity, and generating the second graph representation based on at least on the second intermediate graph representation.

Abstract: A system for processing image data and position data, the system comprising: a memory for storing the image data and the position data; and processing circuitry in communication with the memory. The processing circuitry is configured to: apply a first encoder to extract, from the image data, a first set of features; apply a first decoder to determine, based on the first set of features, a first uncertainty score. Additionally, the processing circuitry is configured to apply a second encoder to extract, from the position data, a second set of features; apply a second decoder to determine, based on the second set of features, a second uncertainty score; and fuse the first set of features and the second set of features based on the first uncertainty score and the second uncertainty score.

Abstract: An apparatus includes a memory and processing circuitry in communication with the memory. The processing circuitry is configured to apply, based on a positional encoding model, a first feature conditioning module to a set of bird's eye view (BEV) position data features corresponding to position data to generate a set of conditioned BEV position data features, and apply, based on the position encoding model, a second feature conditioning module to a set of perspective image data features corresponding to image data to generate a set of conditioned perspective image data features. The processing circuitry is also configured to generate, based on the positional encoding model, the set of conditioned BEV position data features, and the set of conditioned perspective image data features, a weighted summation. Additionally, the processing circuitry is configured to generate, based on the weighted summation, a set of BEV image data features.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. A method is disclosed for aligning top-down features from two sensor arrangements and generating vehicle control instructions. The method includes receiving first sensor data from a first sensor arrangement and second sensor data from a second sensor arrangement. The method further includes determining a first set of top-down and a second set of top-down features based on the sensor data. A transformation is determined based on the first set of top-down features and the second set of top-down features to align the second set of top-down features with the first set of top-down features. Finally, vehicle control instructions for a vehicle are determined based on the transformation. Other aspects and features are also claimed and described.

Abstract: A sensor data processing system includes various elements, including a perception unit that collects data representing positions of sensors on a vehicle and obtains environmental information around the vehicle via the sensors. The sensor data processing system also includes a feature fusion unit that combines the first environmental information from the sensors into first fused feature data representing first positions of objects around the vehicle, provides the first fused feature data to the object tracking unit, receives feedback for the first fused feature data from the object tracking unit, and combines second environmental information from the sensors using the feedback into second fused feature data representing second positions of objects around the vehicle. The sensor data processing system may then at least partially control operation of the vehicle using the second fused feature data.

Abstract: Disclosed are techniques for localization of an object. For example, a device can generate, based on sensor data obtained from sensor(s) associated with an object, a predicted map comprising predicted nodes associated with a predicted location of the object within an environment. The device can receive a high definition (HD) map comprising HD nodes associated with a HD location of the object within the environment. The device can further match the predicted nodes with the HD nodes to determine pair(s) of matched nodes between the predicted map and the HD map. The device can determine, based on a comparison between nodes in each pair of the pair(s) of matched nodes, a respective node score for each pair of the pair(s) of matched nodes. The device can determine, based on the respective node score for each pair of the pair(s) of matched nodes, a location of the object within the environment.