Abstract: This disclosure provides systems, methods, and devices that support image processing. In a first aspect, a method for multi-sensor fusion includes receiving first information indicative of a first set of BEV features of image data captured by an image sensor; receiving second information indicative of a second set of BEV features of non-image sensor data captured by a non-image sensor; and determining fused data that combines the image data and the non-image sensor data based on the first information, the second information, and third information indicative of differences between BEV features of training data and the first set of BEV features and the second set of BEV features. The BEV features of the training data include a third set of BEV features associated with the image sensor and a fourth set of BEV features associated with the non-image sensor. Other aspects and features are also claimed and described.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of image processing includes receiving an image frame from an image sensor of a camera; receiving an indicator associated with a type of lens of the camera; determining a first tensor grid associated with the indicator, the first tensor grid including a plurality of image framework positions associated with the type of lens; and determining, using a machine learning model, a BEV feature map corresponding to the image frame based on features of the image frame and the first tensor grid. Other aspects and features are also claimed and described.

Abstract: Systems and techniques are described herein for determining road profiles. For instance, a method for determining a road profiles is provided. The method may include extracting image features from one or more images of an environment, wherein the environment includes a road; generating a segmentation mask based on the image features; determining a subset of the image features based on the segmentation mask; generating image-based three-dimensional features based on the subset of the image features; obtaining point-cloud-based three-dimensional features derived from a point cloud representative of the environment; combining the image-based three-dimensional features and the point-cloud-based three-dimensional features to generate combined three-dimensional features; and generating a road profile based on the combined three-dimensional features.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of image processing includes receiving, by a processor, image data from a camera image sensor; receiving, by the processor, point cloud data from a light detection and ranging (LiDAR) sensor; generating, by the processor and using a first machine learning model, fused image data that combines the image data and the point cloud data; and determining, by the processor and using a second machine learning model, whether the fused image data satisfies a criteria based on whether a population risk function of the first machine learning model exceeds a threshold. Other aspects and features are also claimed and described.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of image processing includes receiving a plurality of image frames by a computing device and using machine learning models to identify corrupted or occluded image frames. A first machine learning model may identify corrupted image frames, while a second machine learning model may identify partially occluded image frames. The method may further include generating updated versions of image frames captured by vehicle cameras, such as based on feature vectors from the first and second machine learning models. The feature vectors may be fused and provided to a third machine learning model to generate updated versions of occluded image frames. The method may further include determining vehicle control instructions based on the updated versions. Other aspects and features are also claimed and described.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method is provided to train a machine learning model using image data and position data to identify contact points and ground surface normal vectors. Image data is received that depicts an object, and position data for the object is also received, such as point cloud position information for various points along the object's exterior surface. Two sets of labels may then be determined based on the position data, with one set identifying where the object contacts a ground surface and another identifying at least one normal vector for the ground surface. The machine learning model may then be trained based on both sets of labels to determine three-dimensional bounding boxes, normal maps, or combinations thereof. Other aspects and features are also claimed and described.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method is provided that includes determining a first set of feature vectors for received images for a top view representation of an area surrounding a vehicle and a second set of feature vectors for a cylindrical representation of the area. The method may further include determining a first set of locations based on the first set of feature vectors and determining a second set of locations based on the second set of feature vectors. A third set of locations may be determined based on the first and second sets of locations, such as combining the first and second sets using a transformer attention process. Vehicle control instructions may then be determined based on the third set of locations. Other aspects and features are also claimed and described.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method is provided for determining the locations and bounding surfaces of objects depicted in image frames captured by fisheye image sensors attached to a vehicle. The method includes receiving raw fisheye image data from the sensor and using machine learning models to determine the locations and three-dimensional bounding surfaces of objects in the image frame. The bounding surfaces may be defined by three-dimensional polar coordinates representing portions of the viewing area of the fisheye image sensor. Control instructions for the vehicle may then be determined based on the bounding surfaces. In certain implementations, the bounding surfaces may be determined in three-dimensional polar coordinates. Other aspects and features are also claimed and described.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method of fusing features from near-field images and far-field images is provided that includes determining feature vectors and spatial locations for received images from near-field and far-field image sensors. A first set of weighted feature vectors may be determined based on spatial locations of the features and a second set of weighted feature vectors may be determined based on corresponding features between the feature vectors. Fused feature vectors may then be determined based on the weighted feature vectors, such as using a transformer attention process trained to select and combine features from both sets of weighted feature vectors. Vehicle control instructions may be determined based on the fused feature vectors. Other aspects and features are also claimed and described.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method is provided that includes generating a top view image of an object using a plurality of images captured from different views. The method involves determining portions of the images that depict the object and generating novel views of the object from at least one novel view not present within the plurality of images. Corresponding portions containing an occluded view and an unobstructed view of the object are identified and corrected views for occluded views are determined based on corresponding unobstructed views using a machine learning model. A top view image may be then generated based on the corrected views. The invention enables improved visibility for autonomous driving systems in situations where objects are occluded or partially obstructed. Other aspects and features are also claimed and described.

Abstract: Systems and techniques are described herein for training an object-detection model. For instance, a method for training an object-detection model is provided. The method may include obtaining a light detection and ranging (LIDAR) capture; obtaining a first LIDAR-based representation of an object as captured from a first distance; obtaining a second LIDAR-based representation of the object as captured from a second distance; augmenting the LIDAR capture using the first LIDAR-based representation of the object and the second LIDAR-based representation of the object to generate an augmented LIDAR capture; and training a machine-learning object-detection model using the augmented LIDAR capture.

Abstract: Disclosed are systems, apparatuses, processes, and computer-readable media for processing image data. For example, a process can include obtaining segmentation information associated with an image of a scene, the image including a plurality of pixels having a resolution, and obtaining depth information associated with one or more objects in the scene. A plurality of features can be generated corresponding to the plurality of pixels, wherein each feature of the plurality of features corresponds to a particular pixel of the plurality of pixels, and wherein each feature includes respective segmentation information of the particular pixel and respective depth information of the particular pixel. The plurality of features can be processed to generate a dense depth output corresponding to the image.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a method is provided that includes receiving sensor data from a plurality of sensors on a vehicle and determining a three-dimensional representation of an area surrounding the vehicle by mapping the sensor data onto a three-dimensional surface. The plurality of sensors may include at least one perspective view sensor and at least one top view sensor, and the three-dimensional surface may include sensor data from the at least one perspective view sensor and sensor data from the at least one top view sensor. The method may further include determining, with a machine learning model, one or more characteristics of the area surrounding the vehicle based on the three-dimensional representation. Other aspects and features are also claimed and described.

Abstract: Techniques and systems are provided for processing sensor data. For instance a process can include obtaining first sensor data of an environment, wherein the first sensor data includes a representation of a first object occluding a second object, obtaining second sensor data of the environment, wherein the second sensor data includes points associated with the first object and points associated with the second object, generating estimated segment data from the first sensor data, wherein the estimated segment data includes a first segment corresponding to the first object; matching points associated with the first object to the first segment, and deemphasizing points associated with the second object based on matching the points associated with the first object to the first segment.

Abstract: This disclosure provides systems, methods, and devices for image signal processing that support training object recognition models. In a first aspect, a method of image processing includes training a first modality imaging system; receiving time-synchronized first input data samples and second input data samples from the first modality imaging system and a second modality imaging system, respectively; processing the first input data samples in the first modality imaging system to generate first output; processing the second input data samples in the second modality imaging system to generate second output; and training the second modality imaging system based on the first output and the second output. Other aspects and features are also claimed and described.

Abstract: Techniques and systems are provided for generating depth information for an image. For instance, a process can include obtaining one or more images of an environment. The process can further include generating a set of features for the one or more images. The process can also include combining the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative height above a ground level. The process can further include generating depth information of the environment based on the combined feature distance information, and outputting the depth information of the environment.

Abstract: Techniques and systems are provided for training a machine learning (ML) model. A technique can include generating a first set of features for objects in images, predicting image feature labels for the first set of features, comparing the predicted image feature labels to ground truth image feature labels to evaluate a first loss function, perform a perspective transform on the first set of features to generate a birds eye view (BEV) projected image features, combining the BEV projected image features and a first set of flattened features to generate combined image features, generating a segmented BEV map of the environment based on the combined image features, comparing the segmented BEV map to a ground truth segmented BEV map to evaluate a second loss function, and training the ML model for generation of segmented BEV maps based on the evaluated first loss function and the evaluated second loss function.