Abstract: The present disclosure describes methods, computer-readable media, and apparatuses for operating neural networks. For example, an apparatus may receive a set of sparse weight vectors. The apparatus may perform a sparse computation based on the set of sparse weight vectors. The apparatus may combine sparse weight vectors in response to determining a combined time to perform respective numbers of MAC operations for the sparse weight vectors satisfies a threshold number of clock cycles. The apparatus may operate a neural network based at least in part on one or more partial sums produced in performing the sparse computation.

Abstract: Example systems and techniques are described for training a machine learning model. A system includes memory configured to store image data captured by a plurality of cameras and one or more processors communicatively coupled to the memory. The one or more processors are configured to execute a machine learning model on the image data, the machine learning model including a plurality of layers. The one or more processors are configured to apply a non-linear mapping function to output of one layer of the plurality of layers to generate depth data. The one or more processors are configured to train the machine learning model based on the depth data to generate a trained machine learning model.

Abstract: The present disclosure describes methods, computer-readable media, and apparatuses for operating neural networks. For example, a first apparatus may receive a set of sparse weight vectors. The first apparatus may compress the set of sparse weight vectors to produce a compressed set of sparse weight vectors. The first apparatus may operate a neural network based on the compressed set of sparse weight vectors. In another example, a second apparatus may receive a set of sparse weight vectors. The second apparatus may perform a sparse computation based on the set of sparse weight vectors, and the performance of the sparse computation may produce one or more partial sums. The second apparatus may operate a neural network based at least in part on the one or more partial sums.

Abstract: A device for processing image data is disclosed. The device can obtain a radar point cloud and one or more frames of camera data. The device can determine depth estimates of one or more pixels of the one or more frames of camera data. The device can generate a pseudo lidar point cloud using the depth estimates of the one or more pixels of the one or more frames of camera data, wherein the pseudo lidar point cloud comprises a three-dimensional representation of at least one frame of the one or more frames of camera data. The device can determine one or more object bounding boxes based on the radar point cloud and the pseudo lidar point cloud.

Abstract: In some aspects, a system may receive, from a first one-dimensional radar array, first information based at least in part on first reflections associated with an azimuthal plane. The system may further receive, from a second one-dimensional radar array, second information based at least in part on second reflections associated with an elevation plane. Accordingly, the system may detect an object based at least in part on the first information and may determine an elevation associated with the object based at least in part on the second information. Numerous other aspects are described.

Abstract: An apparatus, method and computer-readable media are disclosed for processing images. For example, a method is provided for processing images for one or more visual perception tasks using a machine learning system including one or more transformer layers. The method includes: obtaining a plurality of input images associated with a plurality of spatial views of a scene; generating, using a machine learning-based encoder of a machine learning system, a plurality of features from the plurality of input images; and combining timing information associated with capture of the plurality of input images with at least one input of the machine learning system to synchronize the plurality of features in time.

Abstract: Example systems and techniques are described for controlling operation of a vehicle and training a machine learning model for controlling operation of a vehicle. A system includes memory configured to store point cloud data associated with the vehicle and one or more processors communicatively coupled to the memory. The one or more processors are configured to determine a depth map indicative of distance of one or more objects to the vehicle and control operation of a vehicle based on the depth map. The depth map is based on executing a machine learning model, the machine learning model being trained with a slice loss function determined from training point cloud data having a respective depth that is greater than the average depth for a set of points of the point cloud data plus a threshold.

Abstract: Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. Data from a source domain and data from a target domain is accessed. A set of machine learning models is trained, based on the data from the source domain and the data from the target domain, to generate depth outputs based on input images. Training the set of machine learning models includes: generating a discriminator output based at least in part on an input image frame from either the source domain or the target domain, generating an adversarial loss based on the discriminator output, and refining one or more machine learning models of the set of machine learning models based on the adversarial loss.

Abstract: An example device for processing image data includes a memory configured to store image data; and a processing system comprising one or more processors implemented in circuitry, the processing system being configured to: obtain an image to be processed; obtain a first auxiliary value for the image and a second auxiliary value for the image; generate an input component including the first auxiliary value and the second auxiliary value arranged in a pattern according to a stride of a neural network; and provide the image and the input component to the neural network.

Abstract: A processing system for cross-sensor calibration is configured to perform a first edge detection process on a camera image to generate an edge detected camera image, and perform a second edge detection process on a point cloud frame to generate an edge detected point cloud frame. The processing system projects the edge detected point cloud frame onto the edge detected camera image using an initial calibration matrix, determines an objective function representing an overlap of points in the edge detected point cloud frame and corresponding edge pixel values in the edge detected camera image, and determines a final calibration matrix based on the objective function.

Abstract: An example device for processing image data includes a memory configured to store image data; and one or more processors implemented in circuitry and configured to: determine a set of keypoints representing objects in an image of the image data captured by a camera of a vehicle; determine depth values for the objects in the image; determine positions of the objects relative to the vehicle using the set of keypoints and the depth values; and at least partially control operation of the vehicle according to the positions of the objects. For example, the depth values may represent descriptors for the keypoints or be used to determine the descriptors for the keypoints.

Abstract: An example device for processing image data includes a processing unit configured to: receive, from a camera of a vehicle, a first image frame at a first time and a second image frame at a second time; receive, from an odometry unit of the vehicle, a first position of the vehicle at the first time and a second position of the vehicle at a second time; calculate a pose difference value representing a difference between the second and first positions; form a pose frame having a size corresponding to the first and second image frames and sample values including the pose difference value; and provide the first and second image frames and the pose frame to a neural networking unit configured to calculate depth for objects in the first image frame and the second image frame, the depth for the objects representing distances between the objects and the vehicle.

Abstract: A method of exploiting activation sparsity in deep neural networks is described. The method includes retrieving an activation tensor and a weight tensor where the activation tensor is a sparse activation tensor. The method also includes generating a compressed activation tensor comprising non-zero activations of the activation tensor, where the compressed activation tensor has fewer columns than the activation tensor. The method further includes processing the compressed activation tensor and the weight tensor to generate an output tensor.

Abstract: This disclosure provides systems, methods, and devices for image signal processing that support multi-source pose merging for depth estimation. In a first aspect, a method of image processing includes generating, in accordance with first image data of a first image frame and second image data of a second image frame, a first mask indicating one or more pixels determined not to change position between the first image frame and the second image frame, generating, in accordance with the first image data and the second image data, a second mask indicating one or more pixels determined not to change position between the first image frame and the second image frame, and combining the first mask with the second mask to generate a third mask. Other aspects and features are also claimed and described.

Abstract: According to some aspects of the disclosure, techniques for compression techniques for the radar data that can be used in real-time applications for automated or self-driving vehicles. One or more compression techniques can be selected and/or configured based on information regarding operational conditions provided by a central (vehicle) computer. Operational conditions can include environmental data (e.g., weather, traffic), processing capabilities, mode of operation, and more. Compression techniques can facilitate transport of compressed radar data from a radar sensor to the central computer for processing of the radar data for object detection, identification, positioning, etc.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a computing device receives a predicted depth map determined by a model and determines differences between the predicted depth map and a depth mask. The depth mask may be predetermined to include values indicating a probability that a corresponding region of the predicted depth map is a sky region. A first loss term for the predicted depth map is determined based on the differences and the model is trained based on the first loss term. Other aspects and features are also claimed and described.

Abstract: Systems and techniques are provided for detecting objects. For example, an apparatus may include: at least one memory and at least one processor coupled to the at least one memory. The at least one processor may be configured to determine, for a column of an image captured by a camera, a pixel associated with an object. The at least one processor may be further configured to obtain a distance between the camera and the object. The at least one processor may be further configured to determine, based on the distance, a probability of occupancy of a space relative to the camera.

Abstract: This disclosure provides systems, methods, and devices for vehicle driving assistance systems that support image processing. In a first aspect, a computing device may receive a predicted depth map and a measured depth map and may determine a time difference between the predicted depth map and the measured depth map. A supervision loss term may be determined based on the time difference, such as by weighting the supervision loss term based on the time difference. The computing device may train a model based on the supervision loss term, such as a model that generated the predicted depth map. Other aspects and features are also claimed and described.

Abstract: In some aspects, a device may receive sensor data associated with a vehicle and a set of frames. The device may aggregate, using a first pose, the sensor data associated with the set of frames to generate an aggregated frame, wherein the aggregated frame is associated with a set of cells. The device may obtain an indication of a respective occupancy label for each cell from the set of cells, wherein the respective occupancy label includes a first occupancy label or a second occupancy label, and wherein a subset of cells from the set of cells are associated with the first occupancy label. The device may train, using data associated with the aggregated frame, a machine learning model to generate an occupancy grid, based on a loss function that only calculates a loss for respective cells from the subset of cells. Numerous other aspects are described.

Abstract: Disclosed are techniques for fusing camera and radar frames to perform object detection in one or more spatial domains. In an aspect, an on-board computer of a host vehicle receives, from a camera sensor of the host vehicle, a plurality of camera frames, receives, from a radar sensor of the host vehicle, a plurality of radar frames, performs a camera feature extraction process on a first camera frame of the plurality of camera frames to generate a first camera feature map, performs a radar feature extraction process on a first radar frame of the plurality of radar frames to generate a first radar feature map, converts the first camera feature map and/or the first radar feature map to a common spatial domain, and concatenates the first radar feature map and the first camera feature map to generate a first concatenated feature map in the common spatial domain.