Abstract: Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Abstract: Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Abstract: Techniques are disclosed for implementing a neural network that outputs embeddings. Furthermore, techniques are disclosed for using sensor data to train a neural network to learn such embeddings. In some examples, the neural network may be trained to learn embeddings. The embeddings may be used for object identification, object matching, object classification, and/or object tracking in various examples.

Abstract: Training a machine-learning (ML) architecture to determine three or more outputs at a rate of 30 or more frames per second on consumer grade hardware may comprise jointly training components of the ML using loss(es) determined across the components and/or consistency losses determined between outputs of two or more components. The ML architecture discussed herein may comprise one or more sets of neural network layers and/or respective components for determining a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image.

Abstract: The techniques discussed herein may comprise refining a classification of an object detected as being represented in sensor data. For example, refining the classification may comprise determining a sub-classification of the object.

Abstract: Improved techniques for training a machine learning (ML) model are discussed herein. Training the ML model can be based on a subset of examples. In particular, the training can include identifying a reference region associated with an area of the image representing an object, and selecting, based at least in part on a first confidence score associated with a first bounding box, a first hard example for inclusion in the subset of examples. In some cases, the first confidence score and the first bounding box can be associated with a first portion of the feature map. Next, the training can include determining that a first degree of alignment of the first bounding box to the reference region is above a threshold degree of alignment, and in response, replacing the first hard example with a second hard example.

Abstract: A machine-learning (ML) architecture may comprise a first ML model and/or an optical flow model that receive, as input, a first image and a second image. The first ML model may output a first feature map corresponding to the first image and a second feature map corresponding to the second image. The optical flow model may output an estimated optical flow. A deformation component may modify the second feature map, as a deformed feature map, based at least in part on the estimated optical flow. The deformed feature map and the first feature may be concatenated together as a concatenated feature map, which may be provided to a second ML model. The second ML model may be trained to output an output ROI and/or a track in association with an object represented in the first image.

Abstract: Techniques are discussed for determining a velocity of an object in an environment from a sequence of images (e.g., two or more). A first image of the sequence is transformed to align the object with an image center. Additional images in the sequence are transformed by the same amount to form a sequence of transformed images. Such sequence is input into a machine learned model trained to output a scaled velocity of the object (a relative object velocity (ROV)) according to the transformed coordinate system. The ROV is then converted to the camera coordinate system by applying an inverse of the transformation. Using a depth associated with the object and the ROV of the object in the camera coordinate frame, an actual velocity of the object in the environment is determined relative to the camera.

Abstract: Techniques are disclosed for implementing a neural network that outputs embeddings. Furthermore, techniques are disclosed for using sensor data to train a neural network to learn such embeddings. In some examples, the neural network may be trained to learn embeddings for instance segmentation of an object based on an embedding for a bounding box associated with the object being trained to match pixel embeddings for pixels associated with the object. The embeddings may be used for object identification, object matching, object classification, and/or object tracking in various examples.

Abstract: Using detection of low variance regions for improving detection is described. In an example, sensor data can be received from a sensor associated with a vehicle. The sensor data can represent an environment. An indication of a low variance region associated with the sensor data can be determined and an indication of a high variance region associated with the sensor data can be determined based at least in part on the indication of the low variance region. The vehicle can be controlled based on at least one of the sensor data or the indication of the high variance region.

Abstract: Low variance detection training is described herein. In an example, annotated data can be determined based on sensor data received from a sensor associated with a vehicle. The annotated data can comprise an annotated low variance region and/or an annotated high variance region. The sensor data can be input into a model, and the model can determine an output comprising a high variance output and a low variance output. In an example, a difference between the annotated data and the output can be determined and one or more parameters associated with the model can be altered based at least in part on the difference. The model can be transmitted to a vehicle configured to be controlled by another output of the model.

Abstract: Techniques are discussed for determining a velocity of an object in an environment from a sequence of images (e.g., two or more). A first image of the sequence is transformed to align the object with an image center. Additional images in the sequence are transformed by the same amount to form a sequence of transformed images. Such sequence is input into a machine learned model trained to output a scaled velocity of the object (a relative object velocity (ROV)) according to the transformed coordinate system. The ROV is then converted to the camera coordinate system by applying an inverse of the transformation. Using a depth associated with the object and the ROV of the object in the camera coordinate frame, an actual velocity of the object in the environment is determined relative to the camera.

Abstract: Techniques are disclosed for implementing a neural network that outputs embeddings. Furthermore, techniques are disclosed for using sensor data to train a neural network to learn such embeddings. In some examples, the neural network may be trained to learn embeddings. The embeddings may be used for object identification, object matching, object classification, and/or object tracking in various examples.

Abstract: A machine-learning (ML) architecture may comprise a first ML model and/or an optical flow model that receive, as input, a first image and a second image. The first ML model may output a first feature map corresponding to the first image and a second feature map corresponding to the second image. The optical flow model may output an estimated optical flow. A deformation component may modify the second feature map, as a deformed feature map, based at least in part on the estimated optical flow. The deformed feature map and the first feature map may be concatenated together as a concatenated feature map, which may be provided to a second ML model. The second ML model may be trained to output an output ROI and/or a track in association with an object represented in the first image.

Abstract: Techniques for using instance segmentation with machine learning (ML) models are discussed herein. An image can be provided as input to a ML model, which can generate, as an output from the ML model, a feature map comprising a plurality of features. Each feature of the plurality of features can comprise a confidence score, classification information, and a region of interest (ROI) determined in accordance with a non-maximal suppression (NMS) technique. Individual ROIs that are similar can be associated together for segmentation purposes. That is, instead of requiring a second ML model and/or a second operation to segment the image (e.g., identify which pixels correspond with the detected object, for example, by outputting a mask or set of lines and/or curves), the techniques discussed herein substantially simultaneously detect an object (e.g., determine an ROI) and segment the image.

Abstract: Techniques for determining and/or predicting a trajectory of an object by using the appearance of the object, as captured in an image, are discussed herein. Image data, sensor data, and/or a predicted trajectory of the object (e.g., a pedestrian, animal, and the like) may be used to train a machine learning model that can subsequently be provided to, and used by, an autonomous vehicle for operation and navigation. In some implementations, predicted trajectories may be compared to actual trajectories and such comparisons are used as training data for machine learning.

Abstract: The techniques discussed herein may comprise refining a classification of an object detected as being represented in sensor data. For example, refining the classification may comprise determining a sub-classification of the object.

Abstract: Improved techniques for training a machine learning (ML) model are discussed herein. Training the ML model can be based on a subset of examples. In particular, the training can include identifying a reference region associated with an area of the image representing an object, and selecting, based at least in part on a first confidence score associated with a first bounding box, a first hard example for inclusion in the subset of examples. In some cases, the first confidence score and the first bounding box can be associated with a first portion of the feature map. Next, the training can include determining that a first degree of alignment of the first bounding box to the reference region is above a threshold degree of alignment, and in response, replacing the first hard example with a second hard example.

Abstract: Techniques for utilizing multiple scales of images as input to machine learning (ML) models are discussed herein. Operations can include providing an image associated with a first scale to a first ML model. An output of the first ML model can include a first bounding box indicative of a first region of the image representing a first object, with the first bounding box falling within a first range of sizes. Next, a scaled image can be generated by scaling the image. The scaled image can be provided to a second ML model, which can output a second bounding box indicative of a second region of the image representing a second object, the second bounding falling within a second range of sizes. Thus, inputting a scaled image to a same ML model (or to different ML models) can result in different detected features in the images.

Abstract: Techniques for using instance segmentation with machine learning (ML) models are discussed herein. An image can be provided as input to a ML model, which can generate, as an output from the ML model, a feature map comprising a plurality of features. Each feature of the plurality of features can comprise a confidence score, classification information, and a region of interest (ROI) determined in accordance with a non-maximal suppression (NMS) technique. Individual ROIs that are similar can be associated together for segmentation purposes. That is, instead of requiring a second ML model and/or a second operation to segment the image (e.g., identify which pixels correspond with the detected object, for example, by outputting a mask or set of lines and/or curves), the techniques discussed herein substantially simultaneously detect an object (e.g., determine an ROI) and segment the image.