Abstract: A vehicle computing system may implement techniques to determine attributes (or intent) of an object detected by a vehicle operating in the environment. The techniques may include determining a set of features with respect to a detected object by a first model and determining, by a second model and based on the set of features, one or more attributes of the object. The first model and the second model may be configured to process at least one image frame to determine the one or more attributes of the object. A model may receive sensor data as an input, and output features and/or an attribute for the detected object. Based on the attribute(s) of the object, a vehicle computing system may control operation of the vehicle.

Abstract: A trajectory estimate of a wheeled vehicle can be determined based at least in part on determining a wheel angle associated with the vehicle. In some examples, at least a portion of the image associated with the wheeled vehicle may be input into a machine-learned model that is trained to classify and/or regress wheel directions of wheeled vehicles. The machine-learned model may output a predicted wheel direction. The wheel direction and/or additional or historical sensor data may be used to estimate a trajectory of the wheeled vehicle. The predicted trajectory of the object can then be used to generate and refine an autonomous vehicle's trajectory as the autonomous vehicle proceeds through the environment.

Abstract: Techniques for detecting and responding to an emergency vehicle are discussed. A vehicle computing system may determine that an emergency vehicle based on sensor data, such as audio and visual data. In some examples, the vehicle computing system may determine aggregate actions of objects (e.g., other vehicles yielding) proximate the vehicle based on the sensor data. In such examples, a determination that the emergency vehicle is operating may be based on the actions of the objects. The vehicle computing system may, in turn, identify a location to move out of a path of the emergency vehicle (e.g., yield) and may control the vehicle to the location. The vehicle computing system may determine that the emergency vehicle is no longer relevant to the vehicle and may control the vehicle along a route to a destination. Determining to yield and/or returning to a mission may be confirmed by a remote operator.

Abstract: The techniques discussed herein may comprise refining a classification of an object detected as being represented in sensor data. For example, refining the classification may comprise determining a sub-classification of the object.

Abstract: Techniques are disclosed for implementing a neural network that outputs embeddings. Furthermore, techniques are disclosed for using sensor data to train a neural network to learn such embeddings. In some examples, the neural network may be trained to learn embeddings. The embeddings may be used for object identification, object matching, object classification, and/or object tracking in various examples.

Abstract: Techniques for maintaining and synchronizing data is a processing pipeline data between multiple processing units to improve a system latency are described herein. For example, the techniques may include determining, in response to an invocation of vision processing on first vision data stored in a first memory range in a first memory associated with a central processing unit (CPU), that second vision data stored in a second memory range in a second memory associated with a graphic processing unit (GPU) is a modified copy of the first vision data. The second vision data may be obtained using a non-blocking operation from the second memory range. The first vision data stored in the first memory range may be replaced with the second vision data obtained from the second memory range. The vision processing may then be performed using the second vision data stored in the first memory.

Abstract: A machine-learning (ML) architecture may comprise a first ML model and/or an optical flow model that receive, as input, a first image and a second image. The first ML model may output a first feature map corresponding to the first image and a second feature map corresponding to the second image. The optical flow model may output an estimated optical flow. A deformation component may modify the second feature map, as a deformed feature map, based at least in part on the estimated optical flow. The deformed feature map and the first feature map may be concatenated together as a concatenated feature map, which may be provided to a second ML model. The second ML model may be trained to output an output ROI and/or a track in association with an object represented in the first image.

Abstract: Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Abstract: A trajectory estimate of a wheeled vehicle can be determined based at least in part on determining a wheel angle associated with the vehicle. In some examples, at least a portion of the image associated with the wheeled vehicle may be input into a machine-learned model that is trained to classify and/or regress wheel directions of wheeled vehicles. The machine-learned model may output a predicted wheel direction. The wheel direction and/or additional or historical sensor data may be used to estimate a trajectory of the wheeled vehicle. The predicted trajectory of the object can then be used to generate and refine an autonomous vehicle's trajectory as the autonomous vehicle proceeds through the environment.

Abstract: Techniques are disclosed for tracking objects in sensor data, such as multiple images or multiple LIDAR clouds. The techniques may include comparing segmentations of sensor data such as by, for example, determining a similarity of a first segmentation of first sensor data and a second segmentation of second sensor data. Comparing the similarity may comprise determining a first embedding associated with the first segmentation and a second embedding associated with the second segmentation and determining a distance between the first embedding and the second embedding. The techniques may improve the accuracy and/or safety of systems integrating the techniques discussed herein.

Abstract: Techniques are disclosed for tracking objects in sensor data, such as multiple images or multiple LIDAR clouds. The techniques may include comparing segmentations of sensor data such as by, for example, determining a similarity of a first segmentation of first sensor data and a second segmentation of second sensor data. Comparing the similarity may comprise determining a first embedding associated with the first segmentation and a second embedding associated with the second segmentation and determining a distance between the first embedding and the second embedding. The techniques may improve the accuracy and/or safety of systems integrating the techniques discussed herein.

Abstract: Techniques relating to monitoring map consistency are described. In an example, a monitoring component associated with a vehicle can receive sensor data associated with an environment in which the vehicle is positioned. The monitoring component can generate, based at least in part on the sensor data, an estimated map of the environment, wherein the estimated map is encoded with policy information for driving within the environment. The monitoring component can then compare first information associated with a stored map of the environment with second information associated with the estimated map to determine whether the estimated map and the stored map are consistent. Component(s) associated with the vehicle can then control the object based at least in part on results of the comparing.

Abstract: Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Abstract: Techniques for determining and/or predicting a trajectory of an object by using the appearance of the object, as captured in an image, are discussed herein. Image data, sensor data, and/or a predicted trajectory of the object (e.g., a pedestrian, animal, and the like) may be used to train a machine learning model that can subsequently be provided to, and used by, an autonomous vehicle for operation and navigation. In some implementations, predicted trajectories may be compared to actual trajectories and such comparisons are used as training data for machine learning.

Abstract: A sensor degradation detection and remediation system includes one or more sensors configured to collect image data from an environment. A combination of techniques may be used to detect degradations within regions of the image data captured by a sensor, including one or more of determining a level of the visual consistency between the associated image regions captured by different sensors, determining a level of opaqueness of the image regions, and/or measuring temporal movement of the image regions captured by a sensor over a period of time. Operations of a vehicle or other system may be controlled based at least in part on the detection of degradations of the image data captured by the sensors, including automated cleaning of a sensor surface, reducing a level of reliance on the image data received from the sensor, and/or changing a direction of travel of the vehicle.

Abstract: The techniques discussed herein may comprise refining a classification of an object detected as being represented in sensor data. For example, refining the classification may comprise determining a sub-classification of the object.

Abstract: A sensor degradation detection and remediation system includes one or more sensors configured to collect image data from an environment. A combination of techniques may be used to detect degradations within regions of the image data captured by a sensor, including one or more of determining a level of the visual consistency between the associated image regions captured by different sensors, determining a level of opaqueness of the image regions, and/or measuring temporal movement of the image regions captured by a sensor over a period of time. Operations of a vehicle or other system may be controlled based at least in part on the detection of degradations of the image data captured by the sensors, including automated cleaning of a sensor surface, reducing a level of reliance on the image data received from the sensor, and/or changing a direction of travel of the vehicle.

Abstract: Techniques for detecting and responding to an emergency vehicle are discussed. A vehicle computing system may determine that an emergency vehicle based on sensor data, such as audio and visual data. In some examples, the vehicle computing system may determine aggregate actions of objects (e.g., other vehicles yielding) proximate the vehicle based on the sensor data. In such examples, a determination that the emergency vehicle is operating may be based on the actions of the objects. The vehicle computing system may, in turn, identify a location to move out of a path of the emergency vehicle (e.g., yield) and may control the vehicle to the location. The vehicle computing system may determine that the emergency vehicle is no longer relevant to the vehicle and may control the vehicle along a route to a destination. Determining to yield and/or returning to a mission may be confirmed by a remote operator.

Abstract: A machine-learning (ML) architecture for determining three or more outputs, such as a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image. The ML architecture may output these outputs at a rate of 30 or more frames per second on consumer grade hardware.

Abstract: Techniques are disclosed for implementing a neural network that outputs embeddings. Furthermore, techniques are disclosed for using sensor data to train a neural network to learn such embeddings. In some examples, the neural network may be trained to learn embeddings for instance segmentation of an object based on an embedding for a bounding box associated with the object being trained to match pixel embeddings for pixels associated with the object. The embeddings may be used for object identification, object matching, object classification, and/or object tracking in various examples.