Abstract: Vehicle computing systems may receive and fuse long-wave infrared data with radar data to detect and track objects in low-visibility driving environments. In some examples, radar data including position and velocity data may be projected over infrared image data to detect, classify, and track infrared-emitting objects. Machine-learned transformer models with attention also may be trained to output object detections based on combined infrared and radar data. The fusion of infrared and radar data may be used individual and/or may be synchronized with other sensor modalities. In some examples, the fusion and analysis of infrared and radar data may be used in specific low-visibility driving environments, using low-light, fog, and shadowed areas of the environment, to detect and track infrared-emitting and/or moving objects such as pedestrians and animals that may be obscured from detection by other sensor modalities.

Abstract: Techniques for determining a presence of an object, especially an object such as animal or debris, in a path of a vehicle, are discussed herein. For example, sensors of various modalities, which may include multispectral sensors, may capture data representing an environment the vehicle is traversing. In examples, one or more trained machine learned (ML) models, operating on a vehicle computing system, may detect and/or classify objects in the environment, based on input data of one or more modalities or spectral bands. The ML models may be pre-trained using training data including real sensor data, synthetic data, and/or augmented data, along with auto-generated annotations. In some examples, hyperspectral data may be used to identify materials associated with detected objects. A confidence score associated with the detection of the object may also be computed. The vehicle may be controlled based on detection of the object and its classification.

Abstract: A vehicle computing system may implement techniques to determine whether two objects in an environment are related as an articulated object. The techniques may include applying heuristics and algorithms to object representations (e.g., bounding boxes) to determine whether two objects are related as a single object with two portions that articulate relative to each other. The techniques may include predicting future states of the articulated object in the environment. One or more model(s) may be used to determine presence of the articulated object and/or predict motion of the articulated object in the future. Based on the presence and/or motion of the articulated object, the vehicle computing system may control operation of the vehicle.

Abstract: Techniques are discussed herein for controlling autonomous vehicles within a driving environment, including generating and using bounding contours associated with objects detected in the environment. Image data may be captured and analyzed to identify and/or classify objects within the environment. Image-based and/or lidar-based techniques may be used to determine depth data associated with the objects, and a bounding contour may be determined based on the object boundaries and associated depth data. An autonomous vehicle may use the bounding contours of objects within the environment to classify the objects, predict the positions, poses, and trajectories of the objects, and determine trajectories and perform other vehicle control actions while safely navigating the environment.

Abstract: Techniques for detecting and labeling traffic lights and/or traffic signs and the traffic lanes with which such lights and signs are associated in an environment are disclosed. Images may be evaluated to identify pixels that may be associated with a light, sign, or lane. Associations between lights and/or signs and lanes in the environment may be determined along with probabilities for the individual pixels that the pixels may be associated with one of the light/sign and lane associations. Those pixels having a sufficient probability of be associated with a light/sign and lane association may be assigned a corresponding label. An output image with such labels can be provided for vehicle control and for other operations, such as top-down segmentation and trajectory determination.

Abstract: Techniques for updating data operations in a perception system are discussed herein. A vehicle may use a perception system to capture data about an environment proximate to the vehicle. The perception system may receive image data, lidar data, and/or radar data to determine information about an object in the environment. As different sensors may be associated with different time periods for capturing and/or processing operations, the techniques include updating object data with data from sensors associated with a shorter time period to generate intermediate object data. Such intermediate object data may reduce a delay in updating a position of an object in an environment.

Abstract: Techniques for determining a presence of an object, especially an object such as animal or debris, in a path of a vehicle, are discussed herein. For example, sensors of various modalities, which may include multispectral sensors, may capture data representing an environment the vehicle is traversing. In examples, one or more trained machine learned (ML) models, operating on a vehicle computing system, may detect and/or classify objects in the environment, based on input data of one or more modalities or spectral bands. The ML models may be pre-trained using training data including real sensor data, synthetic data, and/or augmented data, along with auto-generated annotations. In some examples, hyperspectral data may be used to identify materials associated with detected objects. A confidence score associated with the detection of the object may also be computed. The vehicle may be controlled based on detection of the object and its classification.

Abstract: A vehicle computing system may implement techniques to determine whether two objects in an environment are related as an articulated object. The techniques may include applying heuristics and algorithms to object representations (e.g., bounding boxes) to determine whether two objects are related as a single object with two portions that articulate relative to each other. The techniques may include predicting future states of the articulated object in the environment. One or more model(s) may be used to determine presence of the articulated object and/or predict motion of the articulated object in the future. Based on the presence and/or motion of the articulated object, the vehicle computing system may control operation of the vehicle.

Abstract: A system for faster object attribute and/or intent classification may include an machine-learned (ML) architecture that processes temporal sensor data (e.g., multiple instances of sensor data received at different times) and includes a cache in an intermediate layer of the ML architecture. The ML architecture may be capable of classifying an object's intent to enter a roadway, idling near a roadway, or active crossing of a roadway. The ML architecture may additionally or alternatively classify indicator states, such as indications to turn, stop, or the like. Other attributes and/or intentions are discussed herein.

Abstract: A computer-implemented method. Includes determining an estimate of a state of an object detected at a first time step, based on a pointwise estimate of the state of the object at the first time step and pointwise measurements of the state of the object at a plurality of further time steps. Includes generating, using the estimate of the state of the object, a proposed annotation associated with the object at the first time step. Includes rendering, via a user interface, a visual representation of the environment at the first time step and a visual representation of the proposed annotation. Includes receiving, via the user interface, user input indicating a user-approved annotation associated with the object at the first time step. Includes generating training data for a machine learning model for use in controlling an autonomous vehicle, based at least in part on the user-approved annotation.

Abstract: Techniques for using an inference wrapper to execute a machine learned model using various types of machine learning engines are disclosed. A resource allocator may interoperate with the inference wrapper to request the utilization of resources, such as memory, and generate and log utilization data. The utilization data can be used to generate visual representations of resource utilization that can then be used to improve resource allocation for machine learned models.

Abstract: Disclosed are techniques for combining data using transformer-based machine learning models. In some examples, a first transformer is used to combine a first dataset with a second dataset. The results are then combined with a third dataset, using a second transformer. Each dataset can represent data from a different sensor modality. The transformers compute scores based on queries and apply the scores to values. The first dataset can be used to generate queries for the first transformer, and the values for the first transformer can be derived from the second dataset. Similarly, the third dataset can be used to generate queries for the second transformer, and the values for the second transformer can be derived from the output of the first transformer. The output of the second transformer is therefore a combination of all three datasets and can be used for object detection, for example, determining three-dimensional boundaries of objects.

Abstract: A system for faster object attribute and/or intent classification may include an machine-learned (ML) architecture that processes temporal sensor data (e.g., multiple instances of sensor data received at different times) and includes a cache in an intermediate layer of the ML architecture. The ML architecture may be capable of classifying an object's intent to enter a roadway, idling near a roadway, or active crossing of a roadway. The ML architecture may additionally or alternatively classify indicator states, such as indications to turn, stop, or the like. Other attributes and/or intentions are discussed herein.

Abstract: Vehicle computing systems may receive and fuse long-wave infrared data with radar data to detect and track objects in low-visibility driving environments. In some examples, radar data including position and velocity data may be projected over infrared image data to detect, classify, and track infrared-emitting objects. Machine-learned transformer models with attention also may be trained to output object detections based on combined infrared and radar data. The fusion of infrared and radar data may be used individual and/or may be synchronized with other sensor modalities. In some examples, the fusion and analysis of infrared and radar data may be used in specific low-visibility driving environments, using low-light, fog, and shadowed areas of the environment, to detect and track infrared-emitting and/or moving objects such as pedestrians and animals that may be obscured from detection by other sensor modalities.

Abstract: Techniques are described herein for encoding of an input data, to handle both variable bitrate requirements and varying importance of content of different portions of the input data. The encoding vectors may be based on a subset of data from the input data. Potential distortion in the reconstruction on a decoder side may be alleviated by transmitting a difference dataset as a complement to encoding vectors encoded from the input data. The difference dataset may be determined taking into account the importance of content of different portions of the input image to reduce the size, for example by masking out portion of the input data that is considered less important. The difference dataset may be compressed based on an available bandwidth.

Abstract: Techniques are discussed herein for controlling autonomous vehicles within a driving environment, including generating and using bounding contours associated with objects detected in the environment. Image data may be captured and analyzed to identify and/or classify objects within the environment. Image-based and/or lidar-based techniques may be used to determine depth data associated with the objects, and a bounding contour may be determined based on the object boundaries and associated depth data. An autonomous vehicle may use the bounding contours of objects within the environment to classify the objects, predict the positions, poses, and trajectories of the objects, and determine trajectories and perform other vehicle control actions while safely navigating the environment.

Abstract: Techniques are disclosed for a covariance model that may generate observation covariances based on observation data of object detections. Techniques may include determining observation data for an object detection of an object represented in sensor data, determining that track data of a track is associated with the object, and inputting the observation data associated with the object detection into a machine-learned model configured to output a covariance (a covariance model). The covariance model may output one or more observation covariance values for the observation data. In some examples, the techniques may include determining updated track data based on the track data, the one or more observation covariance values, and the observation data.

Abstract: A transformer-based machine-learned model may use cross-attention between map data and various sensor data and/or perception data, such as an object detection, to augment perception tasks. In particular, the transformer-based machine-learned model may comprise two or more encoders, one of which may determine a first embedding from map data and a second encoder that may determine a second embedding from sensor data and/or perception data. An encoder may determine a score that may be used to determine various outputs that may improve partially occluded object detection, ground plane classification, static object detection, and suppress false positive object detections.

Abstract: A system for faster object attribute and/or intent classification may include an machine-learned (ML) architecture that processes temporal sensor data (e.g., multiple instances of sensor data received at different times) and includes a cache in an intermediate layer of the ML architecture. The ML architecture may be capable of classifying an object's intent to enter a roadway, idling near a roadway, or active crossing of a roadway. The ML architecture may additionally or alternatively classify indicator states, such as indications to turn, stop, or the like. Other attributes and/or intentions are discussed herein.

Abstract: Systems and techniques for detecting and locating low impact collisions, such as relatively low energy impact collisions involving an autonomous vehicle, are disclosed herein. Sensor data from audio sensors configured on a vehicle may be used to determine whether a threshold number of such sensors have detected sounds associated with a collision. Scores determined for the sensors based on audio energy and confidence values can be used to determine a location estimate for the detected collision.