Abstract: A machine-learned architecture may use sensor data, data derived from the sensor data, and/or to determine a grid indicating objects that are occluded to one or more sensors of a vehicle and velocities associated therewith. This grid may be used to generate a candidate object detection associated with an occluded object and/or an object track associated with the occluded object. The candidate object detection and/or object track may be used to determine a predicted trajectory of the occluded object and, in cases where the occluded object is affirmed by a perception component of the vehicle to be a true positive, may be used to initialize a track associated with the newly disoccluded object.

Abstract: A modified Kalman filter may include one or more neural networks to augment or replace components of the Kalman filter in such a way that the human interpretability of the filter's inner functions is preserved. The neural networks may include a neural network to account for bias in measurement data, a neural network to account for unknown controls in predicting a state of an object, a neural network ensemble that is trained differently based on different sensor data, a neural network for determining the Kalman gain, and/or a set of Kalman filters including various neural networks that determine independent estimated states, which may be fused using Bayesian fusion to determine a final estimated state.

Abstract: A vehicle computing system may implement techniques to determine whether two objects in an environment are related as an articulated object. The techniques may include applying heuristics and algorithms to object representations (e.g., bounding boxes) to determine whether two objects are related as a single object with two portions that articulate relative to each other. The techniques may include predicting future states of the articulated object in the environment. One or more model(s) may be used to determine presence of the articulated object and/or predict motion of the articulated object in the future. Based on the presence and/or motion of the articulated object, the vehicle computing system may control operation of the vehicle.

Abstract: Systems and techniques for determining more accurate object parameter values, such as orientation values (e.g., yaw, location, position, etc.), for objects detected in an environment are disclosed. A vehicle computing system may receive sensor data from multiple sensor systems that indicates a parameter value determined at the individual sensor systems. The vehicle computing system may determine values and probabilities for particular parameters modes based on the sensor data parameter values and may further filter these values using a mixture model to determine probability distributions for the modes and associated values. These filtered values and modes may then be used to determine predicted object trajectories that can be used to control a vehicle.

Abstract: Object detection and tracking systems may use machine-learned transformer models with self-attention for detecting, classifying, and/or tracking objects in an environment. Techniques described herein may include receiving sensor data generated by different sensor modalities of a vehicle, determining different bounding shapes based on the different sensor modalities, and using a machine-learned transformer model to determine associated and/or combined bounding shapes. The machine-learned transformer model may receive a variable number of input bounding shapes representing any number of objects and various sensor modalities. Multiple stages of the transformer may be used to determine associated bounding shapes and to assign attributes for the associated bounding shapes, based on the individual bounding shapes of the different sensor modalities and/or previous bounding shapes for objects detected and tracked in a previous scene in the environment.

Abstract: Techniques for performing object detection in a vehicle environment using sensor data captured by one or more sensors of the vehicle are described herein. In some cases, an object in a vehicle environment can be detected based on at least one of (i) a first similarity matrix that represents a first similarity value for two sensor observations associated with the vehicle environment, or (ii) a second similarity matrix that represents a second similarity value for a sensor observation and a track associated with the vehicle environment.

Abstract: A vehicle computing system may implement techniques to determine whether two objects in an environment are related as an articulated object. The techniques may include applying heuristics and algorithms to object representations (e.g., bounding boxes) to determine whether two objects are related as a single object with two portions that articulate relative to each other. The techniques may include predicting future states of the articulated object in the environment. One or more model(s) may be used to determine presence of the articulated object and/or predict motion of the articulated object in the future. Based on the presence and/or motion of the articulated object, the vehicle computing system may control operation of the vehicle.

Abstract: A computer-implemented method. Includes determining an estimate of a state of an object detected at a first time step, based on a pointwise estimate of the state of the object at the first time step and pointwise measurements of the state of the object at a plurality of further time steps. Includes generating, using the estimate of the state of the object, a proposed annotation associated with the object at the first time step. Includes rendering, via a user interface, a visual representation of the environment at the first time step and a visual representation of the proposed annotation. Includes receiving, via the user interface, user input indicating a user-approved annotation associated with the object at the first time step. Includes generating training data for a machine learning model for use in controlling an autonomous vehicle, based at least in part on the user-approved annotation.

Abstract: An evaluation computing system may implement techniques to validate a vehicle controller, such as based on an update thereto. The evaluation computing system may access data associated with an operation of the vehicle in an environment as controlled by the controller. The evaluation computing system may modify a portion of the data representative of a fault associated with the controller. The evaluation computing system may run a simulation utilizing modified log data to determine whether the controller detects and/or mitigates the fault within a threshold time. Based on a determination that the controller does not detect and/or mitigate the fault within the threshold time, the evaluation computing system may determine an error associated with the controller. Based on a determination that the controller detects and/or mitigates the fault within the threshold time, the evaluation computing system may validate the controller.

Abstract: Techniques for training a model for detecting objects in an environment are discussed herein. For example, techniques can include determining losses associated with spatial features of candidate bounding boxes output by a machine-learned (ML) model and utilizing the losses to train the ML model. Techniques may include determining candidate bounding box(es) associated with an object detected in an environment using the ML model and receiving a ground truth bounding box associated with the detected object. A yaw error loss may be determined by comparing yaw features of the candidate bounding box to the ground truth bounding box. The candidate bounding box may be axis aligned with respect to the ground truth bounding box and an intersection over union (IoU) loss may be determined based on an IoU between the axis aligned candidate bounding box and the ground truth bounding box. The ML model may be trained based on the losses.

Abstract: Techniques for training a model for detecting objects in an environment are discussed herein. For example, techniques can include determining losses associated with spatial features of candidate bounding boxes output by a machine-learned (ML) model and utilizing the losses to train the ML model. Techniques may include determining candidate bounding box(es) associated with an object detected in an environment using the ML model and receiving a ground truth bounding box associated with the detected object. A yaw error loss may be determined by comparing yaw features of the candidate bounding box to the ground truth bounding box. The candidate bounding box may be axis aligned with respect to the ground truth bounding box and an intersection over union (IoU) loss may be determined based on an IoU between the axis aligned candidate bounding box and the ground truth bounding box. The ML model may be trained based on the losses.

Abstract: A vehicle computing system may implement techniques to determine whether two objects in an environment are related as an articulated object. The techniques may include applying heuristics and algorithms to object representations (e.g., bounding boxes) to determine whether two objects are related as a single object with two portions that articulate relative to each other. The techniques may include predicting future states of the articulated object in the environment. One or more model(s) may be used to determine presence of the articulated object and/or predict motion of the articulated object in the future. Based on the presence and/or motion of the articulated object, the vehicle computing system may control operation of the vehicle.

Abstract: Techniques for performing object detection in a vehicle environment using sensor data captured by one or more sensors of the vehicle are described herein. In some cases, an object in a vehicle environment can be detected based on at least one of (i) a first similarity matrix that represents a first similarity value for two sensor observations associated with the vehicle environment, or (ii) a second similarity matrix that represents a second similarity value for a sensor observation and a track associated with the vehicle environment.

Abstract: A vehicle computing system may implement techniques to determine whether two objects in an environment are related as an articulated object. The techniques may include applying heuristics and algorithms to object representations (e.g., bounding boxes) to determine whether two objects are related as a single object with two portions that articulate relative to each other. The techniques may include predicting future states of the articulated object in the environment. One or more model(s) may be used to determine presence of the articulated object and/or predict motion of the articulated object in the future. Based on the presence and/or motion of the articulated object, the vehicle computing system may control operation of the vehicle.

Abstract: A computer implemented method. Includes: obtaining, from a sensor system onboard a vehicle, a plurality of hypotheses for an orientation of an object in a vicinity of the vehicle, relative to a coordinate system of the vehicle; estimating, based on the plurality of hypotheses, values of a probability distribution for the orientation of the object at a plurality of candidate orientations; and estimating, based at least in part on the estimated values of the probability distribution at the plurality of candidate orientations, the orientation of the object relative to the coordinate system.

Abstract: A modified Kalman filter may include one or more neural networks to augment or replace components of the Kalman filter in such a way that the human interpretability of the filter's inner functions is preserved. The neural networks may include a neural network to account for bias in measurement data, a neural network to account for unknown controls in predicting a state of an object, a neural network ensemble that is trained differently based on different sensor data, a neural network for determining the Kalman gain, and/or a set of Kalman filters including various neural networks that determine independent estimated states, which may be fused using Bayesian fusion to determine a final estimated state.

Abstract: Techniques to use a trained model to determine a yaw of an object are described. For example, a system may implement various techniques to generate multiple representations for an object in an environment. Each representation vary based on the technique and data used. An estimation component may estimate a representation from the multiple representations. The model may be implemented to output a yaw for the object using the multiple representations, the estimated representation, and/or additional information. The output yaw may be used to track an object, generate a trajectory, or otherwise control a vehicle.

Abstract: A vehicle computing system may implement techniques to determine whether two objects in an environment are related as an articulated object. The techniques may include applying heuristics and algorithms to object representations (e.g., bounding boxes) to determine whether two objects are related as a single object with two portions that articulate relative to each other. The techniques may include predicting future states of the articulated object in the environment. One or more model(s) may be used to determine presence of the articulated object and/or predict motion of the articulated object in the future. Based on the presence and/or motion of the articulated object, the vehicle computing system may control operation of the vehicle.

Abstract: The techniques discussed herein include modifying a Kalman filter to additionally include a loss component that dampens the effect measurements with large errors (or measurements indicating states that are rather different than the predicted state) have on the Kalman filter and, in particular, the updated uncertainty and/or updated prediction. In some examples, the techniques include scaling a Kalman gain based at least in part on a loss function that is based on the innovation determined by the Kalman filter. The techniques additionally or alternatively include a reformulation of a Kalman filter that ensures that the uncertainties determined by the Kalman filter remain symmetric and positive definite.

Abstract: A computer-implemented method. Includes obtaining pointwise data indicating, for a plurality of time steps, a pointwise measurement of a state of an object detected by an object detection system. Includes obtaining, from a runtime model, runtime data indicating, for the plurality of time steps, a runtime estimate of the state of the object. Includes processing, by a benchmark model, the pointwise data to determine, for the plurality of time steps, a benchmark estimate of the state of the object. Includes evaluating a metric measuring, for the plurality of time steps, a deviation between the runtime estimate and the benchmark estimate of the state of the object. Includes updating, based on the on the evaluation of the metric, the runtime model.