Abstract: Techniques relating to monitoring map consistency are described. In an example, a monitoring component associated with a vehicle can receive sensor data associated with an environment in which the vehicle is positioned. The monitoring component can generate, based at least in part on the sensor data, an estimated map of the environment, wherein the estimated map is encoded with policy information for driving within the environment. The monitoring component can then compare first information associated with a stored map of the environment with second information associated with the estimated map to determine whether the estimated map and the stored map are consistent. Component(s) associated with the vehicle can then control the object based at least in part on results of the comparing.

Abstract: A collision avoidance system may validate, reject, or replace a trajectory generated to control a vehicle. The collision avoidance system may comprise a secondary perception component that may receive sensor data, receive and/or determine a corridor associated with operation of a vehicle, classify a portion of the sensor data associated with the corridor as either ground or an object, determine a position and/or velocity of at least the nearest object, determine a threshold distance associated with the vehicle, and control the vehicle based at least in part on the position and/or velocity of the nearest object and the threshold distance.

Abstract: Tracking a current and/or previous position, velocity, acceleration, and/or heading of an object using sensor data may comprise determining whether to associate a current object detection generated from recently received (e.g., current) sensor data with a previous object detection generated from formerly received sensor data. In other words, a track may identify that an object detected in former sensor data is the same object detected in current sensor data. However, multiple types of sensor data may be used to detect objects and some objects may not be detected by different sensor types or may be detected differently, which may confound attempts to track an object. An ML model may be trained to receive outputs associated with different sensor types and/or a track associated with an object, and determine a data structure comprising a region of interest, object classification, and/or a pose associated with the object.

Abstract: A collision avoidance system may validate, reject, or replace a trajectory generated to control a vehicle. The collision avoidance system may comprise a secondary perception component that may receive sensor data, receive and/or determine a corridor associated with operation of a vehicle, classify a portion of the sensor data associated with the corridor as either ground or an object, determine a position and/or velocity of at least the nearest object, determine a threshold distance associated with the vehicle, and control the vehicle based at least in part on the position and/or velocity of the nearest object and the threshold distance.

Abstract: Techniques for determining a prediction probability associated with a disengagement event are discussed herein. A first prediction probability can include a probability that a safety driver associated with a vehicle (such as an autonomous vehicle) may assume control over the vehicle. A second prediction probability can include a probability that an object in an environment is associated the disengagement event. Sensor data can be captured and represented as a top-down representation of the environment. The top-down representation can be input to a machine learned model trained to output prediction probabilities associated with a disengagement event. The vehicle can be controlled based the prediction probability and/or the interacting object probability.

Abstract: Techniques for receiving and processing sensor data captured by a fleet of vehicle are discussed herein. In some examples, a fleet dashcam system can receive sensor data captured by electronic devices on a fleet of vehicles and can use that data to detect collision and near-collision events. The data of the collision or near-collision event can be used to determine a simulation scenario and a response of an autonomous vehicle control to the simulation scenario and/or it can be used to create a collision heat map to aid in operation of an autonomous vehicle.

Abstract: A machine-learning (ML) architecture for determining three or more outputs, such as a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image. The ML architecture may output these outputs at a rate of 30 or more frames per second on consumer grade hardware.

Abstract: Energy consumption for a vehicle may be reduced based at least in part on an environment characteristic associated with the environment through which the vehicle travels or an operation characteristic associated with operation of the vehicle, thereby increasing an operational time of the vehicle. In some situations, reducing energy consumption may be associated with operation of one or more of a sensor (e.g., turning the sensor off, reducing a frequency or resolution of the sensor, etc.) and/or one or more processors associated with the vehicle (e.g., turning a processor off, reducing a rate of computation, etc.) based at least in part on one or more of the environment characteristic signal or the operation characteristic signal.

Abstract: Techniques for utilizing multiple scales of images as input to machine learning (ML) models are discussed herein. Operations can include providing an image associated with a first scale to a first ML model. An output of the first ML model can include a first bounding box indicative of a first region of the image representing a first object, with the first bounding box falling within a first range of sizes. Next, a scaled image can be generated by scaling the image. The scaled image can be provided to a second ML model, which can output a second bounding box indicative of a second region of the image representing a second object, the second bounding falling within a second range of sizes. Thus, inputting a scaled image to a same ML model (or to different ML models) can result in different detected features in the images.

Abstract: Techniques for determining an error model based on vehicle data and ground truth data are discussed herein. To determine whether a complex system (which may be not capable of being inspected) is able to operate safely, various operating regimes (scenarios) can be identified based on operating data. To provide safe operation of such a system, an error model can be determined that can provide a probability associated with perception data and a vehicle can determine a trajectory based on the probability of an error associated with the perception data.

Abstract: A collision avoidance system may validate, reject, or replace a trajectory generated to control a vehicle. The collision avoidance system may comprise a secondary perception component that may comprise one or more machine learned models, each of which may be trained to output one or more occupancy maps based at least in part on sensor data of different types. The occupancy maps may include a prediction of whether at least a portion of an environment is occupied at a future time by any one of multiple object types. Occupancy maps associated with a same time may be aggregated into a data structure that may be used to validate, reject, or replace the trajectory.

Abstract: A vehicle can include various sensors to detect objects in an environment. In some cases, the object may be within a planned path of travel of the vehicle. In these cases, leaving the planned path may be dangerous to the passengers so the vehicle may, based on dimensions of the object, dimensions of the vehicle, and semantic information of the object, determine operational parameters associate with passing the object while maintaining a position within the planned path, if possible.

Abstract: Techniques for determining a safety metric associated with a vehicle controller are discussed herein. To determine whether a complex system (which may be uninspectable) is able to operate safely, various operating regimes (scenarios) can be identified based on operating data and associated with a scenario parameter to be adjusted. To validate safe operation of such a system, a scenario may be identified for inspection. Error metrics of a subsystem of the system can be quantified. The error metrics, in addition to stochastic errors of other systems/subsystems can be introduced to the scenario. The scenario parameter may also be perturbed. Any multitude of such perturbations can be instantiated in a simulation to test, for example, a vehicle controller. A safety metric associated with the vehicle controller can be determined based on the simulation, as well as causes for any failures.

Abstract: Techniques relating to monitoring map consistency are described. In an example, a monitoring component associated with a vehicle can receive sensor data associated with an environment in which the vehicle is positioned. The monitoring component can generate, based at least in part on the sensor data, an estimated map of the environment, wherein the estimated map is encoded with policy information for driving within the environment. The monitoring component can then compare first information associated with a stored map of the environment with second information associated with the estimated map to determine whether the estimated map and the stored map are consistent. Component(s) associated with the vehicle can then control the object based at least in part on results of the comparing.

Abstract: Using detection of low variance regions for improving detection is described. In an example, sensor data can be received from a sensor associated with a vehicle. The sensor data can represent an environment. An indication of a low variance region associated with the sensor data can be determined and an indication of a high variance region associated with the sensor data can be determined based at least in part on the indication of the low variance region. The vehicle can be controlled based on at least one of the sensor data or the indication of the high variance region.

Abstract: Low variance detection training is described herein. In an example, annotated data can be determined based on sensor data received from a sensor associated with a vehicle. The annotated data can comprise an annotated low variance region and/or an annotated high variance region. The sensor data can be input into a model, and the model can determine an output comprising a high variance output and a low variance output. In an example, a difference between the annotated data and the output can be determined and one or more parameters associated with the model can be altered based at least in part on the difference. The model can be transmitted to a vehicle configured to be controlled by another output of the model.

Abstract: Techniques for utilizing multiple scales of images as input to machine learning (ML) models are discussed herein. Operations can include providing an image associated with a first scale to a first ML model. An output of the first ML model can include a first bounding box indicative of a first region of the image representing a first object, with the first bounding box falling within a first range of sizes. Next, a scaled image can be generated by scaling the image. The scaled image can be provided to a second ML model, which can output a second bounding box indicative of a second region of the image representing a second object, the second bounding falling within a second range of sizes. Thus, inputting a scaled image to a same ML model (or to different ML models) can result in different detected features in the images.

Abstract: A vehicle can include various sensors to detect objects in an environment. In some cases, the object may be within a planned path of travel of the vehicle. In these cases, leaving the planned path may be dangerous to the passengers so the vehicle may, based on dimensions of the object, dimensions of the vehicle, and semantic information of the object, determine operational parameters associate with passing the object while maintaining a position within the planned path, if possible.

Abstract: Techniques for encoding image data are discussed. Image data containing multiple image frames can be received from an image sensor associated with a vehicle. A first image frame and a second image frame may be associated with a first data chunk. A first metadata associated with the first image frame can be identified. Additionally, a second metadata associated with the second image frame can be identified. The first data chunk can be created that includes the first image frame, the second image frame, the first metadata, and the second metadata. The first data chunk can be stored in a video container, where the video container can be indexed to access the first image frame with the first metadata or the second image frame with the second metadata.

Abstract: A collision avoidance system may validate, reject, or replace a trajectory generated to control a vehicle. The collision avoidance system may comprise a secondary perception component that may receive sensor data, receive and/or determine a corridor associated with operation of a vehicle, classify a portion of the sensor data associated with the corridor as either ground or an object, determine a position and/or velocity of at least the nearest object, determine a threshold distance associated with the vehicle, and control the vehicle based at least in part on the position and/or velocity of the nearest object and the threshold distance.