Abstract: The described aspects and implementations enable efficient object detection and tracking. In one implementation, disclosed is a method and a system to perform the method, the system including the sensing system configured to obtain sensing data characterizing an environment of the vehicle. The system further includes a data processing system operatively coupled to the sensing system and configured to process the sensing data using a first (second) set of neural network (NN) layers to obtain a first (second) set of features for a first (second) region of the environment, the first (second) set of features is associated with a first (second) spatial resolution. The data processing system is further to process the two sets of features using a second set of NN layers to detect a location of obj ect(s) in the environment of the vehicle and a state of motion of the object(s).

Abstract: Energy consumption for a vehicle may be reduced based at least in part on an environment characteristic associated with the environment through which the vehicle travels or an operation characteristic associated with operation of the vehicle, thereby increasing an operational time of the vehicle. In some situations, reducing energy consumption may be associated with operation of one or more of a sensor (e.g., turning the sensor off, reducing a frequency or resolution of the sensor, etc.) and/or one or more processors associated with the vehicle (e.g., turning a processor off, reducing a rate of computation, etc.) based at least in part on one or more of the environment characteristic signal or the operation characteristic signal.

Abstract: A vehicular vision system includes a camera disposed at an in-cabin side of a windshield of a vehicle. Responsive at least in part to processing of captured image data, the system determines a camera-derived path of travel of the vehicle along a road. Responsive at least in part to a geographic location of the vehicle, the system determines a geographic-derived path of travel of the vehicle along the road. Control of the vehicle along the road is based on diminished reliance on the determined geographic-derived path of travel when a geographic location reliability level of the determined geographic-derived path of travel is below a threshold geographic location reliability level. Control of the vehicle along the road is based on diminished reliance on the determined camera-derived path of travel of the vehicle when a camera reliability level of the determined camera-derived path of travel is below a threshold camera reliability level.

Abstract: A vehicle may include a primary system and a secondary system to validate operation of the primary system and to control the vehicle to avoid collisions. For example, the secondary system may receive multiple trajectories from the primary system, such as a primary trajectory and a secondary, contingent, trajectory associated with a deceleration or other maneuver. The secondary system may determine if a trajectory is associated with a potential collision, if the trajectory is consistent with a current or previous pose, if the trajectory is compatible with a capability of the vehicle, etc. The secondary system may select the primary trajectory if valid, the secondary trajectory if the primary trajectory is invalid, or another trajectory generated by the secondary system if the primary trajectory and the secondary trajectory are invalid. If no valid trajectory is determined, the vehicle may decelerate at a maximum rate.

Abstract: Techniques for determining an error model based on vehicle data and ground truth data are discussed herein. To determine whether a complex system (which may be not capable of being inspected) is able to operate safely, various operating regimes (scenarios) can be identified based on operating data. To provide safe operation of such a system, an error model can be determined that can provide a probability associated with perception data and a vehicle can determine a trajectory based on the probability of an error associated with the perception data.

Abstract: Techniques for determining a safety metric associated with a vehicle controller are discussed herein. To determine whether a complex system (which may be uninspectable) is able to operate safely, various operating regimes (scenarios) can be identified based on operating data and associated with a scenario parameter to be adjusted. To validate safe operation of such a system, a scenario may be identified for inspection. Error metrics of a subsystem of the system can be quantified. The error metrics, in addition to stochastic errors of other systems/subsystems can be introduced to the scenario. The scenario parameter may also be perturbed. Any multitude of such perturbations can be instantiated in a simulation to test, for example, a vehicle controller. A safety metric associated with the vehicle controller can be determined based on the simulation, as well as causes for any failures.

Abstract: A vehicle computing system may implement techniques to predict behavior of objects detected by a vehicle operating in the environment. The techniques may include determining a feature with respect to a detected objects (e.g., likelihood that the detected object will impact operation of the vehicle) and/or a location of the vehicle and determining based on the feature a model to use to predict behavior (e.g., estimated states) of proximate objects (e.g., the detected object). The model may be configured to use one or more algorithms, classifiers, and/or computational resources to predict the behavior. Different models may be used to predict behavior of different objects and/or regions in the environment. Each model may receive sensor data as an input, and output predicted behavior for the detected object. Based on the predicted behavior of the object, a vehicle computing system may control operation of the vehicle.

Abstract: Techniques for detecting and responding to an emergency vehicle are discussed. A vehicle computing system may determine that an emergency vehicle based on sensor data, such as audio and visual data. In some examples, the vehicle computing system may determine aggregate actions of objects (e.g., other vehicles yielding) proximate the vehicle based on the sensor data. In such examples, a determination that the emergency vehicle is operating may be based on the actions of the objects. The vehicle computing system may, in turn, identify a location to move out of a path of the emergency vehicle (e.g., yield) and may control the vehicle to the location. The vehicle computing system may determine that the emergency vehicle is no longer relevant to the vehicle and may control the vehicle along a route to a destination. Determining to yield and/or returning to a mission may be confirmed by a remote operator.

Abstract: Techniques are discussed for controlling a vehicle, such as an autonomous vehicle, based on occluded areas in an environment. An occluded area can represent areas where sensors of the vehicle are unable to sense portions of the environment due to obstruction by another object or sensor limitation. An occluded region for an object is determined by the vehicle as part of an occlusion grid, from the perspective of the vehicle. The vehicle may receive another occlusion grid from another source, such as another vehicle or a remote computing device that stores and distributes occlusion grids. The other occlusion grid may be from a different perspective than the occlusion grid generated by the vehicle, and may include occupancy data for the region that is otherwise occluded from the perspective of the vehicle. The vehicle can be controlled to traverse the environment based on the occupancy data received from the other source.

Abstract: Techniques for detecting and responding to an emergency vehicle are discussed. A vehicle computing system may determine that an emergency vehicle based on sensor data, such as audio and visual data. In some examples, the vehicle computing system may determine aggregate actions of objects (e.g., other vehicles yielding) proximate the vehicle based on the sensor data. In such examples, a determination that the emergency vehicle is operating may be based on the actions of the objects. The vehicle computing system may, in turn, identify a location to move out of a path of the emergency vehicle (e.g., yield) and may control the vehicle to the location. The vehicle computing system may determine that the emergency vehicle is no longer relevant to the vehicle and may control the vehicle along a route to a destination. Determining to yield and/or returning to a mission may be confirmed by a remote operator.

Abstract: Remote controlling of a vehicle, such as an autonomous vehicle, may sometimes be more efficient and/or reliable. Such control, however, may require processes for ensuring safety of surrounding persons and objects. Aspects of this disclosure include using onboard sensors to detect objects in an environment and alter remote commands according to such objects, e.g. by reducing a maximum permitted velocity of the vehicle as a function of distance to detected objects. In some examples described herein, such remote controlling may be performed by using objects in the environment as control objects, with movements of the control objects resulting in movement of the vehicle.

Abstract: Techniques for determining an error model associated with a system/subsystem of vehicle controller are discussed herein. To determine whether a complex system (which may be uninspectable) is able to operate safely, errors can be introduced into operating regimes (scenarios) to validate the safe operation of such a system. By comparing captured and/or generated vehicle data with ground truth data, an error of the system can be statistically quantified and modeled. The statistical model can be used to introduce errors to the scenario to perturb the scenario to test, for example, a vehicle controller. Based on a simulation of the vehicle controlled in the perturbed scenario, a safety metric associated with the vehicle controller can be determined, as well as causes for any failures.

Abstract: A vehicular vision system includes a camera disposed at an in-cabin side of a windshield of a vehicle. Responsive at least in part to processing of captured image data, the system determines a camera-derived path of travel of the vehicle along a road. Responsive at least in part to a geographic location of the vehicle, the system determines a geographic-derived path of travel of the vehicle along the road. Control of the vehicle along the road is based on diminished reliance on the determined geographic-derived path of travel when a geographic location reliability level of the determined geographic-derived path of travel is below a threshold geographic location reliability level. Control of the vehicle along the road is based on diminished reliance on the determined camera-derived path of travel of the vehicle when a camera reliability level of the determined camera-derived path of travel is below a threshold camera reliability level.

Abstract: Command determination for controlling a vehicle, such as an autonomous vehicle, is described. In an example, individual requests for controlling the vehicle relative to each of multiple objects or conditions in an environment are received (substantially simultaneously) and based on the request type and/or additional information associated with a request, command controllers can determine control commands (e.g., different accelerations, steering angles, steering rates, and the like) associated with each of the one or more requests. The command controllers may have different controller gains (which may be based on functions of distance, distance ratios, time to estimated collisions, etc.) for determining the controls and a control command may be determined based on the all such determined controls.

Abstract: A vehicle may include a primary system and a secondary system to validate operation of the primary system and to control the vehicle to avoid collisions. For example, the secondary system may receive multiple trajectories from the primary system, such as a primary trajectory and a secondary, contingent, trajectory associated with a deceleration or other maneuver. The secondary system may determine if a trajectory is associated with a potential collision, if the trajectory is consistent with a current or previous pose, if the trajectory is compatible with a capability of the vehicle, etc. The secondary system may select the primary trajectory if valid, the secondary trajectory if the primary trajectory is invalid, or another trajectory generated by the secondary system if the primary trajectory and the secondary trajectory are invalid. If no valid trajectory is determined, the vehicle may decelerate at a maximum rate.

Abstract: A method for controlling a vehicle as it travels along a road includes processing at an ECU provided image data captured by a forward viewing camera and processing at the ECU a provided output indicative of a determined geographical location of the vehicle. Responsive at least in part to processing of the provided output, a geographically-derived path of travel of the vehicle is generated. Responsive to determination of a traffic lane in which the vehicle is traveling along the road, a camera-derived path of travel of the vehicle is generated. The vehicle is controlled based on (i) the geographically-derived path of travel of the vehicle and/or (ii) the camera-derived path of travel of the vehicle. The vehicle is controlled based on diminished weight of the geographically-derived path or diminished weight of the camera-derived path when a respective reliability level is below a threshold level.

Abstract: A vehicle can include a primary computing device and a secondary computing device. The primary computing device can receive a trajectory and can generate control data to control the vehicle based on a computed state. Further, the primary computing device can send the internal data to the secondary computing device configured to control the vehicle in the event of a failure of the primary computing device. The secondary computing device can receive the internal data as first internal data and determine a capability associated with the primary computing device. Using the first internal data, the secondary computing device can determine second internal data and, based on the capability (e.g., in event of a failure of the primary computing device), can control the vehicle to follow a trajectory using the second internal data. Transferring state between an active to a standby computing device can ensure algorithmic synchronization and safe operation.

Abstract: Command determination for controlling a vehicle, such as an autonomous vehicle, is described. In an example, individual requests for controlling the vehicle relative to each of multiple objects or conditions in an environment are received (substantially simultaneously) and based on the request type and/or additional information associated with a request, command controllers can determine control commands (e.g., different accelerations, steering angles, steering rates, and the like) associated with each of the one or more requests. The command controllers may have different controller gains (which may be based on functions of distance, distance ratios, time to estimated collisions, etc.) for determining the controls and a control command may be determined based on the all such determined controls.

Abstract: A vehicle may include a primary system and a secondary system to validate operation of the primary system and to control the vehicle to avoid collisions. For example, the secondary system may receive multiple trajectories from the primary system, such as a primary trajectory and a secondary, contingent, trajectory associated with a deceleration or other maneuver. The secondary system may determine if a trajectory is associated with a potential collision, if the trajectory is consistent with a current or previous pose, if the trajectory is compatible with a capability of the vehicle, etc. The secondary system may select the primary trajectory if valid, the secondary trajectory if the primary trajectory is invalid, or another trajectory generated by the secondary system if the primary trajectory and the secondary trajectory are invalid. If no valid trajectory is determined, the vehicle may decelerate at a maximum rate.

Abstract: A vehicle can include a primary computing device and a secondary computing device. The primary computing device can receive a trajectory and can generate control data to control the vehicle based on a computed state. Further, the primary computing device can send the internal data to the secondary computing device configured to control the vehicle in the event of a failure of the primary computing device. The secondary computing device can receive the internal data as first internal data and determine a capability associated with the primary computing device. Using the first internal data, the secondary computing device can determine second internal data and, based on the capability (e.g., in event of a failure of the primary computing device), can control the vehicle to follow a trajectory using the second internal data. Transferring state between an active to a standby computing device can ensure algorithmic synchronization and safe operation.