Abstract: Techniques for determining predictions on a top-down representation of an environment based on object movement are discussed herein. Sensors of a first vehicle (such as an autonomous vehicle) may capture sensor data of an environment, which may include object(s) separate from the first vehicle (e.g., a vehicle, a pedestrian, a bicycle). A multi-channel image representing a top-down view of the object(s) and the environment may be generated based in part on the sensor data. Environmental data (object extents, velocities, lane positions, crosswalks, etc.) may also be encoded in the image. Multiple images may be generated representing the environment over time and input into a prediction system configured to output a trajectory template (e.g., general intent for future movement) and a predicted trajectory (e.g., more accurate predicted movement) associated with each object. The prediction system may include a machine learned model configured to output the trajectory template(s) and the predicted trajector(ies).

Abstract: Techniques for determining predictions on a top-down representation of an environment based on vehicle action(s) are discussed herein. Sensors of a first vehicle (such as an autonomous vehicle) can capture sensor data of an environment, which may include object(s) separate from the first vehicle (e.g., a vehicle or a pedestrian). A multi-channel image representing a top-down view of the object(s) and the environment can be generated based on the sensor data, map data, and/or action data. Environmental data (object extents, velocities, lane positions, crosswalks, etc.) can be encoded in the image. Action data can represent a target lane, trajectory, etc. of the first vehicle. Multiple images can be generated representing the environment over time and input into a prediction system configured to output prediction probabilities associated with possible locations of the object(s) in the future, which may be based on the actions of the autonomous vehicle.

Abstract: Model-based control of dynamical systems typically requires accurate domain-specific knowledge and specifications system components. Generally, steering actuator dynamics can be difficult to model due to, for example, an integrated power steering control module, proprietary black box controls, etc. Further, it is difficult to capture the complex interplay of non-linear interactions, such as power steering, tire forces, etc. with sufficient accuracy. To overcome this limitation, a recurring neural network can be employed to model the steering dynamics of an autonomous vehicle. The resulting model can be used to generate feedforward steering commands for embedded control. Such a neural network model can be automatically generated with less domain-specific knowledge, can predict steering dynamics more accurately, and perform comparably to a high-fidelity first principle model when used for controlling the steering system of a self-driving vehicle.

Abstract: Techniques for determining predictions on a top-down representation of an environment based on vehicle action(s) are discussed herein. Sensors of a first vehicle (such as an autonomous vehicle) can capture sensor data of an environment, which may include object(s) separate from the first vehicle (e.g., a vehicle or a pedestrian). A multi-channel image representing a top-down view of the object(s) and the environment can be generated based on the sensor data, map data, and/or action data. Environmental data (object extents, velocities, lane positions, crosswalks, etc.) can be encoded in the image. Action data can represent a target lane, trajectory, etc. of the first vehicle. Multiple images can be generated representing the environment over time and input into a prediction system configured to output prediction probabilities associated with possible locations of the object(s) in the future, which may be based on the actions of the autonomous vehicle.

Abstract: Model-based control of dynamical systems typically requires accurate domain-specific knowledge and specifications system components. Generally, steering actuator dynamics can be difficult to model due to, for example, an integrated power steering control module, proprietary black box controls, etc. Further, it is difficult to capture the complex interplay of non-linear interactions, such as power steering, tire forces, etc. with sufficient accuracy. To overcome this limitation, a recurring neural network can be employed to model the steering dynamics of an autonomous vehicle. The resulting model can be used to generate feedforward steering commands for embedded control. Such a neural network model can be automatically generated with less domain-specific knowledge, can predict steering dynamics more accurately, and perform comparably to a high-fidelity first principle model when used for controlling the steering system of a self-driving vehicle.