Abstract: Spatio-temporal reconstruction modeling includes receiving images of a scene, dividing each of the images into patches; generating an image token for each patch; appending one or more motion tokens to the image tokens to generate an input token vector; processing the input token vector with a machine learning (ML) model to generate an output token vector with output image and motion tokens; decoding each output image token to generate a 3D Gaussian and a motion key; decoding each output motion token to generate a velocity basis and a motion query; generating of velocity vectors based on the motion queries and the motion keys; generating a 2D image for a first timestep based on the 3D Gaussians and the velocity vectors; training the ML model based on the 2D image; generating optimized 3D Gaussians using the trained ML model; and generating a dynamic reconstructed 3D scene from the optimized 3D Gaussians.

Abstract: Imitation learning, or artificial intelligence-based learning from demonstration, aims to acquire an agent policy by observing and mimicking the behavior demonstrated in expert demonstrations. Imitation learning can be used to generate reliable and robust learned policies in a variety of tasks involving sequential decision-making, such as autonomous driving and robotics tasks. However, existing methods that use next-token-prediction (NTP) models, where the policy reduces to a classifier over a discrete set of trajectory tokens, suffer from covariate shift due to their open-loop training a closed-loop execution. The present disclosure provides closed-loop fine tuning of autonomous agent policies in a manner that can mitigate covariate shift.

Abstract: In various examples, a control stack may include a sequence of machine learning models (MLMs) respectively predicting a sequence of differentiable outputs to determine one or more control sequences. Disclosed approaches may be used to implement an AV stack that is differentiable and modular end-to-end-allowing for interpretability of the outputs and propagation of gradients backwards so that upstream predictions are learned with respect to downstream decision making. The disclosure provides various approaches for interfacing perception with motion prediction in a differentiable manner, as well as for interfacing motion prediction with motion planning and motion control in a differentiable manner.

Abstract: At least one embodiment is directed towards a computer-implemented method for generating generalized scene representations. The computer-implemented method includes extracting feature information from a plurality of scene images, encoding the feature information to generate a plurality of feature images, and estimating depths of at least a plurality of pixels in each feature image included in the plurality of feature images to produce a plurality of feature frustra. The computer-implemented method also includes generating a plurality of octree voxels from the plurality of feature frusta, sampling points along a plurality of views from different proposed camera angles relative to the plurality of octree voxels to produce feature angles and depths that are subsequently aggregated into a plurality of predicted feature maps, and decoding the plurality of predicted feature maps to generate a plurality of final features maps.

Abstract: In various examples, a motion planner include an analytical function to predict motion plans for a machine based on predicted trajectories of actors in an environment, where the predictions are differentiable with respect to parameters of a neural network of a motion predictor used to predict the trajectories. The analytical function may be used to determine candidate trajectories for the machine based on a predicted trajectory, to compute cost values for the candidate trajectories, and to select a reference trajectory from the candidate trajectories. For differentiability, a term of the analytical function may correspond to the predicted trajectory. A motion controller may use the reference trajectory to predict a control sequence for the machine using an analytical function trained to generate predictions that are differentiable with respect to at least one parameter of the analytical function used to compute the cost values.

Abstract: In various examples, a gradient-based motion planner evaluates a cost function corresponding to routes for a machine and an obstacle to jointly update the routes. The cost function may include terms to penalize deviation from an initial route predicted for the obstacle and acceleration or jerk for the obstacle. The routes for the machine and the obstacle that are updated may be selected using motion classes that characterize relative motion between a route for the machine and a route for the obstacle. A motion class may be based at least on an angular distance between the machine and the agent and free-end homotopy, where members of the class execute the same relative motion with respect to other agents while being continuously transformable to any other member of the class. The members of the class may have the same start point and different end points.

Abstract: One embodiment of a method for controlling a system includes generating a plurality of initializations using a trained machine learning model, performing a plurality of instances of an iterative technique based on the plurality of initializations to generate a plurality of results, generating a control signal based on one or more results included in the plurality of results, and transmitting the control signal to the system to cause the system to perform one or more operations.

Abstract: In various examples, policy planning using behavior models for autonomous and semi-autonomous systems and applications is described herein. Systems and methods are disclosed that determine a policy for navigating a vehicle, such as a semi-autonomous vehicle or an autonomous vehicle (or other machine), where the policy allows for multistage reasoning that leverages future reactive behaviors of one or more other objects. For instance, a first behavior model (e.g., a trajectory tree) may be generated that represents candidate trajectories for the vehicle and one or more second behavior models (e.g., one or more scenario trees) may be generated that respectively represent future behaviors of the other object(s). The first behavior model and the second behavior model(s) may then be processed, such as in a closed-loop simulation based on a realistic data-driven traffic model, to determine the policy for navigating the vehicle.

Abstract: In various examples, a motion planner include an analytical function to predict motion plans for a machine based on predicted trajectories of actors in an environment, where the predictions are differentiable with respect to parameters of a neural network of a motion predictor used to predict the trajectories. The analytical function may be used to determine candidate trajectories for the machine based on a predicted trajectory, to compute cost values for the candidate trajectories, and to select a reference trajectory from the candidate trajectories. For differentiability, a term of the analytical function may correspond to the predicted trajectory. A motion controller may use the reference trajectory to predict a control sequence for the machine using an analytical function trained to generate predictions that are differentiable with respect to at least one parameter of the analytical function used to compute the cost values.