Abstract: Scene reconstruction is a computer vision process that creates a model of a scene from a given input, usually including creating a three-dimensional (3D) scene model from one or more input two-dimensional (2D) images of the scene. High-quality scene reconstruction and rendering is useful for various applications, such as autonomous agent applications and scene editing applications. Existing scene reconstruction methods encounter limitations in dynamic scenes where moving objects are not consistent across views from different times, and these methods generally lack the motion cues essential for effective object-environment decomposition and further lead to incomplete environment reconstruction of novel views from persistent occlusion of environmental structures.

Abstract: Spatio-temporal reconstruction modeling includes receiving images of a scene, dividing each of the images into patches; generating an image token for each patch; appending one or more motion tokens to the image tokens to generate an input token vector; processing the input token vector with a machine learning (ML) model to generate an output token vector with output image and motion tokens; decoding each output image token to generate a 3D Gaussian and a motion key; decoding each output motion token to generate a velocity basis and a motion query; generating of velocity vectors based on the motion queries and the motion keys; generating a 2D image for a first timestep based on the 3D Gaussians and the velocity vectors; training the ML model based on the 2D image; generating optimized 3D Gaussians using the trained ML model; and generating a dynamic reconstructed 3D scene from the optimized 3D Gaussians.

Abstract: One embodiment of a method for generating scene representations includes processing a first image using a first trained machine learning model to generate one or more second images, processing the one or more second images using a second trained machine learning model to generate three-dimensional (3D) geometry and camera information, and generating a four-dimensional (4D) scene representation based on the 3D geometry and the camera information.

Abstract: Imitation learning, or artificial intelligence-based learning from demonstration, aims to acquire an agent policy by observing and mimicking the behavior demonstrated in expert demonstrations. Imitation learning can be used to generate reliable and robust learned policies in a variety of tasks involving sequential decision-making, such as autonomous driving and robotics tasks. However, existing methods that use next-token-prediction (NTP) models, where the policy reduces to a classifier over a discrete set of trajectory tokens, suffer from covariate shift due to their open-loop training a closed-loop execution. The present disclosure provides closed-loop fine tuning of autonomous agent policies in a manner that can mitigate covariate shift.

Abstract: Techniques for performing low-rank self-calibration of 3D geometric foundation models include receiving a plurality of unlabeled images of a scene, generating a pair of point maps and a pair of confidence maps for a first pair of unlabeled images of the plurality of unlabeled images, determining intrinsic camera parameters for the first pair of unlabeled images, refining the pair of point maps based on the intrinsic camera parameters to generate a refined pair of point maps, generating pseudo-labels for the first pair of unlabeled images based on the refined pair of point maps and the pair of confidence maps, and fine tuning a pretrained machine learning model based on the pseudo-labels to generate a fine-tuned machine learning model.

Abstract: A technique for rendering is provided. The technique includes determining a level of detail for a shade space texture and a screen space; shading the shade space texture having a resolution based on the level of detail; and for a reconstruction operation, performing sampling from the shade space texture, the sampling including a high frequency attenuation of samples of the shade space texture.

Abstract: Apparatuses, systems, and techniques to obtain one or more captions for a video using machine learning. In at least one embodiment, at least one machine learning process is used to generate at least one output caption using at least one image-level caption, at least one video-level caption, and/or at least one motion caption. In at least one embodiment, the video-level caption(s) is/are generated by one or more second machine learning processes using the video, and the image-level caption(s) is/are generated by one or more third machine learning processes using one or more images sampled from the video.

Abstract: Apparatuses, systems, and techniques to perform trajectory predictions within one or more images. In at least one embodiment, a processor comprises one or more circuits to cause one or more neural networks to perform trajectory predictions of two or more objects detected within a plurality of frames without tracking the two or more objects based, at least in part, on processing a sequence of data of the one or more objects as a whole.

Abstract: In various examples, a control stack may include a sequence of machine learning models (MLMs) respectively predicting a sequence of differentiable outputs to determine one or more control sequences. Disclosed approaches may be used to implement an AV stack that is differentiable and modular end-to-end-allowing for interpretability of the outputs and propagation of gradients backwards so that upstream predictions are learned with respect to downstream decision making. The disclosure provides various approaches for interfacing perception with motion prediction in a differentiable manner, as well as for interfacing motion prediction with motion planning and motion control in a differentiable manner.

Abstract: Apparatuses, systems, and techniques to generate trajectory predictions. In at least one embodiment, trajectory predictions are generated based on, for example, one or more neural networks.

Abstract: Apparatuses, systems, and techniques for enhancing autonomous driving systems. In at least one embodiment, visual input corresponding to an observable environment is tokenized into object-level knowledge and provided to a large language model (LLM). Object-level tokens are processed by the LLM to enhance autonomous vehicle route-planning, reducing trajectory error and decreasing collision rates.

Abstract: In various examples, cost probability distributions corresponding to predicted locations of an object in an environment and potential locations for a machine in the environment and may be evaluated using corresponding observed costs corresponding to the machine and the object. The cost probability distributions may be evaluated based on comparing the observed costs to threshold values, which may be determined based on sampling a predicted cost function. A threshold value may be selected to provide false-positive rate and/or false-negative rate guarantees for anomaly detection. Control operations may be performed based on results of the evaluation of the cost probability distributions. For example, based on the results, a motion planner may reuse a planned trajectory for a future planning cycle (e.g., thereby avoiding re-planning computations) or generate and/or select a new planned trajectory (e.g., based at least on one or more anomalies being detected).

Abstract: At least one embodiment is directed towards a computer-implemented method for generating generalized scene representations. The computer-implemented method includes extracting feature information from a plurality of scene images, encoding the feature information to generate a plurality of feature images, and estimating depths of at least a plurality of pixels in each feature image included in the plurality of feature images to produce a plurality of feature frustra. The computer-implemented method also includes generating a plurality of octree voxels from the plurality of feature frusta, sampling points along a plurality of views from different proposed camera angles relative to the plurality of octree voxels to produce feature angles and depths that are subsequently aggregated into a plurality of predicted feature maps, and decoding the plurality of predicted feature maps to generate a plurality of final features maps.

Abstract: Apparatuses, systems, and techniques to generate trajectory predictions. In at least one embodiment, trajectory predictions are generated based on, for example, one or more neural networks.

Abstract: A method includes obtaining, at a data reduction module, metrics of a first block of an input video frame and a second block of a reference frame. The data reduction module includes an analysis module and a filter. A perceptual importance of the first block of the input video frame is determined at the analysis module using the metrics. An entropy of the input video frame provided to an encoder is adjusted at the filter of the data reduction module based on the perceptual importance of the first block of the input video frame.

Abstract: In various examples, a motion planner include an analytical function to predict motion plans for a machine based on predicted trajectories of actors in an environment, where the predictions are differentiable with respect to parameters of a neural network of a motion predictor used to predict the trajectories. The analytical function may be used to determine candidate trajectories for the machine based on a predicted trajectory, to compute cost values for the candidate trajectories, and to select a reference trajectory from the candidate trajectories. For differentiability, a term of the analytical function may correspond to the predicted trajectory. A motion controller may use the reference trajectory to predict a control sequence for the machine using an analytical function trained to generate predictions that are differentiable with respect to at least one parameter of the analytical function used to compute the cost values.

Abstract: One embodiment of a method for controlling a vehicle includes receiving sensor data and information associated with the vehicle, and processing the sensor data and the information via a machine learning model in which a plurality of modules execute in parallel based on one or more cross-attention features to generate a planned motion for the vehicle.

Abstract: One embodiment of a method for controlling a vehicle includes receiving first text that includes a description of a scene and a first plan for driving a vehicle, extracting at least one portion of a set of traffic rules based on the description of the scene and the first plan, generating a first prompt that requests driving instructions and includes the description of the scene, the first plan, and the at least one portion of the set of traffic rules, processing the first prompt via a first trained language model to generate a second plan for driving the vehicle, and generating driving instructions based on the second plan.

Abstract: A technique for rendering is provided. The technique includes determining a level of detail for a shade space texture and a screen space; shading the shade space texture having a resolution based on the level of detail; and for a reconstruction operation, performing sampling from the shade space texture, the sampling including a high frequency attenuation of samples of the shade space texture.

Abstract: Apparatuses, systems, and techniques to generate an image of an environment. In at least one embodiment, one or more neural networks are used to identify one or more static and dynamic features of an environment to be used to generate a representation of the environment.