Abstract: A method for generating photorealistic 4D scenes from text inputs is disclosed. The method utilizes a text-to-video diffusion model to generate a reference video and a freeze-time video. A canonical 3D representation is reconstructed using deformable 3D Gaussian Splats (D-3DGS) based on the freeze-time video. Temporal deformations are learned to capture dynamic interactions in the reference video. The method employs a novel Score Distillation Sampling strategy combining multi-view and temporal aspects to enhance consistency and robustness. The resulting 4D scenes feature multiple objects interacting with detailed background environments, viewable from different angles and times. The method enables flexible camera control and integration with augmented and virtual reality applications. Some examples include features such as image-to-4D generation.

Abstract: Examples relate to systems and methods for cross-modal audiovisual generation using diffusion features. A multi-modal diffusion model includes a first modality path comprising diffusion transformer (DiT) blocks pretrained to generate outputs of a first modality, a second modality path comprising DiT blocks pretrained to generate outputs of a second modality, and fusion blocks connecting the paths. Each fusion block receives activations from both modality paths, establishes temporal correspondence between them using self-attention mechanisms with rotary position embeddings, and provides modified activations back to subsequent DiT blocks in both paths. The system processes an input in one modality to generate temporally-aligned output in the other modality, with the fusion blocks enabling bidirectional information exchange while maintaining temporal alignment. The approach eliminates the need for specialized feature extractors by leveraging the activations of frozen pretrained diffusion models.

Abstract: Examples relate to systems and methods for generating a video depicting multiple subjects. The systems and methods receive a text prompt describing a video to be generated, a plurality of reference images each depicting a corresponding subject, and binding information that associates each reference image with a corresponding entity word in the text prompt. The systems and methods extract features from each reference image and fuse the extracted features with text embeddings of the corresponding entity word based on the binding information to create personalization embeddings. The systems and methods generate an initial noisy video and iteratively de-noise the noisy video using a diffusion model conditioned on both the text prompt and the personalization embeddings to generate a personalized video that preserves identity of the corresponding subjects in the reference images while allowing for natural motion and pose variations.

Abstract: Examples relate to computer-implemented systems and methods for generating four-dimensional (4D) video content. A 4D video generation model receives a freeze-time video showing a scene varying in viewpoint and a fixed-view video showing the scene varying in time. The model processes these inputs through parallel pathways—a freeze-time pathway and a fixed-view pathway—each comprising pretrained diffusion transformer blocks. The pathways are synchronized via interleaved synchronization layers to generate a grid of video frames varying consistently in both time and viewpoint. The generated frame grid can be used to reconstruct a three-dimensional representation of the scene for real-time rendering of views from arbitrary viewpoints.

Abstract: Example systems take as input a text prompt describing events and generate a multi-event video, audio, or data sequence based on the text prompt. Example operations include accessing a sequence of events, the sequence of events comprising time ranges for the events of the sequence of events and text descriptions for the events, accessing a global caption comprising a text description for the sequence of events, and inputting the sequence of events and the global caption simultaneously into a trained neural network to generate a video, audio, or data sequence, where frames of the video, sound of the audio, or data of the data sequence during a time range of the time ranges are in accordance with a corresponding text description of the text descriptions. Additionally, systems are disclosed that train a neural network to generate the multi-event video, audio, or a data sequence from the text prompt.

Abstract: Examples relate to systems and methods for generating videos with precise camera control using a camera-conditioned video diffusion transformer (DiT) model. The model performs a denoising process through a series of pretrained video DiT blocks, where camera trajectory information is processed through a camera conditioning branch to generate camera activations. During an initial portion of denoising passes, the process is conditioned on camera activations for an initial subset of video DiT blocks, while later denoising passes proceed without camera conditioning. This approach leverages the insight that camera motion is established early in the denoising process, enabling precise camera control while maintaining high visual quality.

Abstract: A text-to-video framework including a far-reaching interleaved transformer (FIT) block configured to learn a compressed representation of video input using a set of learnable latent tokens. The FIT block includes a diffusion framework and joint spatiotemporal modeling. The FIT block performs patchification of the video input to produce a sequence of patch tokens that are divided into groups. The FIT block instantiates the set of latent tokens and applies a sequence of computational blocks, and projects the patch tokens to generate video frames.

Abstract: A method for generating photorealistic 4D scenes from text inputs is disclosed. The method utilizes a text-to-video diffusion model to generate a reference video and a freeze-time video. A canonical 3D representation is reconstructed using deformable 3D Gaussian Splats (D-3DGS) based on the freeze-time video. Temporal deformations are learned to capture dynamic interactions in the reference video. The method employs a novel Score Distillation Sampling strategy combining multi-view and temporal aspects to enhance consistency and robustness. The resulting 4D scenes feature multiple objects interacting with detailed background environments, viewable from different angles and times. The method enables flexible camera control and integration with augmented and virtual reality applications. Some examples include features such as image-to-4D generation.

Abstract: Hierarchical patch-wise diffusion models (HPDMs) use a diffusion paradigm that learns a hierarchical distribution of patches instead of whole videos for efficient patch-wise training of diffusion models. To enforce consistency between the patches, deep context fusion may be used to propagate the context information from low-scale to high-scale patches in a hierarchical manner. To accelerate patch-wise training and inference, adaptive computation also may be used to allocate more computational resources and network capacity towards coarse image details and to cheapen synthesis of high-frequency texture details. All the processing stages are jointly trained to provide spatially aligned global context to the higher levels of the cascade. As a result, the model does not operate on the full-resolution inputs, which allows the model to be trained on high-resolution video datasets in an end-to-end fashion.

Abstract: Examples relate to systems and methods for generating digital effects experiences. The system performs operations including accessing a set of instructions that defines a digital effects experience. The system processes the set of instructions by a generative machine learning model to generate one or more digital effects comprising the digital effects experience. The system continuously processes one or more inputs, received by a user device, while the one or more digital effects comprising the digital effects experience are presented on the user device, along with the set of instructions in real time by the generative machine learning model to update presentation of the one or more digital effects comprising the digital effects experience.

Abstract: A method for generating photorealistic 4D scenes from text inputs is disclosed. The method utilizes a text-to-video diffusion model to generate a reference video and a freeze-time video. A canonical 3D representation is reconstructed using deformable 3D Gaussian Splats (D-3DGS) based on the freeze-time video. Temporal deformations are learned to capture dynamic interactions in the reference video. The method employs a novel Score Distillation Sampling strategy combining multi-view and temporal aspects to enhance consistency and robustness. The resulting 4D scenes feature multiple objects interacting with detailed background environments, viewable from different angles and times. The method enables flexible camera control and integration with augmented and virtual reality applications. Some examples include features such as image-to-4D generation.

Abstract: The present disclosure addresses technological challenges arising in the field of artificial intelligence (AI) with respect to inefficient use of computing resources and runtime delay. In particular, the present disclosure provides for development of a machine learning model that generates an image sample for a video in a single forward pass. The development of this machine learning model uses an adversarial training approach involving training two machine learning models, a generator model and a discriminator model. With the generator model trained in this way, the generator model can be used to generate image samples for a video in a single forward pass.

Abstract: An environment synthesis framework generates virtual environments from a synthesized two-dimensional (2D) satellite map of a geographic area, a three-dimensional (3D) voxel environment, and a voxel-based neural rendering framework. In an example implementation, the synthesized 2D satellite map is generated by a map synthesis generative adversarial network (GAN) which is trained using sample city datasets. The multi-stage framework lifts the 2D map into a set of 3D octrees, generates an octree-based 3D voxel environment, and then converts it into a texturized 3D virtual environment using a neural rendering GAN and a set of pseudo ground truth images. The resulting 3D virtual environment is texturized, lifelike, editable, traversable in virtual reality (VR) and augmented reality (AR) experiences, and very large in scale.

Abstract: Hierarchical patch-wise diffusion models (HPDMs) use a diffusion paradigm that learns a hierarchical distribution of patches instead of whole videos for efficient patch-wise training of diffusion models. To enforce consistency between the patches, deep context fusion may be used to propagate the context information from low-scale to high-scale patches in a hierarchical manner. To accelerate patch-wise training and inference, adaptive computation also may be used to allocate more computational resources and network capacity towards coarse image details and to cheapen synthesis of high-frequency texture details. All the processing stages are jointly trained to provide spatially aligned global context to the higher levels of the cascade. As a result, the model does not operate on the full-resolution inputs, which allows the model to be trained on high-resolution video datasets in an end-to-end fashion.

Abstract: Unsupervised volumetric 3D animation (UVA) of non-rigid deformable objects without annotations learns the 3D structure and dynamics of objects solely from single-view red/green/blue (RGB) videos and decomposes the single-view RGB videos into semantically meaningful parts that can be tracked and animated. Using a 3D autodecoder framework, paired with a keypoint estimator via a differentiable perspective-n-point (PnP) algorithm, the UVA model learns the underlying object 3D geometry and parts decomposition in an entirely unsupervised manner from still or video images. This allows the UVA model to perform 3D segmentation, 3D keypoint estimation, novel view synthesis, and animation. The UVA model can obtain animatable 3D objects from a single or a few images. The UVA method also features a space in which all objects are represented in their canonical, animation-ready form. Applications include the creation of lenses from images or videos for social media applications.

Abstract: A framework trains game-engine-like neural models from annotated videos to generate a Learnable Game Engine (LGE) that maintains states of the scene, objects and agents in it, and enables rendering the environment from a controllable viewpoint. The LGE models the logic of the game and the rules of physics, making it possible for the user to play the game by specifying both high- and low-level action sequences. The LGE also unlocks a director's mode where the game is played by plotting behind the scenes, specifying high-level actions and goals for the agents using text-based instructions. To implement the director's mode, a trained diffusion-based animation model navigates the scene using high-level constraints, to enable play against an adversary, and to devise the strategy to win a point. To render the resulting state of the environment and its agents, a compositional neural radiance field (NeRF) representation is used in a synthesis model.

Abstract: An environment synthesis framework generates virtual environments from a synthesized two-dimensional (2D) satellite map of a geographic area, a three-dimensional (3D) voxel environment, and a voxel-based neural rendering framework. In an example implementation, the synthesized 2D satellite map is generated by a map synthesis generative adversarial network (GAN) which is trained using sample city datasets. The multi-stage framework lifts the 2D map into a set of 3D octrees, generates an octree-based 3D voxel environment, and then converts it into a texturized 3D virtual environment using a neural rendering GAN and a set of pseudo ground truth images. The resulting 3D virtual environment is texturized, lifelike, editable, traversable in virtual reality (VR) and augmented reality (AR) experiences, and very large in scale.

Abstract: Unsupervised volumetric 3D animation (UVA) of non-rigid deformable objects without annotations learns the 3D structure and dynamics of objects solely from single-view red/green/blue (RGB) videos and decomposes the single-view RGB videos into semantically meaningful parts that can be tracked and animated. Using a 3D autodecoder framework, paired with a keypoint estimator via a differentiable perspective-n-point (PnP) algorithm, the UVA model learns the underlying object 3D geometry and parts decomposition in an entirely unsupervised manner from still or video images. This allows the UVA model to perform 3D segmentation, 3D keypoint estimation, novel view synthesis, and animation. The UVA model can obtain animatable 3D objects from a single or a few images. The UVA method also features a space in which all objects are represented in their canonical, animation-ready form. Applications include the creation of lenses from images or videos for social media applications.

Abstract: An asymmetrically distributed convolution-attention neural network (AsCAN) includes a simple hybrid architecture in which the number of convolutional and transformer blocks is asymmetrically distributed in different processing stages. AsCAN adopts more convolutional blocks in the early processing stages, where the feature maps have relatively large spatial sizes, and more transformer blocks at the later processing stages. Transformer layers are incorporated in the early processing stages as well, except that fewer transformer blocks are used compared to convolutions in the early part. This trend is reversed at the lower resolution in the later processing stages. This uneven distribution of the convolutional and transformer blocks yields better throughput due to improved accelerator utilization at various batch sizes during the inference stage.

Abstract: A framework trains game-engine-like neural models from annotated videos to generate a Learnable Game Engine (LGE) that maintains states of the scene, objects and agents in it, and enables rendering the environment from a controllable viewpoint. The LGE models the logic of the game and the rules of physics, making it possible for the user to play the game by specifying both high- and low-level action sequences. The LGE also unlocks a director's mode where the game is played by plotting behind the scenes, specifying high-level actions and goals for the agents using text-based instructions. To implement the director's mode, a trained diffusion-based animation model navigates the scene using high-level constraints, to enable play against an adversary, and to devise the strategy to win a point. To render the resulting state of the environment and its agents, a compositional neural radiance field (NeRF) representation is used in a synthesis model.