Abstract: In implementations of techniques and systems for employing a training framework for diffusion models to track points in video generation, a processing device receives a training input of noisy video frames of a first digital video depicting a target object exhibiting a reference movement to train a machine-learning model. A denoiser of the machine-learning model is trained to remove noise from the noisy video frames of the first digital video by minimizing a diffusion loss that quantifies noise removal and data reconstruction. A refiner module of the machine-learning model is trained to refine latent features of the noisy video frames by minimizing a correspondence loss that quantifies a spatial correspondence among multiple points in the noisy video frames in relation to the reference movement. The trained machine-learning model is configured to generate a second digital video that depicts a particular object exhibiting a particular movement.

Abstract: A high dynamic range editing system is configured to generate visualizations to aide digital image editing in both high dynamic ranges and standard dynamic ranges. In a first example, the visualization is generated as a histogram. In a second example, the visualization is generated to indicate high dynamic range capabilities. In a third example, the visualization is generated to indicate ranges of luminance values within a digital image. In a fourth example, the visualization is generated as a point curve that defines a mapping between detected luminance values from a digital image and output luminance values over both a standard dynamic range and a high dynamic range. In a fifth example, the visualization is generated as a preview to convert pixels from the digital image in a high dynamic range into a standard dynamic range.

Abstract: Three-dimensional object edit and visualization techniques and systems are described. In a first example, a content navigation control is implemented by a content editing system to aid navigation through a history of how a three-dimensional environment and a three-dimensional object included in the environment is created. In a second example, the content editing system is configured to streamline placement of a three-dimensional object within a three-dimensional environment. The content editing system, for instance, generates a manipulation visualization in support of corresponding editing operations to act as a guide, e.g., as an alignment guide or an option guide. In a third example, the content editing system implements a shadow control that is usable as part of an editing and as a visualization to control rendering of illumination within a three-dimensional environment.

Abstract: In implementations of systems for generating digital content, a computing device implements a generation system to receive a user input specifying a characteristic for digital content. The generation system generates input text based on the characteristic for processing by a first machine learning model. Output text generated by the first machine learning model based on processing the input text is received. The output text describes a digital content component. The generation system generates the digital content component by processing the output text using a second machine learning model. The generation system generates the digital content including the digital content component for display in a user interface based on the characteristic.

Abstract: Digital video editing techniques are described that are based on a target digital image. In one or more implementations, inputs are received. The inputs include a target text prompt, a target digital image depicting a target object, and a source digital video having a plurality of frames depicting a source object. Regions-of-interest are identified in the plurality of frames of the source digital video, respectively, based on the target text prompt and the target digital image using a machine-learning model, e.g., a diffusion model. A plurality of frames of a target digital video are generated as having the target object using a generative machine-learning model. The generating is based on the regions-of-interest, the target digital image, the source digital video, and a source text prompt describing the source digital video.

Abstract: Selective content recording is described. A web-based recording system leverages web application programming interfaces (APIs) to capture and synchronize multiple media streams simultaneously, including display content (e.g., application windows, browser tabs, and so forth) and user content (e.g., audio and video capturing a computing device user). The recording system operates within a web browser environment, eliminating the need for platform-specific installations or updates, and captures only portions of display content and portions of user content that are explicitly selected by a user. Selected portions of display media and selected portions of user media are rendered as separate elements on a single canvas, enabling real-time merging of different media streams via a web browser in a platform-agnostic manner.

Abstract: Video content creation is described. A video generation system employs a machine learning system to automate video generation, from textual input to final video output. The video generation system receives a textual description and optional parameters as user input and uses this input to generate a prompt for input to at least one machine learning model. The prompt causes the at least one machine learning model to output a summary understanding of the user-provided inputs along with a scene-by-scene script for a video. Based on the script, the video generation system automatically imports or generates visuals, synthesizes narration, and selects audio for each video scene. The video generation system is then configured to adjust individual scene elements for optimal cohesion and timing during video playback. In some implementations, the video generation system produces multiple video variations, enabling users to ideate by visualizing and comparing different options.

Abstract: A method, apparatus, non-transitory computer readable medium, apparatus, and system for image processing include obtaining a text prompt and an input image depicting a person, generating a latent code based on the text prompt and the input image, wherein the latent code is optimized by an identity preserving loss, and generating, using an image generator of a machine learning model, a synthetic image based on the latent code, wherein the synthetic image includes an element of the text prompt and preserves an identity of the person in the input image.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for generating intertwined digital designs according to the visual order of structural graph nodes. In particular, in one or more embodiments, the disclosed systems generate, by at least one processor, a structural graph of a digital design that represents overlapping surfaces of objects in the digital design as nodes and object paths between the overlapping surfaces as edges. Further, the disclosed systems assign, by the at least one processor, a visual order to the nodes based on a configuration of the structural graph. Moreover, the disclosed systems generate, by the at least one processor, an intertwined digital design by ordering the overlapping surfaces of the objects in accordance with the assigned visual order of the nodes.

Abstract: This disclosure describes one or more implementations of a panoptic segmentation system that generates panoptic segmented digital images that classify both known and unknown instances of digital images. For example, the panoptic segmentation system builds and utilizes a panoptic segmentation neural network to discover, cluster, and segment new unknown object subclasses for previously unknown object instances. In addition, the panoptic segmentation system can determine additional unknown object instances from additional digital images. Moreover, in some implementations, the panoptic segmentation system utilizes the newly generated unknown object subclasses to refine and tune the panoptic segmentation neural network to improve the detection of unknown object instances in input digital images.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate digital materials from digital images using a diffusion neural network. For instance, in one or more embodiments, the disclosed systems receive a digital image portraying a scene to be replicated as a digital material. The disclosed systems also generate, using a conditioning neural network, a spatial condition from the digital image. Using a controlled diffusion neural network and based on the spatial condition, the disclosed systems generate a plurality of material maps corresponding to the scene portrayed by the digital image.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for generating digital images with a diffusion-based generative neural network conditioned on background-extracted lighting features. The disclosed system determines, in response to a request to generate a digital image, a target background image for inserting a foreground object into the target background image. The disclosed system generates, from the target background image and utilizing a lighting conditioning neural network, a lighting feature representation indicating one or more lighting parameters of the target background image. Additionally, the disclosed system generates, utilizing a diffusion-based generative neural network conditioned on the lighting feature representation, the digital image including the foreground object inserted into the target background image based on a composite image comprising the foreground object and the target background image with a foreground mask corresponding to the foreground object.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for generating digital images via a generative neural network with localized constraints. The disclosed system generates, utilizing one or more encoder neural networks, a sequence of embeddings comprising a prompt embedding representing a text prompt and an object text embedding representing a phrase indicating an object in the text prompt. The disclosed system generates, utilizing the one or more encoder neural networks, a visual embedding representing an object image corresponding to the object. The disclosed system determines a modified sequence of embeddings by replacing the object text embedding with the visual embedding in the sequence of embeddings. The disclosed system also generates, utilizing a generative neural network, a synthetic digital image from the modified sequence of embeddings comprising the visual embedding.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation includes obtaining a text prompt describing an object and a keyable background and generating an image including the object and the keyable background based on the text prompt. Some embodiments generate an alpha image by replacing the keyable background with an alpha channel.

Abstract: Change detection and updates using probabilistic data structures are described. In one or more examples, a change is detected to a dataset record that is used as a basis to generate a first sketch as a probabilistic data structure. A second sketch is generated as a probabilistic data structure based on the change to the dataset record. The first sketch is replaced with the second sketch as stored in a database, the database supporting a probabilistic result to a query operation.

Abstract: Context-based prompt generation techniques for generative machine-learning models are described. In one or more examples, a user interface (UI) selection of a UI element of first digital content is received. A processing device captures contextual information associated with the UI selection. A machine-learning model generates a prompt based on the UI selection and the contextual information. Generative artificial intelligence (AI) generates second digital content based on the prompt. The generative AI is implemented using one or more machine-learning models. The processing device then presents the second digital content for display in a user interface.

Abstract: Implementations of systems and methods for determining viewpoints suitable for performing one or more digital operations on a three-dimensional object are disclosed. Accordingly, a set of candidate viewpoints is established. The subset of candidate viewpoints provides views of an outer surface of a three-dimensional object and those views provide overlapping surface data. A subset of activated viewpoints is determined from the set of candidate viewpoints, the subset of activated viewpoints providing less of the overlapping surface data. The subset of activated viewpoints is used to perform one or more digital operation on the three-dimensional object.

Abstract: In implementation of techniques for generating video from an image based on variable speed, a computing device implements a variable speed system to receive a static digital image displayed in a user interface, an indication of a movement direction, and an indication of a variable rate of speed. The variable speed system generates a digital video based on the static digital image, the digital video having pixels that move in the movement direction at a speed that varies based on the variable rate of speed based on pixels from the static digital image. The variable speed system then displays the digital video in the user interface.

Abstract: Embodiments are disclosed for music generation. The method may include obtaining a music representation. A high-fidelity vocoder can then generate low-resolution audio data based on the music representation. A bandwidth extension module generates high-resolution monophonic audio data based on the low-resolution audio data and the music representation. A mono-to-stereo module generates high-resolution stereophonic audio data based on the high-resolution monophonic audio data.

Abstract: Techniques for optimizing digital content information are described. A conversion detection module identifies a set of conversion elements encoded into digital content information using a machine learning (ML) model. A content selection module selects a candidate conversion element from the set of conversion elements. A content block module identifies a candidate content block comprising the candidate conversion element and context information associated with the candidate conversion element. A prompt generation module generates a prompt for a generative artificial intelligence (GAI) agent, the prompt including the candidate content block and instructions to generate a content block variant comprising the candidate conversion element and a context information variant for the context information associated with the candidate conversion element. A content update module receives the content block variant from the GAI agent.