Abstract: Suggested question generation using machine learning is described. In one or more examples, training data is synthesized that is configured to train a question machine-learning model to generate a suggested question based on digital content. Synthesis of the training data includes extracting a plurality of portions from one or more items of digital content, generating a plurality of overviews, respectively, based on the plurality of portions, generating one or more profiles based on the plurality of overviews and the plurality of portions, forming one or more prompts to cause a language model to extract text from the item of digital content that has a likelihood of being of interest based on the one or more profiles, and forming at least one prompt to cause the language model to generate a set of training questions as part of the training data based on a respective said profile and the text.

Abstract: An audio generation system is described. The audio generation system employs machine learning models to create custom background music for video data. The audio generation system receives video data as input, generates a transcript and visual scene captions for the video data, and produces a “vibe” description of the video. The vibe description, along with user inputs and preferences, is used to generate multiple music compositions. The audio generation system provides an interface for users to preview, adjust, and refine generated music compositions. Users can modify musical attributes and properties, which the system uses to regenerate compositions based on the refined vibe description. The system enables granular music generation, such as on a scene-by-scene basis, and offers features like extending, condensing, or looping specific sections of generated music, such that the generated music can be output for playback seamlessly with the video data.

Abstract: Embodiments are disclosed for long range dense video tracking. The method may include obtaining query points corresponding to pixels in a first frame of video data. Trajectories of the query points across multiple frames of the video data can then be determined using optical flow. When an error is detected in the one or more trajectories, then the one or more trajectories are iteratively updated using an error compensation module to obtain one or more corrected trajectories. The one or more corrected trajectories are then output.

Abstract: In implementations of techniques and systems for bounding box motion control for image-to-video generation, a processing device receives an input defining an initial location in an initial video frame and a target location in a target video frame. The processing device determines, for a set of intermediary video frames, a set of intermediary locations as exhibiting movement between the initial location and the target location. The processing device generates a digital video that includes an object as disposed at the initial location in the initial video frame, the set of intermediary locations in the set of intermediary video frames, and the target location in the target video frame using a transformer-based diffusion model.

Abstract: A method of multimodal content generation includes receiving a request to generate a draft document. The request can include a text theme and one or more reference documents which include text content. Sentences can then be extracted from the text content and used, along with the text theme, to generate a text paragraph that is grounded in the sentences. A visual classifier can determine whether this generated text is visual text and, if so, image content can be retrieved or generated to be added to the draft document.

Abstract: Systems and methods for image processing are described. Embodiments of the present disclosure obtain a low-resolution image and a text description of the low-resolution image. A mapping network generates a style vector representing the text description of the low-resolution image. An adaptive convolution component generates an adaptive convolution filter based on the style vector. An image generation network generates a high-resolution image corresponding to the low-resolution image based on the adaptive convolution filter.

Abstract: Some aspects relate to technologies for generating data filters from natural language queries and using the data filters to retrieve data from a structured dataset. In accordance with some aspects, a natural language query is received. A generative model generates an initial filter based on the natural language query, where the initial filter includes an initial attribute name and an initial attribute value. A valid attribute value corresponding to the initial attribute value is identified, where the valid attribute value comprises an attribute value in the structured dataset. Additionally, a valid attribute name corresponding to the initial attribute name is identified, where the valid attribute name comprises an attribute name in the structured dataset. A valid filter is generated using the valid attribute value and the valid attribute name, and data is retrieved from the structured dataset using the valid filter.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that generates a long-range event relation dataset by augmenting a digital document with a set of synthetic sentences. For example, the disclosed systems access a digital document from a short-range event relation dataset that includes an event pair. In some embodiments, the disclosed systems generate a set of synthetic sentences utilizing a generative language model for inserting within the digital document between the event pair to satisfy a long-range event relation threshold. In these or other embodiments, the disclosed systems generate a long-range event relation dataset by augmenting the digital document within the short-range event relation dataset to include the set of synthetic sentences.

Abstract: This disclosure relates to segmenting repeating patterns in images to provide representations of individual objects. For example, a method involves segmenting an input image including a repeating pattern to identify input image regions of a minimal cell. The method further involves representing image regions using nodes of a graph, and defining edges between nodes that represent adjoining image regions. The method also involves identifying, using the graph, portions of toroidally connected image regions and portions of mergeable adjoining image regions. The method additionally involves joining each portion of each mergeable adjoining image region, and unwrapping each portion of each toroidally connected image region. The method also involves rendering or storing each of the resultant image regions as an object from the repeating pattern.

Abstract: Techniques and systems for training and/or implementing a text-to-image generation model are provided. A pre-trained multimodal model is leveraged for avoiding slower and more labor-intensive methodologies for training a text-to-image generation model. Accordingly, images without associated text (i.e., bare images) are provided to the pre-trained multimodal model so that it can produce generated text-image pairs. The generated text-image pairs are provided to the text-to-image generation model for training and/or implementing the text-to-image generation model.

Abstract: A search system determines a hierarchy of attributes for a set of search results and returns the search results based on a top-ranked attribute. The search system identifies item listings based on a search input and determines attributes of the item listings. A machine learning model generates a hierarchy of the attributes. The search results are provided based on a top-ranked attribute from the hierarchy of the attributes.

Abstract: Methods, computer systems, computer-storage media, and graphical user interfaces are provided for efficiently generating video insights based on text representations of videos. In embodiments, text data associated with a video is obtained. Thereafter, a model prompt to be input into a large language model is generated. The model prompt includes the text data associated with the video. As output from the large language model, a text representation that represents the video in natural language based on the text data is obtained. The text representation is provided as input into a machine learning model to generate a video insight that indicates context of the video.

Abstract: Techniques for reference image based material retrieval are described that support identification of procedural materials based on visual features of input images. A processing device, for instance, receives an input image that has a particular visual appearance. The processing device generates a histogram representation of the input image that represents a color prominence of the input image and generates a color distribution based on the color prominence. The processing device leverages a vision language model to filter candidate procedural materials by a semantic similarity to the input image. The processing device then identifies a procedural material that has a visual similarity to the particular visual appearance by comparing the color distribution for the input image to color distributions associated with the filtered candidate procedural materials.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image processing include obtaining an input texture image and a plurality of image masks, generating a plurality of image assets corresponding to the plurality of image masks based on the input texture image, and generating a combined asset including the plurality of image assets. The plurality of image assets have a consistent texture based on the input texture image.

Abstract: The present disclosure is directed toward systems, methods, and non-transitory computer readable media that provide assistive guides for path tracing of raster images. In particular, in one or more implementations, the disclosed systems determine a set of outlines corresponding to boundaries of a set of segments within a raster image. The disclosed systems select, from the set of outlines, an outline corresponding to a segment in response to a client device input indicating point(s) located within a threshold distance of the outline. The disclosed systems provide, for display within a graphical user interface of a client device, a highlighted indication of the outline corresponding to the segment. The disclosed systems generate, within a vector image, a vector path based on the outline corresponding to the segment in response to a selection of the outline via the graphical user interface.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for generating segmentations of a raster image via a half-edge mesh structure with scanline operations. The disclosed system determines, during scanline operations on a raster image, a plurality of sets of adjacent pixels having a common color value in the raster image. The disclosed system determines, during the scanline operations on the raster image, a plurality of half-edges at edges of pixels along a boundary of a set of adjacent pixels of the plurality of sets of adjacent pixels with next half-edge directions indicating directions of subsequent half-edges along the boundary of the set of adjacent pixels. The disclosed system generates one or more oriented polyline boundary loops representing the boundary of the set of adjacent pixels from the plurality of half-edges and the next half-edge directions of the set of adjacent pixels.

Abstract: Embodiments are disclosed for video composition retargeting. The method may include identifying at least one asset and associated source lexical data based on a source video and identifying corresponding target lexical data based on a target video. At least one asset retargeting time associated with the at least one asset is determined using the source lexical data and the target lexical data. At least one asset retargeting position associated with the at least one asset is determined based on the at least one asset retargeting time and the target video. The at least one asset is added to the target video using the at least one asset retargeting time and the at least one asset retargeting position.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing offline models to warm start online bandit learner models. For example, the disclosed system can determine relevant offline models for an environment based on reward estimate differences between the offline models and the online model. The disclosed system can then utilize the relevant offline models (if any) to select an arm for the environment. The disclosed system can update the online model based on observed rewards for the selected arm. Additionally, the disclosed system can also use entropy reduction of arms to determine the utility of the arms in differentiating relevant and irrelevant offline models. For example, the disclosed system can select an arm based on a combination of the entropy reduction of the arm and the reward estimate for the arm and use the observed reward to update an observation history.

Abstract: Techniques for modeling secondary motion based on three-dimensional models are described as implemented by a secondary motion modeling system, which is configured to receive a plurality of three-dimensional object models representing an object. Based on the three-dimensional object models, the secondary motion modeling system determines three-dimensional motion descriptors of a particular three-dimensional object model using one or more machine learning models. Based on the three-dimensional motion descriptors, the secondary motion modeling system models at least one feature subjected to secondary motion using the one or more machine learning models. The particular three-dimensional object model having the at least one feature is rendered by the secondary motion modeling system.

Abstract: Techniques for generating traversable three-dimensional scenes from text prompts are described and are implementable to create interactive virtual environments. In an example, a text-based query that specifies visual aspects of a desired three-dimensional scene is received. A spherical representation of the scene is generated by processing the query using one or more machine learning models. A traversable three-dimensional scene depicting the specified visual aspects is then constructed by processing the spherical representation using the one or more machine learning models. The traversable scene is output as part of a three-dimensional environment that allows navigation and viewing from multiple perspectives.