Abstract: Approaches presented herein provide systems and methods for generating three-dimensional (3D) objects using compressed data as an input. One or more models may learn from a hash table of latent features to map different features to a reconstruction domain, using a hash function as part of a learned process. A 3D shape for an object may be encoded to a multi-layered grid and represented by a series of embeddings, where given point within the grid may be interpolated based on the embeddings for a given layer of the multi-layered grid. A decoder may then be trained to use the embeddings to generate an output object.

Abstract: Systems and methods of the present disclosure include interactive editing for generated three-dimensional (3D) models, such as those represented by neural radiance fields (NeRFs). A 3D model may be presented to a user in which the user may identify one or more localized regions for editing and/or modification. The localized regions may be selected and a corresponding 3D volume for that region may be provided to one or more generative networks, along with a prompt, to generate new content for the localized regions. Each of the original NeRF and the newly generated NeRF for the new content may then be combined into a single NeRF for a combined 3D representation with the original content and the localized modifications.

Abstract: Various examples, systems, and methods are disclosed relating to reconstructing, segmenting, and/or simulating pipeline. A first computing system can obtain video data including a depth map of a scene. The first computing system can reconstruct a three-dimensional (3D) representation of the scene using at least one or more Gaussian splat representations of one or more objects in the scene and the depth map. The first computing system can segment at least one object in the 3D representation. The first computing system can update at least one of the plurality of regions of the 3D representation within a threshold distance of the at least one object in the scene. The first computing system can generate at least one image of the 3D representation that depicts at least a portion of the at least one object for display.

Abstract: Various examples, systems, and methods are disclosed relating to reconstructing, segmenting, and/or simulating pipeline. A first computing system can obtain at least one object segmented from video data. The first computing system can densify the at least one object. The first computing system can simulate one or more interactions of the voxelized volume of the at least one densified object to update at least one physical attribute of the at least one object. The first computing system can generate at least one image depicting at least a portion of the at least one object with the at least one updated physical attribute.

Abstract: Approaches presented herein provide for efficient rendering of high quality, novel views of a scene, in this case achieved through a combination of volumetric particle representations and ray tracing. An object can be represented using a set of volumetric particles (e.g., 3D distributions) that are aligned to the underlying structure or geometry of the object. Volumetric particles can be encapsulated in a bounding mesh or proxy geometry that can be used to efficiently compute ray-particle intersections. For a view to be rendered, ray tracing can be performed to determine an intersection of the rays with the proxy geometry. When a hit is determined, the precise intersection location with the volumetric particle is computed and the value of the distribution returned for that ray. If a ray passes through multiple semi-transparent volumetric particles then the color value is determined based upon the values returned from those particles.

Abstract: Systems and methods of the present disclosure include providing higher levels of detail (LODs) for generated three-dimensional (3D) models, such as those represented by neural radiance fields (NeRFs). A 3D model may be presented to a user in which the user may request additional LODs, such as to zoom into the image or to receive information about features within the image. A request to generate finer levels of detail may include using one or more diffusion models to generate images at higher resolutions and/or to hallucinate finer details based on information extracted from the original image or text prompts. Newly generated images may then be added to a set of images associated with the 3D models to enable later model generation to have finer details.

Abstract: Systems and methods of the present disclosure include interactive editing for generated three-dimensional (3D) models, such as those represented by neural radiance fields (NeRFs). A 3D model may be presented to a user in which the user may identify one or more localized regions for editing and/or modification. The localized regions may be selected and a corresponding 3D volume for that region may be provided to one or more generative networks, along with a prompt, to generate new content for the localized regions. Each of the original NeRF and the newly generated NeRF for the new content may then be combined into a single NeRF for a combined 3D representation with the original content and the localized modifications.

Abstract: Approaches presented herein provide systems and methods for generating three-dimensional (3D) objects using compressed data as an input. One or more models may learn from a hash table of latent features to map different features to a reconstruction domain, using a hash function as part of a learned process. A 3D shape for an object may be encoded to a multi-layered grid and represented by a series of embeddings, where given point within the grid may be interpolated based on the embeddings for a given layer of the multi-layered grid. A decoder may then be trained to use the embeddings to generate an output object.

Abstract: Techniques for grammar-based automated generation of annotated synthetic form training data for machine learning are described. A training data generation engine utilizes a defined grammar to construct a layout for a form, select key-value units to place within the layout, and select attribute variants for the key-value units. The form is rendered and stored at a storage location, where it can be provided along with other similarly-generated forms to be used as training data for a machine learning model.

Abstract: Techniques for automated form understanding via layout-agnostic identification of keys and corresponding values are described. An embedding generator creates embeddings of pixels from an image including a representation of a form. The generated embeddings are similar for pixels within a same key-value unit, and far apart for pixels not in a same key-value unit. A weighted bipartite graph is constructed including a first set of nodes corresponding to keys of the form and a second set of nodes corresponding to values of the form. Weights for the edges are determined based on an analysis of distances between ones of the embeddings. The graph is partitioned according to a scheme to identify pairings between the first set of nodes and the second set of nodes that produces a minimum overall edge weight. The pairings indicate keys and values that are associated within the form.

Abstract: Techniques for layout-agnostic clustering-based classification of document keys and values are described. A key-value differentiation unit generates feature vectors corresponding to text elements of a form represented within an electronic image using a machine learning (ML) model. The ML model was trained utilizing a loss function that separates keys from values. The feature vectors are clustered into at least two clusters, and a cluster is determined to include either keys of the form or values of the form via identifying neighbors between feature vectors of the cluster(s) with labeled feature vectors.