Abstract: In some embodiments, a computing system accesses multiple input images of a specular object with a scene. The computing system encodes near-field interreflections of the scene on the specular object to obtain a first set of feature representations of the specular object in multiple viewing directions based on the multiple input images. The computing system encodes far-field reflections of the scene on the specular object to obtain a second set of feature representations in the multiple viewing directions based on the multiple input images. The computing system determines a set of specular color values for the specular object in the multiple viewing directions based on the first set of feature representations and the second set of feature representations using a multi-layer perceptron algorithm. The computing system renders the specular object representation at least based on the set of specular color values using a neural rendering algorithm.

Abstract: In implementation of techniques for sampling light directions on neural materials, a computing device implements a light direction system to receive neural features of a material and an indication of a view direction toward the material. Using a mixture of analytical lobes, a normalizing flow, or a histogram prediction, the light direction system predicts a probability density function (PDF). The light direction system then samples the PDF, calculates prominence values for each of a plurality of candidate light directions based on the PDF, and determines a light direction based on the prominence values.

Abstract: Techniques for depth guided text-based editing of 3D neural radiance fields are provided. A method includes receiving input 2D images corresponding to views of a target and generating a 3D representation from the input 2D images. The 3D representation includes points forming a point cloud, where each point has a color and density value. The method also includes accumulating the color and density values to generate a volumetric 3D scene having a geometry, extracting distance maps from the volumetric 3D scene based on the geometry, and generating a plurality of masks associated with the target for each view. The method also includes aggregating the masks into the volumetric 3D scene using the geometry, providing the input 2D images, the masks, and the distance maps to a diffusion model, and modifying an appearance of the target in the volumetric 3D scene by providing a text command to the diffusion model.

Abstract: In implementing per-asset denoising for real-time rendering of neural radiance fields (NeRFs), a processing device receives a three-dimensional (3D) representation of a scene as a NeRF. The processing device generates an intermediate rendering of the scene using the NeRF. The intermediate rendering is denoised using a machine-learning model to generate a final rendering. The machine-learning model is trained on another rendering of this scene, which was rendered using a non-real-time, high-quality rendering scheme. In other words, the machine-learning model is optimized for each scene and provides a lightweight denoising network to provide real-time NeRF rendering while maintaining the high-quality visuals of non-real-time rendering schemes. The final rendering is then presented via a display device.

Abstract: Techniques for generation of compressed representations for appearance of fiber-based digital assets are described that support computationally efficient and high fidelity rendering of digital assets that include fiber primitives under a variety of lighting conditions and view directions. A processing device, for instance, receives a digital asset that includes fiber primitives to be included in a three-dimensional digital scene. The processing device generates a compressed representation of the digital asset that maintains a geometry of the digital asset and includes a precomputed light transport. The processing device then inserts the compressed representation into the digital scene, such as at a location relative to one or more light sources. The processing device applies one or more lighting effects to the compressed representation based on the precomputed light transport and the location relative to the one or more light sources.

Abstract: In implementation of techniques for generating meshes by decoding volume representations, a computing device implements a mesh generation system to receive digital images depicting an object from different angles. The mesh generation system generates a volume representation of the object using a transformer model based on the digital images. By decoding information from the volume representation using an algorithm, the mesh generation system then generates a mesh of the object from the volume representation. The mesh generation system then presents the mesh of the object in a user interface.

Abstract: In implementations of techniques and systems for digital hair generation using a parametric hair model, a processing device receives a shape parameter and a style parameter. The shape parameter represents the overall shape of a hair model, and the style parameter represents local strand details. Based on the shape parameter, a machine-learning model generates guide strands that sparsely represent the hair model. The machine-learning model also generates wisps based on the style parameter. The wisps indicate local strand details for the hair model. The processing device interpolates the wisps onto the guide strands to generate the digital hair model.

Abstract: Certain aspects and features of this disclosure relate to providing a controllable, dynamic appearance for neural 3D portraits. For example, a method involves projecting a color at points in a digital video portrait based on location, surface normal, and viewing direction for each respective point in a canonical space. The method also involves projecting, using the color, dynamic face normals for the points as changing according to an articulated head pose and facial expression in the digital video portrait. The method further involves disentangling, based on the dynamic face normals, a facial appearance in the digital video portrait into intrinsic components in the canonical space. The method additionally involves storing and/or rendering at least a portion of a head pose as a controllable, neural 3D portrait based on the digital video portrait using the intrinsic components.

Abstract: In implementation of techniques for sampling light directions on neural materials, a computing device implements a light direction system to receive neural features of a material and an indication of a view direction toward the material. Using a mixture of analytical lobes, a normalizing flow, or a histogram prediction, the light direction system predicts a probability density function (PDF). The light direction system then samples the PDF, calculates prominence values for each of a plurality of candidate light directions based on the PDF, and determines a light direction based on the prominence values.

Abstract: Certain aspects and features of this disclosure relate to providing a controllable, dynamic appearance for neural 3D portraits. For example, a method involves projecting a color at points in a digital video portrait based on location, surface normal, and viewing direction for each respective point in a canonical space. The method also involves projecting, using the color, dynamic face normals for the points as changing according to an articulated head pose and facial expression in the digital video portrait. The method further involves disentangling, based on the dynamic face normals, a facial appearance in the digital video portrait into intrinsic components in the canonical space. The method additionally involves storing and/or rendering at least a portion of a head pose as a controllable, neural 3D portrait based on the digital video portrait using the intrinsic components.

Abstract: A scene modeling system accesses a three-dimensional (3D) scene including a 3D object. The scene modeling system applies a silhouette bidirectional texture function (SBTF) model to the 3D object to generate an output image of a textured material rendered as a surface of the 3D object. Applying the SBTF model includes determining a bounding geometry for the surface of the 3D object. Applying the SBTF model includes determining, for each pixel of the output image, a pixel value based on the bounding geometry. The scene modeling system displays, via a user interface, the output image based on the determined pixel values.

Abstract: Embodiments are configured to render 3D models using an importance sampling method. First, embodiments obtain a 3D model including a plurality of density values corresponding to a plurality of locations in a 3D space, respectively. Embodiments then sample the color information from within a random subset of the plurality of locations using a probability distribution based on the plurality of density values. Embodiments have a higher probability to sample each location within the random subset of locations if the location has a higher density probability. Embodiments then an image depicting a view of the 3D model based on the sampling within the random subset of the plurality of locations.

Abstract: Techniques for generation of compressed representations for digital assets are described that support computationally efficient and high fidelity rendering of digital assets with a variety of geometries under arbitrary lighting conditions and view directions. A processing device, for instance, receives a digital asset defined by a three-dimensional geometry to be included in a digital scene. The processing device generates a compressed representation of the digital asset that maintains a geometry of the digital asset and includes a precomputed light transport. The processing device then deploys the compressed representation into the digital scene, such as at a location relative to one or more digital scene elements. The content processing system renders the digital asset by applying one or more lighting effects to the compressed representation based on the precomputed light transport and the location relative to the one or more digital scene elements.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for 3D model generation include obtaining a plurality of input images depicting an object and a set of 3D position embeddings, where each of the plurality of input images depicts the object from a different perspective, encoding the plurality of input images to obtain a plurality of 2D features corresponding to the plurality of input images, respectively, generating 3D features based on the plurality of 2D features and the set of 3D position embeddings, and generating a 3D model of the object based on the 3D features.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media that generate three-dimensional hybrid mesh-volumetric representations for digital objects. For instance, in one or more embodiments, the disclosed systems generate a mesh for a digital object from a plurality of digital images that portray the digital object using a multi-view stereo model. Additionally, the disclosed systems determine a set of sample points for a thin volume around the mesh. Using a neural network, the disclosed systems further generate a three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the set of sample points for the thin volume and the mesh.

Abstract: Embodiments of the present disclosure provide systems, methods, and computer storage media for generating editable synthesized views of scenes by inputting image rays into neural networks using neural basis decomposition. In embodiments, a set of input images of a scene depicting at least one object are collected and used to generate a plurality of rays of the scene. The rays each correspond to three-dimensional coordinates and viewing angles taken from the images. A volume density of the scene is determined by inputting the three-dimensional coordinates from the neural radiance fields into a first neural network to generate a 3D geometric representation of the object. An appearance decomposition is produced by inputting the three-dimensional coordinates and the viewing angles of the rays into a second neural network.

Abstract: Techniques for generation of compressed representations for appearance of fiber-based digital assets are described that support computationally efficient and high fidelity rendering of digital assets that include fiber primitives under a variety of lighting conditions and view directions. A processing device, for instance, receives a digital asset that includes fiber primitives to be included in a three-dimensional digital scene. The processing device generates a compressed representation of the digital asset that maintains a geometry of the digital asset and includes a precomputed light transport. The processing device then inserts the compressed representation into the digital scene, such as at a location relative to one or more light sources. The content processing system applies one or more lighting effects to the compressed representation based on the precomputed light transport and the location relative to the one or more light sources.

Abstract: Certain aspects and features of this disclosure relate to providing a controllable, dynamic appearance for neural 3D portraits. For example, a method involves projecting a color at points in a digital video portrait based on location, surface normal, and viewing direction for each respective point in a canonical space. The method also involves projecting, using the color, dynamic face normals for the points as changing according to an articulated head pose and facial expression in the digital video portrait. The method further involves disentangling, based on the dynamic face normals, a facial appearance in the digital video portrait into intrinsic components in the canonical space. The method additionally involves storing and/or rendering at least a portion of a head pose as a controllable, neural 3D portrait based on the digital video portrait using the intrinsic components.

Abstract: Embodiments of the present disclosure provide systems, methods, and computer storage media for generating editable synthesized views of scenes by inputting image rays into neural networks using neural basis decomposition. In embodiments, a set of input images of a scene depicting at least one object are collected and used to generate a plurality of rays of the scene. The rays each correspond to three-dimensional coordinates and viewing angles taken from the images. A volume density of the scene is determined by inputting the three-dimensional coordinates from the neural radiance fields into a first neural network to generate a 3D geometric representation of the object. An appearance decomposition is produced by inputting the three-dimensional coordinates and the viewing angles of the rays into a second neural network.

Abstract: A scene modeling system accesses a three-dimensional (3D) scene including a 3D object. The scene modeling system applies a silhouette bidirectional texture function (SBTF) model to the 3D object to generate an output image of a textured material rendered as a surface of the 3D object. Applying the SBTF model includes determining a bounding geometry for the surface of the 3D object. Applying the SBTF model includes determining, for each pixel of the output image, a pixel value based on the bounding geometry. The scene modeling system displays, via a user interface, the output image based on the determined pixel values.