Abstract: One embodiment of the present invention sets forth a technique for performing landmark detection. The technique includes generating, via execution of a first machine learning model, a first set of morphable model coefficients associated with a first object depicted in a first image. The technique also includes determining one or more three-dimensional (3D) landmarks on the first object based on the first set of morphable model coefficients and projecting the first set of 3D landmarks onto the first image to generate one or more two-dimensional (2D) landmarks. The technique further includes training the first machine learning model based on one or more losses associated with the one or more 2D landmarks to generate a first trained machine learning model.

Abstract: The present invention sets forth a technique for simulating wrinkles under dynamic facial expression. The technique includes receiving a wrinkle graph, including a plurality of nodes associated with a plurality of pores included in a three-dimensional (3D) representation of a facial structure and a plurality of edges associated with a plurality of wrinkles included in the 3D representation of a facial structure. The technique also includes assigning one or more of the plurality of wrinkles associated with edges in the wrinkle graph to one of a plurality of bins and generating, for each of the bins, a plurality of pre-computed displacement texture maps. The technique further includes generating a per-frame displacement texture map and modifying an animation frame based on the per-frame displacement texture map, such that the modified animation frame depicts the plurality of pores and the plurality of wrinkles included in the 3D representation of the facial structure.

Abstract: The present invention sets forth a technique for generating two-dimensional (2D) renderings of a three-dimensional (3D) scene from an arbitrary camera position under arbitrary lighting conditions. This technique includes determining, based on a plurality of 2D representations of a 3D scene, a radiance field function for a neural radiance field (NeRF) model. This technique further includes determining, based on a plurality of 2D representations of a 3D scene, a radiance field function for a “one light at a time” (OLAT) model. The technique further includes rendering a 2D representation of the scene based on a given camera position and illumination data. The technique further includes computing a rendering loss based on the difference between the rendered 2D representation and an associated one of the plurality of 2D representations of the scene. The technique further includes modifying at least one of the NeRF and OLAT models based on the rendering loss.

Abstract: A technique for generating a sequence of geometries includes converting, via an encoder neural network, one or more input geometries corresponding to one or more frames within an animation into one or more latent vectors. The technique also includes generating the sequence of geometries corresponding to a sequence of frames within the animation based on the one or more latent vectors. The technique further includes causing output related to the animation to be generated based on the sequence of geometries.

Abstract: Techniques are disclosed for generating photorealistic images of head portraits. A rendering application renders a set of images that include the skin of a face and corresponding masks indicating pixels associated with the skin in the images. An inpainting application performs a neural projection technique to optimize a set of parameters that, when input into a generator model, produces a set of projection images, each of which includes a head portrait in which (1) skin regions resemble the skin regions of the face in a corresponding rendered image; and (2) non-skin regions match the non-skin regions in the other projection images when the rendered set of images are standalone images, or transition smoothly between consecutive projection images in the case when the rendered set of images are frames of a video. The rendered images can then be blended with corresponding projection images to generate composite images that are photorealistic.

Abstract: One embodiment of the present invention sets forth a technique for generating a geometry for a shape. The technique includes inputting, into a machine learning model, (i) a noise sample and (ii) one or more conditioning inputs. The technique also includes generating, via execution of the machine learning model based on the noise sample and the one or more conditioning inputs, a two-dimensional (2D) position map associated with the shape. The technique further includes generating a three-dimensional (3D) geometry for the shape based on the 2D position map.

Abstract: One embodiment of the present invention sets forth a technique for training a machine learning model on a geometry generation task. The technique includes generating, via execution of a diffusion model, a first set of training output corresponding to a first set of three-dimensional (3D) geometries based on a first set of conditioning inputs associated with a first conditioning mode, and training the diffusion model based on a first set of loss values associated with the first set of training output. The technique further includes generating, via execution of the diffusion model and a first adapter model, a second set of training output corresponding to a second set of 3D geometries based on a second set of conditioning inputs associated with a second conditioning mode, and training the first adapter model based on a second set of loss values associated with the second set of training output.

Abstract: A technique for performing style transfer between a content sample and a style sample is disclosed. The technique includes applying one or more neural network layers to a first latent representation of the style sample to generate one or more convolutional kernels. The technique also includes generating convolutional output by convolving a second latent representation of the content sample with the one or more convolutional kernels. The technique further includes applying one or more decoder layers to the convolutional output to produce a style transfer result that comprises one or more content-based attributes of the content sample and one or more style-based attributes of the style sample.

Abstract: In one embodiment, a method for generating an output sequence of data utilizing a spline-based transformer is disclosed. The method may include encoding, via a processing element, an input sequence of data using an artificial neural network encoder to generate a plurality of input tokens; processing, via the processing element, the plurality of input tokens and a plurality of control tokens with a transformer encoder into a latent space to generate a plurality of control points; defining, via the processing element, a spline based on the plurality of control points; sampling, via the processing element, a plurality of interpolated control points based on the spline; and decoding, via the processing element, the interpolated control points with an artificial neural network decoder to generate the output sequence of data.

Abstract: Techniques are disclosed for re-aging images of faces and three-dimensional (3D) geometry representing faces. In some embodiments, an image of a face, an input age, and a target age, are input into a re-aging model, which outputs a re-aging delta image that can be combined with the input image to generate a re-aged image of the face. In some embodiments, 3D geometry representing a face is re-aged using local 3D re-aging models that each include a blendshape model for finding a linear combination of sample patches from geometries of different facial identities and generating a new shape for the patch at a target age based on the linear combination. In some embodiments, 3D geometry representing a face is re-aged by performing a shape-from-shading technique using re-aged images of the face captured from different viewpoints, which can optionally be constrained to linear combinations of sample patches from local blendshape models.

Abstract: The present invention sets forth techniques for generating a facial animation. The techniques include receiving a latent identity code including a first set of features describing a neutral facial depiction associated with an identity and receiving a latent expression code including a second set of features describing a facial expression associated with the identity. The techniques also include generating, via a first machine learning model, an identity-specific facial representation based on a canonical facial representation and the latent identity code and generating, via a second machine learning model and based on the latent identity code, the latent expression code, and the identity-specific facial representation, a muscle actuation field tensor and one or more bone transformations associated with the deformed canonical facial representation.

Abstract: Methods and systems for generating three-dimensional (3D) models and facial hair models representative of subjects (e.g., actors or actresses) using facial scanning technology. Initial subject facial data, including facial frames and facial performance frames (e.g., images of the subject collected from a capture system) can be used to accurately predict the structure of the subject's face underneath their facial hair to produce a reference 3D facial shape of the subject. Likewise, image processing techniques can be used to identify facial hairs and generate a reference facial hair model. The reference 3D facial shape and reference facial hair mode can subsequently be used to generate performance 3D facial shapes and a performance facial hair model corresponding to a performance by the subject (e.g., reciting dialog).

Abstract: Embodiments of the present disclosure are directed to methods and systems for generating three-dimensional (3D) models and facial hair models representative of subjects (e.g., actors or actresses) using facial scanning technology. Methods accord to embodiments may be useful for performing facial capture on subjects with dense facial hair. Initial subject facial data, including facial frames and facial performance frames (e.g., images of the subject collected from a capture system) can be used to accurately predict the structure of the subject's face underneath their facial hair to produce a reference 3D facial shape of the subject. Likewise, image processing techniques can be used to identify facial hairs and generate a reference facial hair model. The reference 3D facial shape and reference facial hair mode can subsequently be used to generate performance 3D facial shapes and a performance facial hair model corresponding to a performance by the subject (e.g., reciting dialog).

Abstract: Various embodiments include a system for rendering an object, such as human skin or a human head, from captured appearance data. The system includes a processor executing a near field lighting reconstruction module. The system determines at least one of a three-dimensional (3D) position or a 3D orientation of a lighting unit based on a plurality of captured images of a mirror sphere. For each point light source in a plurality of point light sources included in the lighting unit, the system determines an intensity associated with the point light source. The system determines captures appearance data of the object, where the object is illuminated by the lighting unit. The system renders an image of the object based on the appearance data and the intensities associated with each point light source in the plurality of point light sources.

Abstract: A technique for performing style transfer between a content sample and a style sample is disclosed. The technique includes applying one or more neural network layers to a first latent representation of the style sample to generate one or more convolutional kernels. The technique also includes generating convolutional output by convolving a second latent representation of the content sample with the one or more convolutional kernels. The technique further includes applying one or more decoder layers to the convolutional output to produce a style transfer result that comprises one or more content-based attributes of the content sample and one or more style-based attributes of the style sample.

Abstract: Various embodiments include a system for rendering an object, such as human skin or a human head, from captured appearance data comprising a plurality of texels. The system includes a processor executing a texture space indirect illumination module. The system determines texture coordinates of a vector originating from a first texel where the vector intersects a second texel. The system renders the second texel from the viewpoint of the first texel based on appearance data at the second texel. Based on the rendering of the second texel, the system determines an indirect lighting intensity incident to the first texel from the second texel. The system updates appearance data at the first texel based on a direct lighting intensity and the indirect lighting intensity. The system renders the first texel based on the updated appearance data at the first texel.

Abstract: The present invention sets forth techniques for generating an animation sequence. The techniques include receiving one or more three-dimensional (3D) input meshes, wherein each input mesh includes a representation of an object included in a 3D scene. The techniques also include receiving, for each of the 3D input meshes, a virtual camera position associated with the 3D input mesh and one or more virtual lighting positions associated with the 3D input mesh. The techniques further include generating, for each of the 3D input meshes and via a trained machine learning model, one or more rendered frames associated with the 3D input mesh, wherein each rendered frame includes a two-dimensional (2D) representation of the object as viewed from the virtual camera position and illuminated by one or more virtual lights located at the one or more virtual lighting positions, and generating an output animation sequence based on the rendered frames.

Abstract: One embodiment of the present invention sets forth a technique for performing landmark detection. The technique includes generating, via execution of a first machine learning model, a first set of displacements associated with a first set of query points on a canonical shape based on a first annotation style associated with the first set of query points. The technique also includes determining, via execution of a second machine learning model, a first set of landmarks on a first face depicted in a first image based on the first set of displacements. The technique further includes training the first machine learning model based on one or more losses associated with the first set of landmarks to generate a first trained machine learning model.

Abstract: One embodiment of the present invention sets forth a technique for performing landmark detection. The technique includes applying, via execution of a first machine learning model, a first transformation to a first image depicting a first face to generate a second image. The technique also includes determining, via execution of a second machine learning model, a first set of landmarks on the first face based on the second image. The technique further includes training the first machine learning model based on one or more losses associated with the first set of landmarks to generate a first trained machine learning model.

Abstract: The present invention sets forth a technique for performing face micro detail recovery. The technique includes generating one or more skin texture displacement maps based on images of one or more skin surfaces. The technique also includes transferring, via one or more machine learning models, stylistic elements included in the one or more skin texture displacement maps onto one or more regions included in a modified three-dimensional (3D) facial reconstruction. The technique further includes generating a final 3D facial reconstruction that includes structural elements included in the 3D facial reconstruction and the stylistic elements included in the one or more skin texture displacement maps.