Abstract: Techniques are disclosed for re-aging images of faces and three-dimensional (3D) geometry representing faces. In some embodiments, an image of a face, an input age, and a target age, are input into a re-aging model, which outputs a re-aging delta image that can be combined with the input image to generate a re-aged image of the face. In some embodiments, 3D geometry representing a face is re-aged using local 3D re-aging models that each include a blendshape model for finding a linear combination of sample patches from geometries of different facial identities and generating a new shape for the patch at a target age based on the linear combination. In some embodiments, 3D geometry representing a face is re-aged by performing a shape-from-shading technique using re-aged images of the face captured from different viewpoints, which can optionally be constrained to linear combinations of sample patches from local blendshape models.

Abstract: The present invention sets forth techniques for generating a facial animation. The techniques include receiving a latent identity code including a first set of features describing a neutral facial depiction associated with an identity and receiving a latent expression code including a second set of features describing a facial expression associated with the identity. The techniques also include generating, via a first machine learning model, an identity-specific facial representation based on a canonical facial representation and the latent identity code and generating, via a second machine learning model and based on the latent identity code, the latent expression code, and the identity-specific facial representation, a muscle actuation field tensor and one or more bone transformations associated with the deformed canonical facial representation.

Abstract: Methods and systems for generating three-dimensional (3D) models and facial hair models representative of subjects (e.g., actors or actresses) using facial scanning technology. Initial subject facial data, including facial frames and facial performance frames (e.g., images of the subject collected from a capture system) can be used to accurately predict the structure of the subject's face underneath their facial hair to produce a reference 3D facial shape of the subject. Likewise, image processing techniques can be used to identify facial hairs and generate a reference facial hair model. The reference 3D facial shape and reference facial hair mode can subsequently be used to generate performance 3D facial shapes and a performance facial hair model corresponding to a performance by the subject (e.g., reciting dialog).

Abstract: Embodiments of the present disclosure are directed to methods and systems for generating three-dimensional (3D) models and facial hair models representative of subjects (e.g., actors or actresses) using facial scanning technology. Methods accord to embodiments may be useful for performing facial capture on subjects with dense facial hair. Initial subject facial data, including facial frames and facial performance frames (e.g., images of the subject collected from a capture system) can be used to accurately predict the structure of the subject's face underneath their facial hair to produce a reference 3D facial shape of the subject. Likewise, image processing techniques can be used to identify facial hairs and generate a reference facial hair model. The reference 3D facial shape and reference facial hair mode can subsequently be used to generate performance 3D facial shapes and a performance facial hair model corresponding to a performance by the subject (e.g., reciting dialog).

Abstract: Various embodiments include a system for rendering an object, such as human skin or a human head, from captured appearance data. The system includes a processor executing a near field lighting reconstruction module. The system determines at least one of a three-dimensional (3D) position or a 3D orientation of a lighting unit based on a plurality of captured images of a mirror sphere. For each point light source in a plurality of point light sources included in the lighting unit, the system determines an intensity associated with the point light source. The system determines captures appearance data of the object, where the object is illuminated by the lighting unit. The system renders an image of the object based on the appearance data and the intensities associated with each point light source in the plurality of point light sources.

Abstract: A technique for performing style transfer between a content sample and a style sample is disclosed. The technique includes applying one or more neural network layers to a first latent representation of the style sample to generate one or more convolutional kernels. The technique also includes generating convolutional output by convolving a second latent representation of the content sample with the one or more convolutional kernels. The technique further includes applying one or more decoder layers to the convolutional output to produce a style transfer result that comprises one or more content-based attributes of the content sample and one or more style-based attributes of the style sample.

Abstract: Various embodiments include a system for rendering an object, such as human skin or a human head, from captured appearance data comprising a plurality of texels. The system includes a processor executing a texture space indirect illumination module. The system determines texture coordinates of a vector originating from a first texel where the vector intersects a second texel. The system renders the second texel from the viewpoint of the first texel based on appearance data at the second texel. Based on the rendering of the second texel, the system determines an indirect lighting intensity incident to the first texel from the second texel. The system updates appearance data at the first texel based on a direct lighting intensity and the indirect lighting intensity. The system renders the first texel based on the updated appearance data at the first texel.

Abstract: The present invention sets forth techniques for generating an animation sequence. The techniques include receiving one or more three-dimensional (3D) input meshes, wherein each input mesh includes a representation of an object included in a 3D scene. The techniques also include receiving, for each of the 3D input meshes, a virtual camera position associated with the 3D input mesh and one or more virtual lighting positions associated with the 3D input mesh. The techniques further include generating, for each of the 3D input meshes and via a trained machine learning model, one or more rendered frames associated with the 3D input mesh, wherein each rendered frame includes a two-dimensional (2D) representation of the object as viewed from the virtual camera position and illuminated by one or more virtual lights located at the one or more virtual lighting positions, and generating an output animation sequence based on the rendered frames.

Abstract: One embodiment of the present invention sets forth a technique for performing landmark detection. The technique includes generating, via execution of a first machine learning model, a first set of displacements associated with a first set of query points on a canonical shape based on a first annotation style associated with the first set of query points. The technique also includes determining, via execution of a second machine learning model, a first set of landmarks on a first face depicted in a first image based on the first set of displacements. The technique further includes training the first machine learning model based on one or more losses associated with the first set of landmarks to generate a first trained machine learning model.

Abstract: One embodiment of the present invention sets forth a technique for performing landmark detection. The technique includes applying, via execution of a first machine learning model, a first transformation to a first image depicting a first face to generate a second image. The technique also includes determining, via execution of a second machine learning model, a first set of landmarks on the first face based on the second image. The technique further includes training the first machine learning model based on one or more losses associated with the first set of landmarks to generate a first trained machine learning model.

Abstract: The present invention sets forth a technique for performing face micro detail recovery. The technique includes generating one or more skin texture displacement maps based on images of one or more skin surfaces. The technique also includes transferring, via one or more machine learning models, stylistic elements included in the one or more skin texture displacement maps onto one or more regions included in a modified three-dimensional (3D) facial reconstruction. The technique further includes generating a final 3D facial reconstruction that includes structural elements included in the 3D facial reconstruction and the stylistic elements included in the one or more skin texture displacement maps.

Abstract: One embodiment of the present invention sets forth a technique for performing landmark detection. The technique includes determining a first set of parameters associated with a depiction of a first face in a first image. The technique also includes generating, via execution of a first machine learning model, a first set of three-dimensional (3D) landmarks on the first face based on the first set of parameters, and projecting, based on the first set of parameters, the first set of 3D landmarks onto the first image to generate a first set of two-dimensional (2D) landmarks. The technique further includes training the first machine learning model based on one or more losses associated with the first set of 2D landmarks to generate a first trained machine learning model.

Abstract: Embodiment of the present invention sets forth techniques for performing face reconstruction. The techniques include generating an identity mesh based on an identity encoding that represents an identity associated with a face in one or more images. The techniques also include generating an expression mesh based on an expression encoding that represents an expression associated with the face in the one or more images. The techniques also include generating, by a machine learning model, an output mesh of the face based on the identity mesh and the expression mesh.

Abstract: A technique for rendering an input geometry includes generating a first segmentation mask for a first input geometry and a first set of texture maps associated with one or more portions of the first input geometry. The technique also includes generating, via one or more neural networks, a first set of neural textures for the one or more portions of the first input geometry. The technique further includes rendering a first image corresponding to the first input geometry based on the first segmentation mask, the first set of texture maps, and the first set of neural textures.

Abstract: Techniques are disclosed for generating photorealistic images of objects, such as heads, from multiple viewpoints. In some embodiments, a morphable radiance field (MoRF) model that generates images of heads includes an identity model that maps an identifier (ID) code associated with a head into two codes: a deformation ID code encoding a geometric deformation from a canonical head geometry, and a canonical ID code encoding a canonical appearance within a shape-normalized space. The MoRF model also includes a deformation field model that maps a world space position to a shape-normalized space position based on the deformation ID code. Further, the MoRF model includes a canonical neural radiance field (NeRF) model that includes a density multi-layer perceptron (MLP) branch, a diffuse MLP branch, and a specular MLP branch that output densities, diffuse colors, and specular colors, respectively. The MoRF model can be used to render images of heads from various viewpoints.

Abstract: One embodiment of the present invention sets forth a technique for generating a shape model. The technique includes generating, via execution of a set of neural networks based on a plurality of shapes associated with an object, a set of attributes associated with a set of anatomical constraints for the object. The technique also includes computing, based on the set of attributes, a set of positions of a set of points on the object. The technique further includes generating a three-dimensional (3D) model of the object based on the set of positions of the set of points.

Abstract: The present invention sets forth a technique for performing facial rig generation. The technique includes generating a blendshape model including a plurality of vertices, a plurality of meshes, and a plurality of patches. The technique also includes modifying one or more blendweight values associated with each of the plurality of patches based on a plurality of facial depictions included in a facial database and one or more sample depictions of a target character and generating an output facial rig model based on the blendshape model and the one or more modified blendweight values. The technique further includes generating one or more expressive depictions of the target character based at least on the output facial rig.

Abstract: One embodiment of the present invention sets forth a technique for fitting a shape model for an object to a set of constraints associated with a target shape. The technique includes determining, based on the set of constraints, one or more ground truth positions of one or more points on the target shape. The technique also includes generating, via execution of a set of neural networks, a set of fitting parameters associated with the point(s) and computing, via the shape model, one or more predicted positions of the point(s) based on the set of fitting parameters. The technique further includes training the set of neural networks based on one or more losses associated with the predicted position(s) and the ground truth position(s) and generating, via execution of the trained set of neural networks, a three-dimensional (3D) model corresponding to the target shape.

Abstract: A technique for rendering an input geometry includes generating a first segmentation mask for a first input geometry and a first set of texture maps associated with one or more portions of the first input geometry. The technique also includes generating, via one or more neural networks, a first set of neural textures for the one or more portions of the first input geometry. The technique further includes rendering a first image corresponding to the first input geometry based on the first segmentation mask, the first set of texture maps, and the first set of neural textures.

Abstract: A technique for synthesizing a shape includes generating a first plurality of offset tokens based on a first shape code and a first plurality of position tokens, wherein the first shape code represents a variation of a canonical shape, and wherein the first plurality of position tokens represent a first plurality of positions on the canonical shape. The technique also includes generating a first plurality of offsets associated with the first plurality of positions on the canonical shape based on the first plurality of offset tokens. The technique further includes generating the shape based on the first plurality of offsets and the first plurality of positions.