Abstract: A computer-implemented method of changing a face within an output image or video frame that includes: receiving an input image that includes a face presenting a facial expression in a pose; processing the image with a neural network encoder to generate a latent space point that is an encoded representation of the image; decoding the latent space point to generate an initial output image in accordance with a desired facial identity but with the facial expression and pose of the face in the input image; identifying a feature of the facial expression in the initial output image to edit; applying an adjustment vector to a latent space point corresponding to the initial output image to generate an adjusted latent space point; and decoding the adjusted latent space point to generate an adjusted output image in accordance with the desired facial identity but with the facial expression and pose of the face in the input image altered in accordance with the adjustment vector

Abstract: Embodiments of the disclosure provide systems and methods for motion capture to generate content (e.g., motion pictures, television programming, videos, etc.). An actor or other performing being can have multiple markers on his or her face that are essentially invisible to the human eye, but that can be clearly captured by camera systems of the present disclosure. Embodiments can capture the performance using two different camera systems, each of which can observe the same performance but capture different images of that performance. For instance, a first camera system can capture the performance within a first light wavelength spectrum (e.g., visible light spectrum), and a second camera system can simultaneously capture the performance in a second light wavelength spectrum different from the first spectrum (e.g., invisible light spectrum such as the IR light spectrum). The images captured by the first and second camera systems can be combined to generate content.

Abstract: A method of transferring a facial expression from a subject to a computer generated character that includes receiving a plate with an image of the subject's facial expression, a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters, a model of a camera rig used to capture the plate, and a virtual lighting model that estimates lighting conditions when the image on the plate was captured.

Abstract: A method of transferring a facial expression from a subject to a computer generated character that includes: receiving a plate with an image of the subject's facial expression and an estimate of intrinsic parameters of a camera used to film the plate; generating a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters; solving for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the deformable model with a differentiable renderer and shape-from-shading techniques, using as inputs, the three-dimensional parameterized deformable model, estimated intrinsic camera parameters, estimated lighting conditions and albedo estimates over a series of iterations to infer geometry of the facial expression and generate an intermediate facial; generating, from the intermediate facial mesh, refined albedo estimates for the deformable model; and

Abstract: Embodiments of the disclosure provide systems and methods for motion capture to generate content (e.g., motion pictures, television programming, videos, etc.). An actor or other performing being can have multiple markers on his or her face that are essentially invisible to the human eye, but that can be clearly captured by camera systems of the present disclosure. Embodiments can capture the performance using two different camera systems, each of which can observe the same performance but capture different images of that performance. For instance, a first camera system can capture the performance within a first light wavelength spectrum (e.g., visible light spectrum), and a second camera system can simultaneously capture the performance in a second light wavelength spectrum different from the first spectrum (e.g., invisible light spectrum such as the IR light spectrum). The images captured by the first and second camera systems can be combined to generate content.

Abstract: Embodiments of the disclosure provide systems and methods for motion capture to generate content (e.g., motion pictures, television programming, videos, etc.). An actor or other performing being can have multiple markers on his or her face that are essentially invisible to the human eye, but that can be clearly captured by camera systems of the present disclosure. Embodiments can capture the performance using two different camera systems, each of which can observe the same performance but capture different images of that performance. For instance, a first camera system can capture the performance within a first light wavelength spectrum (e.g., visible light spectrum), and a second camera system can simultaneously capture the performance in a second light wavelength spectrum different from the first spectrum (e.g., invisible light spectrum such as the IR light spectrum). The images captured by the first and second camera systems can be combined to generate content.

Abstract: A method of transferring a facial expression from a subject to a computer generated character that includes receiving a plate with an image of the subject's facial expression, a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters, a model of a camera rig used to capture the plate, and a virtual lighting model that estimates lighting conditions when the image on the plate was captured.

Abstract: A method of transferring a facial expression from a subject to a computer generated character that includes: receiving a plate with an image of the subject's facial expression and an estimate of intrinsic parameters of a camera used to film the plate; generating a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters; solving for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the deformable model with a differentiable renderer and shape-from-shading techniques, using as inputs, the three-dimensional parameterized deformable model, estimated intrinsic camera parameters, estimated lighting conditions and albedo estimates over a series of iterations to infer geometry of the facial expression and generate an intermediate facial; generating, from the intermediate facial mesh, refined albedo estimates for the deformable model; and

Abstract: Embodiments of the disclosure provide systems and methods for motion capture to generate content (e.g., motion pictures, television programming, videos, etc.). An actor or other performing being can have multiple markers on his or her face that are essentially invisible to the human eye, but that can be clearly captured by camera systems of the present disclosure. Embodiments can capture the performance using two different camera systems, each of which can observe the same performance but capture different images of that performance. For instance, a first camera system can capture the performance within a first light wavelength spectrum (e.g., visible light spectrum), and a second camera system can simultaneously capture the performance in a second light wavelength spectrum different from the first spectrum (e.g., invisible light spectrum such as the IR light spectrum). The images captured by the first and second camera systems can be combined to generate content.

Abstract: Embodiments of the disclosure provide systems and methods for motion capture to generate content (e.g., motion pictures, television programming, videos, etc.). An actor or other performing being can have multiple markers on his or her face that are essentially invisible to the human eye, but that can be clearly captured by camera systems of the present disclosure. Embodiments can capture the performance using two different camera systems, each of which can observe the same performance but capture different images of that performance. For instance, a first camera system can capture the performance within a first light wavelength spectrum (e.g., visible light spectrum), and a second camera system can simultaneously capture the performance in a second light wavelength spectrum different from the first spectrum (e.g., invisible light spectrum such as the IR light spectrum). The images captured by the first and second camera systems can be combined to generate content.

Abstract: Embodiments of the disclosure provide systems and methods for motion capture to generate content (e.g., motion pictures, television programming, videos, etc.). An actor or other performing being can have multiple markers on his or her face that are essentially invisible to the human eye, but that can be clearly captured by camera systems of the present disclosure. Embodiments can capture the performance using two different camera systems, each of which can observe the same performance but capture different images of that performance. For instance, a first camera system can capture the performance within a first light wavelength spectrum (e.g., visible light spectrum), and a second camera system can simultaneously capture the performance in a second light wavelength spectrum different from the first spectrum (e.g., invisible light spectrum such as the IR light spectrum). The images captured by the first and second camera systems can be combined to generate content.

Abstract: Systems and techniques for dynamically capturing and reconstructing lighting are provided. The systems and techniques may be based on a stream of images capturing the lighting within an environment as a scene is shot. Reconstructed lighting data may be used to illuminate a character in a computer-generated environment as the scene is shot. For example, a method may include receiving a stream of images representing lighting of a physical environment. The method may further include compressing the stream of images to reduce an amount of data used in reconstructing the lighting of the physical environment and may further include outputting the compressed stream of images for reconstructing the lighting of the physical environment using the compressed stream, the reconstructed lighting being used to render a computer-generated environment.

Abstract: Systems and techniques for dynamically capturing and reconstructing lighting are provided. The systems and techniques may be based on a stream of images capturing the lighting within an environment as a scene is shot. Reconstructed lighting data may be used to illuminate a character in a computer-generated environment as the scene is shot. For example, a method may include receiving a stream of images representing lighting of a physical environment. The method may further include compressing the stream of images to reduce an amount of data used in reconstructing the lighting of the physical environment and may further include outputting the compressed stream of images for reconstructing the lighting of the physical environment using the compressed stream, the reconstructed lighting being used to render a computer-generated environment.

Abstract: A renderer allows for a flexible and temporally coherent ordering of strokes in the context of stroke-based animation. The relative order of the strokes is specified by the artist or inferred from geometric properties of the scene, such as occlusion, for each frame of a sequence, as a set of stroke pair-wise constraints. Using the received constraints, the strokes are partially ordered for each of the frames. Based on these partial orderings, for each frame, a permutation of the strokes is selected amongst the ones consistent with the frame's partial order, so as to globally improve the perceived temporal coherence of the animation. The sequence of frames can then, for instance, be rendered by ordering the strokes according to the selected set of permutations for the sequence of frames.

Abstract: A renderer allows for a flexible and temporally coherent ordering of strokes in the context of stroke-based animation. The relative order of the strokes is specified by the artist or inferred from geometric properties of the scene, such as occlusion, for each frame of a sequence, as a set of stroke pair-wise constraints. Using the received constraints, the strokes are partially ordered for each of the frames. Based on these partial orderings, for each frame, a permutation of the strokes is selected amongst the ones consistent with the frame's partial order, so as to globally improve the perceived temporal coherence of the animation. The sequence of frames can then, for instance, be rendered by ordering the strokes according to the selected set of permutations for the sequence of frames.

Abstract: A renderer allows for a flexible and temporally coherent ordering of strokes in the context of stroke-based animation. The relative order of the strokes is specified by the artist or inferred from geometric properties of the scene, such as occlusion, for each frame of a sequence, as a set of stroke pair-wise constraints. Using the received constraints, the strokes are partially ordered for each of the frames. Based on these partial orderings, for each frame, a permutation of the strokes is selected amongst the ones consistent with the frame's partial order, so as to globally improve the perceived temporal coherence of the animation. The sequence of frames can then, for instance, be rendered by ordering the strokes according to the selected set of permutations for the sequence of frames.

Abstract: A renderer for performing stroke-based rendering determines whether two given overlapping strokes depict an occlusion in a three-dimensional scene. The renderer may then use this information to determine whether to apply an occlusion constraint between the strokes when rendering an image or a frame from an animation. In one implementation, the renderer determines whether the two strokes together depict a single view patch of surface in the scene (i.e., a single portion of three-dimensional surface in the scene as seen from the rendering viewpoint). The renderer builds an image-space patch of surface defined from the union of the two overlapping strokes and then determines whether there exists a single three-dimensional view patch of surface that projects onto the image-space patch and that contains both strokes' three-dimensional anchor points. Which stroke occludes the other can be determined by the relative three-dimensional depth of the strokes' anchor points from the rendering viewpoint.

Abstract: A renderer for performing stroke-based rendering determines whether two given overlapping strokes depict an occlusion in a three-dimensional scene. The renderer may then use this information to determine whether to apply an occlusion constraint between the strokes when rendering an image or a frame from an animation. In one implementation, the renderer determines whether the two strokes together depict a single view patch of surface in the scene (i.e., a single portion of three-dimensional surface in the scene as seen from the rendering viewpoint). The renderer builds an image-space patch of surface defined from the union of the two overlapping strokes and then determines whether there exists a single three-dimensional view patch of surface that projects onto the image-space patch and that contains both strokes' three-dimensional anchor points. Which stroke occludes the other can be determined by the relative three-dimensional depth of the strokes' anchor points from the rendering viewpoint.

Abstract: A renderer allows for a flexible and temporally coherent ordering of strokes in the context of stroke-based animation. The relative order of the strokes is specified by the artist or inferred from geometric properties of the scene, such as occlusion, for each frame of a sequence, as a set of stroke pair-wise constraints. Using the received constraints, the strokes are partially ordered for each of the frames. Based on these partial orderings, for each frame, a permutation of the strokes is selected amongst the ones consistent with the frame's partial order, so as to globally improve the perceived temporal coherence of the animation. The sequence of frames can then, for instance, be rendered by ordering the strokes according to the selected set of permutations for the sequence of frames.