Abstract: A computing system and method can be used to render a 3D shape from one or more images. In particular, the present disclosure provides a general pipeline for learning articulated shape reconstruction from images (LASR). The pipeline can reconstruct rigid or nonrigid 3D shapes. In particular, the pipeline can automatically decompose non-rigidly deforming shapes into rigid motions near rigid-bones. This pipeline incorporates an analysis-by-synthesis strategy and forward-renders silhouette, optical flow, and color images which can be compared against the video observations to adjust the internal parameters of the model. By inverting a rendering pipeline and incorporating optical flow, the pipeline can recover a mesh of a 3D model from the one or more images input by a user.

Abstract: Systems and methods of the present disclosure are directed to a method that can include obtaining a 3D mesh comprising polygons and texture/shading data. The method can include rasterizing the 3D mesh to obtain a 2D raster comprising pixels and coordinates respectively associated with a subset of pixels. The method can include determining an initial color value for the subset of pixels based on the coordinates of the pixel and the associated shading/texture data. The method can include constructing a splat at the coordinates of a respective pixel. The method can include determining an updated color value for a respective pixel based on a weighting of the subset of splats to generate a 2D rendering of the 3D mesh based on the coordinates of a pixel and a splat.

Abstract: A computer-implemented method for decomposing videos into multiple layers (212, 213) that can be re-combined with modified relative timings includes obtaining video data including a plurality of image frames (201) depicting one or more objects. For each of the plurality of frames, the computer-implemented method includes generating one or more object maps descriptive of a respective location of at least one object of the one or more objects within the image frame. For each of the plurality of frames, the computer-implemented method includes inputting the image frame and the one or more object maps into a machine-learned layer Tenderer model. (220) For each of the plurality of frames, the computer-implemented method includes receiving, as output from the machine-learned layer Tenderer model, a background layer illustrative of a background of the video data and one or more object layers respectively associated with one of the one or more object maps.

Abstract: The present disclosure provides systems and methods that perform face reconstruction based on an image of a face. In particular, one example system of the present disclosure combines a machine-learned image recognition model with a face modeler that uses a morphable model of a human's facial appearance. The image recognition model can be a deep learning model that generates an embedding in response to receipt of an image (e.g., an uncontrolled image of a face). The example system can further include a small, lightweight, translation model structurally positioned between the image recognition model and the face modeler. The translation model can be a machine-learned model that is trained to receive the embedding generated by the image recognition model and, in response, output a plurality of facial modeling parameter values usable by the face modeler to generate a model of the face.

Abstract: The present disclosure provides systems and methods that perform face reconstruction based on an image of a face. In particular, one example system of the present disclosure combines a machine-learned image recognition model with a face modeler that uses a morphable model of a human's facial appearance. The image recognition model can be a deep learning model that generates an embedding in response to receipt of an image (e.g., an uncontrolled image of a face). The example system can further include a small, lightweight, translation model structurally positioned between the image recognition model and the face modeler. The translation model can be a machine-learned model that is trained to receive the embedding generated by the image recognition model and, in response, output a plurality of facial modeling parameter values usable by the face modeler to generate a model of the face.

Abstract: The present disclosure provides systems and methods that perform face reconstruction based on an image of a face. In particular, one example system of the present disclosure combines a machine-learned image recognition model with a face modeler that uses a morphable model of a human's facial appearance. The image recognition model can be a deep learning model that generates an embedding in response to receipt of an image (e.g., an uncontrolled image of a face). The example system can further include a small, lightweight, translation model structurally positioned between the image recognition model and the face modeler. The translation model can be a machine-learned model that is trained to receive the embedding generated by the image recognition model and, in response, output a plurality of facial modeling parameter values usable by the face modeler to generate a model of the face.

Abstract: The present disclosure provides systems and methods that perform face reconstruction based on an image of a face. In particular, one example system of the present disclosure combines a machine-learned image recognition model with a face modeler that uses a morphable model of a human's facial appearance. The image recognition model can be a deep learning model that generates an embedding in response to receipt of an image (e.g., an uncontrolled image of a face). The example system can further include a small, lightweight, translation model structurally positioned between the image recognition model and the face modeler. The translation model can be a machine-learned model that is trained to receive the embedding generated by the image recognition model and, in response, output a plurality of facial modeling parameter values usable by the face modeler to generate a model of the face.

Abstract: Methods, systems, and apparatus for obtaining first image features derived from an image of an object, providing the first image features to a three-dimensional estimator neural network, and obtaining, from the three-dimensional estimator neural network, data specifying an estimated three-dimensional shape and texture based on the first image features. The estimated three-dimensional shape and texture are provided to a three-dimensional rendering engine, and a plurality of three-dimensional views of the object are generated by the three-dimensional rendering engine based on the estimated three-dimensional shape and texture. The plurality of three-dimensional views are provided to the object recognition engine, and second image features derived from the plurality of three-dimensional views are obtained from the object recognition engine. A loss is computed based at least on the first and second image features, and the three-dimensional estimator neural network is trained based at least on the computed loss.

Abstract: Methods, systems, and apparatus for obtaining first image features derived from an image of an object, providing the first image features to a three-dimensional estimator neural network, and obtaining, from the three-dimensional estimator neural network, data specifying an estimated three-dimensional shape and texture based on the first image features. The estimated three-dimensional shape and texture are provided to a three-dimensional rendering engine, and a plurality of three-dimensional views of the object are generated by the three-dimensional rendering engine based on the estimated three-dimensional shape and texture. The plurality of three-dimensional views are provided to the object recognition engine, and second image features derived from the plurality of three-dimensional views are obtained from the object recognition engine. A loss is computed based at least on the first and second image features, and the three-dimensional estimator neural network is trained based at least on the computed loss.

Abstract: Methods, systems, and apparatus for obtaining first image features derived from an image of an object, providing the first image features to a three-dimensional estimator neural network, and obtaining, from the three-dimensional estimator neural network, data specifying an estimated three-dimensional shape and texture based on the first image features. The estimated three-dimensional shape and texture are provided to a three-dimensional rendering engine, and a plurality of three-dimensional views of the object are generated by the three-dimensional rendering engine based on the estimated three-dimensional shape and texture. The plurality of three-dimensional views are provided to the object recognition engine, and second image features derived from the plurality of three-dimensional views are obtained from the object recognition engine. A loss is computed based at least on the first and second image features, and the three-dimensional estimator neural network is trained based at least on the computed loss.

Abstract: Methods, systems, and apparatus for obtaining first image features derived from an image of an object, providing the first image features to a three-dimensional estimator neural network, and obtaining, from the three-dimensional estimator neural network, data specifying an estimated three-dimensional shape and texture based on the first image features. The estimated three-dimensional shape and texture are provided to a three-dimensional rendering engine, and a plurality of three-dimensional views of the object are generated by the three-dimensional rendering engine based on the estimated three-dimensional shape and texture. The plurality of three-dimensional views are provided to the object recognition engine, and second image features derived from the plurality of three-dimensional views are obtained from the object recognition engine. A loss is computed based at least on the first and second image features, and the three-dimensional estimator neural network is trained based at least on the computed loss.

Abstract: The present disclosure provides systems and methods that perform face reconstruction based on an image of a face. In particular, one example system of the present disclosure combines a machine-learned image recognition model with a face modeler that uses a morphable model of a human's facial appearance. The image recognition model can be a deep learning model that generates an embedding in response to receipt of an image (e.g., an uncontrolled image of a face). The example system can further include a small, lightweight, translation model structurally positioned between the image recognition model and the face modeler. The translation model can be a machine-learned model that is trained to receive the embedding generated by the image recognition model and, in response, output a plurality of facial modeling parameter values usable by the face modeler to generate a model of the face.