Abstract: A neural human performance capture framework (MVS-PERF) captures the skeleton, body shape and clothes displacement, and appearance of a person from a set of calibrated multiview images. It addresses the ambiguity of predicting the absolute position in monocular human mesh recovery, and bridges the volumetric representation from NeRF to animation-friendly performance capture. MVS-PERF includes three modules to extract feature maps from multiview images and fuse them to a feature volume, regress the feature volume to a naked human parameters vector, generating an SMPL-X skin-tight body mesh with a skeletal pose, body shape, and expression, and leverage a neural radiance field and a deformation field to infer the clothes as the displacement on the naked body using differentiable rendering. Clothed body mesh is obtained by adding the interpolated displacement vectors to the SMPL-X skin-tight body mesh vertices. The obtained radiance field is used for free-view volumetric rendering of the input subject.

Abstract: Methods and systems are provided for rendering photo-realistic images of a subject or an object using a differentiable neural network for predicting indirect light behavior. In one example, the differentiable neural network outputs a volumetric light map comprising a plurality of spherical harmonic representations. Further, using a reflectance neural network, roughness and scattering coefficients associated with the subject or the object is computed. The volumetric light map, as well as the roughness and scattering coefficients are the utilized for rendering a final image under one or more of a desired lighting condition, desired camera view angle, and/or with a desired visual effect (e.g., expression change).

Abstract: A neural human performance capture framework (MVS-PERF) captures the skeleton, body shape and clothes displacement, and appearance of a person from a set of calibrated multiview images. It addresses the ambiguity of predicting the absolute position in monocular human mesh recovery, and bridges the volumetric representation from NeRF to animation-friendly performance capture. MVS-PERF includes three modules to extract feature maps from multiview images and fuse them to a feature volume, regress the feature volume to a naked human parameters vector, generating an SMPL-X skin-tight body mesh with a skeletal pose, body shape, and expression, and leverage a neural radiance field and a deformation field to infer the clothes as the displacement on the naked body using differentiable rendering. Clothed body mesh is obtained by adding the interpolated displacement vectors to the SMPL-X skin-tight body mesh vertices. The obtained radiance field is used for free-view volumetric rendering of the input subject.