Abstract: An electronic device and method for generation of reflectance maps for relightable 3D models is disclosed. The electronic device acquires multi-view image data that includes a set of images of an object and generates a 3D mesh of the object based on the multi-view image data. The electronic device obtains a set of motion-corrected images based on a minimization of a rigid motion associated with the object between images of the set of images and generates texture maps in a UV space based on the set of motion-corrected images and the 3D mesh. The electronic device obtains specular and diffuse reflectance maps based on a separation of specular and diffuse reflectance components from the texture maps. and obtains a relightable 3D model of the object based on the specular and diffuse reflectance maps.

Abstract: An electronic device and method for positioning an eyeball into a 3D mesh of a head portion of an object is provided. Images of an eye of an object, the 3D mesh of the head portion of the object, and a 3D template mesh of an eyeball are acquired. 3D feature points for eye regions are extracted from the images and fit to a sphere. An initial pose transformation between the 3D template mesh and the sphere is estimated. Based on the 3D template mesh, a first set of points corresponding to the eye regions are interpolated. Based on sampling parameters for the first set of points, a second set of points are determined. A final pose transformation is determined based on minimization of difference between the first and the second set of points and the 3D template mesh is fit into an eyeball socket of the 3D mesh.

Abstract: An electronic device and method for 3D face modeling based on neural networks is provided. The electronic device receives a two-dimensional (2D) color image of a human face with a first face expression and obtains a first three-dimensional (3D) mesh of the human face with the first face expression based on the received 2D color image. The electronic device generates first texture information and a first set of displacement maps. The electronic device feeds the generated first texture information and the first set of displacement maps as an input to the neural network and receives an output of the neural network for the fed input. Thereafter, the electronic device generates a second 3D mesh of the human face with a second face expression which is different from the first face expression based on the received output.

Abstract: An electronic device and method for shape refinement of a 3D mesh reconstructed from images is disclosed. A set of images of an object is acquired and used to estimate a first 3D mesh of a head portion of the object. A first set of operations is executed on the first 3D mesh to generate a second 3D mesh. The first set of operations includes a removal of one or more regions which are unneeded for head-shape estimation and/or a removal of one or more mesh artifacts associated with a 3D shape or a topology of the first 3D mesh. A 3D template mesh is processed to determine a set of filling patches which corresponds to a set of holes in the second 3D mesh. Based on the second 3D mesh and the set of filling patches, a hole filling operation is executed to generate a final 3D mesh.

Abstract: A system and method for three-dimensional (3D) microgeometry and reflectance modeling is provided. The system receives images comprising a first set of images of a face and a second set of images of the face. The faces in the first set of images and the second set of images are exposed to omni-directional lighting and directional lighting, respectively. The system generates a 3D face mesh based on the received images and executes a set of skin-reflectance modeling operations by using the generated 3D face mesh and the second set of images, to estimate a set of texture maps for the face. Based on the estimated set of texture maps, the system texturizes the generated 3D face mesh. The texturization includes an operation in which texture information, including microgeometry skin details and skin reflectance details, of the estimated set of texture maps is mapped onto the generated 3D face mesh.

Abstract: An electronic device and method of preservation of geometry details in tracked meshes is provided. A set of three-dimensional (3D) scans and a sequence of tracked meshes of an object of interest are acquired. The sequence of tracked meshes comprises a set of tracked meshes which temporally corresponds to the set of 3D scans. Based on differences between the set of tracked meshes and the set of 3D scans, a set of displacement maps is generated. A plurality of vectors is computed, each of which includes surface tension values associated with mesh vertices of a corresponding tracked mesh. Based on the set of displacement maps and a corresponding set of vectors, a neural network is trained on a task of displacement map generation. The trained neural network model is applied on the plurality of vectors to generate displacement maps. Each tracked mesh is updated based on a corresponding displacement map.

Abstract: An electronic device and method for texture map generation are disclosed. A set of color images from a set of viewpoints and depth information of a subject's face are acquired. A 3D model is generated based on a color image and the depth information. A set of viewpoint-specific projections of the 3D model are generated along the set of viewpoints. The 3D model is refined based on minimization of difference between each viewpoint-specific projection and a corresponding color image. A set of texture maps corresponding to the set of viewpoints is generated, based on the refined 3D model, and set of color images. A dynamic programming workflow is executed to determine seams along which respective portions of the set of texture maps are to be stitched and the respective portions are stitched along the seams to generate a final texture map for the refined 3D model of the subject's face.

Abstract: The mesh tracking described herein involves mesh tracking on 3D face models. In contrast to existing mesh tracking algorithms which generally require user intervention and manipulation, the mesh tracking algorithm is fully automatic once a template mesh is provided. In addition, an eye and mouth boundary detection algorithm is able to better reconstruct the shape of eyes and mouths.

Abstract: An electronic device and method for eyeball positioning for 3D head modeling is provided. Images of an eye of an object, a 3D mesh of a head portion of the object, and a 3D template mesh of an eyeball are acquired. 3D feature points for eye regions are extracted from the images. The 3D feature points are fit to a sphere. An initial pose transformation between the 3D template mesh and the sphere is estimated. Based on the 3D template mesh, a first set of points corresponding to the eye regions are interpolated. Based on sampling parameters for the first set of points, a second set of points are determined. A final pose transformation is determined based on minimization of difference between the first and the second set of points. The 3D template mesh is fit into an eyeball socket of the 3D mesh based on the final pose transformation.

Abstract: An electronic device and method of preservation of geometry details in tracked meshes is provided. A set of three-dimensional (3D) scans and a sequence of tracked meshes of an object of interest are acquired. The sequence of tracked meshes comprises a set of tracked meshes which temporally corresponds to the set of 3D scans. Based on differences between the set of tracked meshes and the set of 3D scans, a set of displacement maps is generated. A plurality of vectors is computed, each of which includes surface tension values associated with mesh vertices of a corresponding tracked mesh. Based on the set of displacement maps and a corresponding set of vectors, a neural network is trained on a task of displacement map generation. The trained neural network model is applied on the plurality of vectors to generate displacement maps. Each tracked mesh is updated based on a corresponding displacement map.

Abstract: An electronic device and method for shape refinement of a 3D mesh reconstructed from images is disclosed. A set of images of an object is acquired and used to estimate a first 3D mesh of a head portion of the object. A first set of operations is executed on the first 3D mesh to generate a second 3D mesh. The first set of operations includes a removal of one or more regions which are unneeded for head-shape estimation and/or a removal of one or more mesh artifacts associated with a 3D shape or a topology of the first 3D mesh. A 3D template mesh is processed to determine a set of filling patches which corresponds to a set of holes in the second 3D mesh. Based on the second 3D mesh and the set of filling patches, a hole filling operation is executed to generate a final 3D mesh.

Abstract: An electronic device and method for 3D modeling based on 2D warping is disclosed. The electronic device acquires a color image of a face of a user, depth information corresponding to the color image, and a point cloud of the face. A 3D mean-shape model of a reference 3D face is acquired, and rigid aligned with the point cloud. A 2D projection of the aligned 3D mean-shape model is generated. The 2D projection includes a set of landmark points associated with the aligned 3D mean-shape model. The 2D projection is warped such that the set of landmark points in the 2D projection is aligned with a corresponding set of feature points in the color image. A 3D correspondence between the aligned 3D mean-shape model and the point cloud is determined for a non-rigid alignment of the aligned 3D mean-shape model, based on the warped 2D projection and the depth information.

Abstract: A system and method for three-dimensional (3D) microgeometry and reflectance modeling is provided. The system receives images comprising a first set of images of a face and a second set of images of the face. The faces in the first set of images and the second set of images are exposed to omni-directional lighting and directional lighting, respectively. The system generates a 3D face mesh based on the received images and executes a set of skin-reflectance modeling operations by using the generated 3D face mesh and the second set of images, to estimate a set of texture maps for the face. Based on the estimated set of texture maps, the system texturizes the generated 3D face mesh. The texturization includes an operation in which texture information, including microgeometry skin details and skin reflectance details, of the estimated set of texture maps is mapped onto the generated 3D face mesh.

Abstract: The mesh tracking described herein involves mesh tracking on 3D face models. In contrast to existing mesh tracking algorithms which generally require user intervention and manipulation, the mesh tracking algorithm is fully automatic once a template mesh is provided. In addition, an eye and mouth boundary detection algorithm is able to better reconstruct the shape of eyes and mouths.

Abstract: An electronic device and method for human skin detection based on a human body-prior is provided. A color image of a person is acquired, and a 3D body model of the person is estimated based on the color image. One or more unclothed parts of the 3D body model are identified. The one or more unclothed parts correspond to one or more body parts, of which at least a portion of skin remains uncovered by clothes. From the color image, pixel information corresponding to the one or more unclothed parts is extracted and classification information is determined based on the pixel information. The classification information includes a set of values, each indicating a likelihood of whether a corresponding pixel of the color image is part of an unclothed skin portion of the person. The unclothed skin portion is detected in the color image based on the classification information.

Abstract: An electronic device and method for human skin detection based on a human body-prior is provided. A color image of a person is acquired, and a 3D body model of the person is estimated based on the color image. One or more unclothed parts of the 3D body model are identified. The one or more unclothed parts correspond to one or more body parts, of which at least a portion of skin remains uncovered by clothes. From the color image, pixel information corresponding to the one or more unclothed parts is extracted and classification information is determined based on the pixel information. The classification information includes a set of values, each indicating a likelihood of whether a corresponding pixel of the color image is part of an unclothed skin portion of the person. The unclothed skin portion is detected in the color image based on the classification information.

Abstract: An electronic device and method for 3D human modeling under a specific body-fitting of clothes is disclosed. A set of depth images of a person wearing clothes is acquired and an initial body model is generated based on a first shape-fitting of a human body prior with at least a first depth scan. A posed-body model is obtained based on a pose-fitting of the initial body model with at least one depth scan. In each depth scan, clothed scan points and unclothed scan points are determined. A final body model is determined based on a second shape-fitting of vertices which belong to the posed-body model and correspond to an under-cloth skin portion of the body of the person. The second shape-fitting is based on the clothed scan points and the unclothed scan points. The final body model is textured based on a skin texture map for the body.

Abstract: A system and method for neural-network based image-to-image translation is provided. The system includes a generator network which receives an image and generates a first synthetic image based on the received image. The first synthetic image is different from the image in appearance, style, or domain. The system includes a feature-to-image generator that extracts a feature map from an intermediate layer of the generator network and generates a second synthetic image. The system further includes a first discriminator network that generates a result indicative of whether the first synthetic image is real or fake in comparison to a first reference image and a second discriminator network that generates a result indicative of whether the second synthetic image is real or fake in comparison to a second reference image. The system further includes a processor which trains the first generator network in adversarial manner based on losses associated with the results.

Abstract: An electronic device and method for 3D face modeling based on neural networks is provided. The electronic device receives a two-dimensional (2D) color image of a human face with a first face expression and obtains a first three-dimensional (3D) mesh of the human face with the first face expression based on the received 2D color image. The electronic device generates first texture information and a first set of displacement maps. The electronic device feeds the generated first texture information and the first set of displacement maps as an input to the neural network and receives an output of the neural network for the fed input. Thereafter, the electronic device generates a second 3D mesh of the human face with a second face expression which is different from the first face expression based on the received output.

Abstract: An electronic apparatus and method for image-based mask frame interpolation is provided. The electronic apparatus receives a sequence of image frames, including a group of image frames and an intermediate image frame. The electronic apparatus generates foreground (FG) masks based on application of a trained Neural Network on the group of image frames and selects a Region-of-Interest (RoI) from a first image frame of the group of image frames. The electronic apparatus extracts first feature vectors for pixels within the selected RoI and second feature vector for a first pixel of the intermediate image frame. The electronic apparatus estimates a first mask value for the first pixel based on application of a statistical function on the first feature vectors and the second feature vector and generates an intermediate FG mask based on the estimated first mask value and re-estimation of mask values for remaining pixels of the intermediate image frame.