Abstract: Apparatus and methods related to light redistribution in images are provided. An example method includes receiving, by a computing device, an input image comprising a subject. The method further includes adjusting, by a neural network, one or more of a specular component or a diffuse component associated with the input image. The adjusting involves redistributing a per-pixel light energy of the input image. The method additionally includes predicting, by the neural network, an output image comprising the subject with the adjusted one or more of the specular component or the diffuse component.

Abstract: A method including capturing, by a wearable device, a plurality of images each having a first resolution, process, by the wearable device, the plurality of images to generate a first image having a second resolution, the second resolution being smaller than the first resolution, selecting, by the wearable device, a second image from the plurality of images having the first resolution based on a setting of the wearable device, and communicating, by the wearable device to a companion device, the first image and the second image. Further, processing, by the companion device, the first image, and merging, by the companion device, the processed first image with the second image to generate a high dynamic range (HDR) image.

Abstract: A method including, in a training phase, training a gaze prediction model including a first model and a second model, the first model and the second model being configured in conjunction to predict segmentation data based on training data, training a third model together with the first model and the second model, the third model being configured to predict a training characteristic using an output of the first model based on the training data, and in an operational phase, receiving operational data and predicting an operational characteristic using the trained first model and the trained third model.

Abstract: Nonlinear peri-codec optimization for image and video coding includes obtaining a source image including pixel values expressed in a first defined image sample space, generating a neuralized image representing the source image, the neuralized image including pixel values that are expressed as neural latent space values, encoding the input image wherein the neural latent space values are used as pixel values in a second defined image sample space and the input image is in an operative image format of the encoder, such that a decoder decodes the encoded image to obtain a reconstructed image in the second defined image sample space, wherein the reconstructed image is a reconstructed neuralized image including reconstructed neural latent space values, such that a deneuralized reconstructed image corresponding to the source image is obtained by a nonlinear post-codec image processor in the first defined image sample space.

Abstract: A method including generating a first vector based on a first grid and a three-dimensional (3D) position associated with a first implicit representation (IR) of a 3D object, generating at least one second vector based on at least one second grid and an upsampled first grid, decoding the first vector to generate a second IR of the 3D object, decoding the at least one second vector to generate at least one third IR of the 3D object, generating a composite IR of the 3D object based on the second IR of the 3D object and the at least one third IR of the 3D object, and generating a reconstructed volume representing the 3D object based on the composite IR of the 3D object.

Abstract: A method comprises receiving a first sequence of images of a portion of a user, the first sequence of images being monocular images; generating an avatar based on the first sequence of images, the avatar being based on a model including a feature vector associated with a vertex; receiving a second sequence of images of the portion of the user; and based on the second sequence of images, modifying the avatar with a displacement of the vertex to represent a gesture of the avatar.

Abstract: A lighting stage includes a plurality of lights that project alternating spherical color gradient illumination patterns onto an object or human performer at a predetermined frequency. The lighting stage also includes a plurality of cameras that capture images of an object or human performer corresponding to the alternating spherical color gradient illumination patterns. The lighting stage also includes a plurality of depth sensors that capture depth maps of the object or human performer at the predetermined frequency. The lighting stage also includes (or is associated with) one or more processors that implement a machine learning algorithm to produce a three-dimensional (3D) model of the object or human performer. The 3D model includes relighting parameters used to relight the 3D model under different lighting conditions.

Abstract: Apparatus and methods related to applying lighting models to images are provided. An example method includes receiving, via a computing device, an image comprising a subject. The method further includes relighting, via a neural network, a foreground of the image to maintain a consistent lighting of the foreground with a target illumination. The relighting is based on a per-pixel light representation indicative of a surface geometry of the foreground. The light representation includes a specular component, and a diffuse component, of surface reflection. The method additionally includes predicting, via the neural network, an output image comprising the subject in the relit foreground. One or more neural networks can be trained to perform one or more of the aforementioned aspects.

Abstract: Systems and methods for training models to predict dense correspondences across images such as human images. A model may be trained using synthetic training data created from one or more 3D computer models of a subject. In addition, one or more geodesic distances derived from the surfaces of one or more of the 3D models may be used to generate one or more loss values, which may in turn be used in modifying the model's parameters during training.

Abstract: Systems and methods for training models to predict dense correspondences across images such as human images. A model may be trained using synthetic training data created from one or more 3D computer models of a subject. In addition, one or more geodesic distances derived from the surfaces of one or more of the 3D models may be used to generate one or more loss values, which may in turn be used in modifying the model's parameters during training.

Abstract: Systems and methods for training models to predict dense correspondences across images such as human images. A model may be trained using synthetic training data created from one or more 3D computer models of a subject. In addition, one or more geodesic distances derived from the surfaces of one or more of the 3D models may be used to generate one or more loss values, which may in turn be used in modifying the model's parameters during training.

Abstract: Techniques include introducing a neural generator configured to produce novel faces that can be rendered at free camera viewpoints (e.g., at any angle with respect to the camera) and relit under an arbitrary high dynamic range (HDR) light map. A neural implicit intrinsic field takes a randomly sampled latent vector as input and produces as output per-point albedo, volume density, and reflectance properties for any queried 3D location. These outputs are aggregated via a volumetric rendering to produce low resolution albedo, diffuse shading, specular shading, and neural feature maps. The low resolution maps are then upsampled to produce high resolution maps and input into a neural renderer to produce relit images.

Abstract: Techniques of tracking a user's gaze includes identifying a region of a display at which a gaze of a user is directed, the region including a plurality of pixels. By determining a region rather than a point, when the regions correspond to elements of a user interface, the improved technique enables a system to activate the element to which a determined region is selected. In some implementations, the system makes the determination using a classification engine including a convolutional neural network; such an engine takes as input images of the user's eye and outputs a list of probabilities that the gaze is directed to each of the regions.

Abstract: Example embodiments relate to techniques for volumetric performance capture with neural rendering. A technique may involve initially obtaining images that depict a subject from multiple viewpoints and under various lighting conditions using a light stage and depth data corresponding to the subject using infrared cameras. A neural network may extract features of the subject from the images based on the depth data and map the features into a texture space (e.g., the UV texture space). A neural renderer can be used to generate an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view. The neural render may resample the features of the subject from the texture space to an image space to generate the output image.

Abstract: Apparatus and methods related to applying lighting models to images of objects are provided. An example method includes applying a geometry model to an input image to determine a surface orientation map indicative of a distribution of lighting on an object based on a surface geometry. The method further includes applying an environmental light estimation model to the input image to determine a direction of synthetic lighting to be applied to the input image. The method also includes applying, based on the surface orientation map and the direction of synthetic lighting, a light energy model to determine a quotient image indicative of an amount of light energy to be applied to each pixel of the input image. The method additionally includes enhancing, based on the quotient image, a portion of the input image. One or more neural networks can be trained to perform one or more of the aforementioned aspects.

Abstract: According to an aspect, a real-time active stereo system includes a capture system configured to capture stereo data, where the stereo data includes a first input image and a second input image, and a depth sensing computing system configured to predict a depth map. The depth sensing computing system includes a feature extractor configured to extract features from the first and second images at a plurality of resolutions, an initialization engine configured to generate a plurality of depth estimations, where each of the plurality of depth estimations corresponds to a different resolution, and a propagation engine configured to iteratively refine the plurality of depth estimations based on image warping and spatial propagation.

Abstract: Systems and methods are directed to encoding and/or decoding of the textures/geometry of a three-dimensional volumetric representation. An encoding computing system can obtain voxel blocks from a three-dimensional volumetric representation of an object. The encoding computing system can encode voxel blocks with a machine-learned voxel encoding model to obtain encoded voxel blocks. The encoding computing system can decode the encoded voxel blocks with a machine-learned voxel decoding model to obtain reconstructed voxel blocks. The encoding computing system can generate a reconstructed mesh representation of the object based at least in part on the one or more reconstructed voxel blocks. The encoding computing system can encode textures associated with the voxel blocks according to an encoding scheme and based at least in part on the reconstructed mesh representation of the object to obtain encoded textures.

Abstract: Techniques of tracking a user's gaze includes identifying a region of a display at which a gaze of a user is directed, the region including a plurality of pixels. By determining a region rather than a point, when the regions correspond to elements of a user interface, the improved technique enables a system to activate the element to which a determined region is selected. In some implementations, the system makes the determination using a classification engine including a convolutional neural network; such an engine takes as input images of the user's eye and outputs a list of probabilities that the gaze is directed to each of the regions.

Abstract: A method including, in a training phase, training a gaze prediction model including a first model and a second model, the first model and the second model being configured in conjunction to predict segmentation data based on training data, training a third model together with the first model and the second model, the third model being configured to predict a training characteristic using an output of the first model based on the training data, and in an operational phase, receiving operational data and predicting an operational characteristic using the trained first model and the trained third model.

Abstract: A method includes receiving a first image including color data and depth data, determining a viewpoint associated with an augmented reality (AR) and/or virtual reality (VR) display displaying a second image, receiving at least one calibration image including an object in the first image, the object being in a different pose as compared to a pose of the object in the first image, and generating the second image based on the first image, the viewpoint and the at least one calibration image.