Abstract: Example embodiments relate to techniques for volumetric performance capture with neural rendering. A technique may involve initially obtaining images that depict a subject from multiple viewpoints and under various lighting conditions using a light stage and depth data corresponding to the subject using infrared cameras. A neural network may extract features of the subject from the images based on the depth data and map the features into a texture space (e.g., the UV texture space). A neural renderer can be used to generate an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view. The neural render may resample the features of the subject from the texture space to an image space to generate the output image.

Abstract: Apparatus and methods related to applying lighting models to images of objects are provided. An example method includes applying a geometry model to an input image to determine a surface orientation map indicative of a distribution of lighting on an object based on a surface geometry. The method further includes applying an environmental light estimation model to the input image to determine a direction of synthetic lighting to be applied to the input image. The method also includes applying, based on the surface orientation map and the direction of synthetic lighting, a light energy model to determine a quotient image indicative of an amount of light energy to be applied to each pixel of the input image. The method additionally includes enhancing, based on the quotient image, a portion of the input image. One or more neural networks can be trained to perform one or more of the aforementioned aspects.

Abstract: Mechanisms for generating compressed images are provided. More particularly, methods, systems, and media for capturing, reconstructing, compressing, and rendering view-dependent immersive light field video with a layered mesh representation are provided.

Abstract: Mechanisms for generating compressed images are provided. More particularly, methods, systems, and media for capturing, reconstructing, compressing, and rendering view-dependent immersive light field video with a layered mesh representation are provided.

Abstract: Systems, methods, and computer program products are described that implement obtaining, at an electronic computing device and for at least one image of a scene rendered in an Augmented Reality (AR) environment, a scene lighting estimation captured at a first time period. The scene lighting estimation may include at least a first image measurement associated with the scene. The implementations may include determining, at the electronic computing device, a second image measurement associated with the scene at a second time period, determining a function of the first image measurement and the second image measurement. Based on the determined function, the implementations may also include triggering calculation of a partial lighting estimation update or triggering calculation of a full lighting estimation update and rendering, on a screen of the electronic computing device and for the scene, the scene using the partial lighting estimation update or the full lighting estimation update.

Abstract: Techniques of estimating lighting from portraits includes generating a lighting estimate from a single image of a face based on a machine learning (ML) system using multiple bidirectional reflection distribution functions (BRDFs) as a loss function. In some implementations, the ML system is trained using images of faces formed with HDR illumination computed from LDR imagery. The technical solution includes training a lighting estimation model in a supervised manner using a dataset of portraits and their corresponding ground truth illumination.

Abstract: An example method, apparatus, and computer-readable storage medium are provided to predict high-dynamic range (HDR) lighting from low-dynamic range (LDR) background images. In an example implementation, a method may include receiving low-dynamic range (LDR) background images of scenes, each LDR background image captured with appearance of one or more reference objects with different reflectance properties; and training a lighting estimation model based at least on the received LDR background images to predict high-dynamic range (HDR) lighting based at least on the trained model. In another example implementation, a method may include capturing a low-dynamic range (LDR) background image of a scene from an LDR video captured by a camera of the electronic computing device; predicting high-dynamic range (HDR) lighting for the image, the predicting, using a trained model, based at least on the LDR background image; and rendering a virtual object based at least on the predicted HDR lighting.

Abstract: Mechanisms for generating compressed images are provided. More particularly, methods, systems, and media for capturing, reconstructing, compressing, and rendering view-dependent immersive light field video with a layered mesh representation are provided.

Abstract: Systems, methods, and computer program products are described that implement obtaining, at an electronic computing device and for at least one image of a scene rendered in an Augmented Reality (AR) environment, a scene lighting estimation captured at a first time period. The scene lighting estimation may include at least a first image measurement associated with the scene. The implementations may include determining, at the electronic computing device, a second image measurement associated with the scene at a second time period, determining a function of the first image measurement and the second image measurement. Based on the determined function, the implementations may also include triggering calculation of a partial lighting estimation update or triggering calculation of a full lighting estimation update and rendering, on a screen of the electronic computing device and for the scene, the scene using the partial lighting estimation update or the full lighting estimation update.

Abstract: Methods, systems, and media for relighting images using predicted deep reflectance fields are provided.

Abstract: Systems and methods for lighting inserted content are provided. For example, the inserted content may include augmented reality content that is inserted into an image of a physical space. An example system and method may include determining a location within an image to insert content. For example, the image may be captured by a camera device. The example system and method may also include identifying a region of the image based on the determined location to insert the content, determining at least one lighting parameter based on the identified region, and rendering the content using the determined at least one lighting parameter.

Abstract: Methods, systems, and media for relighting images using predicted deep reflectance fields are provided.

Abstract: Systems and methods for lighting inserted content are provided. For example, the inserted content may include augmented reality content that is inserted into an image of a physical space. An example system and method may include determining a location within an image to insert content. For example, the image may be captured by a camera device. The example system and method may also include identifying a region of the image based on the determined location to insert the content, determining at least one lighting parameter based on the identified region, and rendering the content using the determined at least one lighting parameter.

Abstract: An imaging apparatus comprises a gantry with a pan and tilt rotating mechanism to which an elongated member is attached. The elongated member enables placing an image capture device at a forward offset from the center of rotation of the pan and tilt rotating mechanism. A real-world scenery is captured via rotation of the pan and tilt rotating mechanism which inscribes a sphere and positions the imaging device at the nodal points of the sphere for capturing images.

Abstract: Provided is a method and apparatus for realistically reproducing an eyeball that may verify and analyze a material property and a deformation property with respect to each of constituent portions of an eyeball and may render each of the constituent portions based on the analyzed priorities, thereby more realistically reproducing the eyeball.

Abstract: A controllable lighting system may include a plurality of light source groups, a group controller for each light source group, a master controller, and a network communication system. Each group controller may be configured to control the light sources in its light source group based on a group control command. The master controller may be configured to receive a master control command relating to the light sources and to issue a group control command to each of the group controllers that collectively effectuate compliance with the master control command. The network communication system may be configured to communicate the group control commands from the master controller to the group controllers.

Abstract: A high dynamic range image editing system for editing an image file having pixels spanning a first range of light intensity levels in an image editing system that only displays differences in the light intensity levels of pixels within a second range of light intensity levels that is less than the first range of light intensity levels, without reducing the range of light intensity levels in the image file.

Abstract: A lighting apparatus may be configured to illuminate a subject while the subject is undergoing a motion during a time period. An imaging system may be configured to generate image data representative of a sequence of frames of the moving subject. A controller may be configured to drive the lighting apparatus and the imaging system so that the lighting apparatus sequentially illuminates the moving subject with a time-multiplexed series of lighting conditions, and so that each one of the frames shows the subject illuminated with a respective one of the lighting conditions. The controller may be further configured to process the image data to generate re-illumination data representative of novel illumination conditions under which the subject can be re-illuminated, subsequent to the time period.

Abstract: A lighting reproduction apparatus for illuminating a subject includes a reproduction light optical source that generates reproduction light. The optical source includes a plurality of light emitters, each characterized by an individual color channel. There may be nine different color channels. A driver drives the light emitter color channels with intensity values at which a substantial spectral match is achieved between the reproduction light and the desired illuminant, so that the subject appears to be illuminated by the desired illuminant. These channel intensity values may be determined by solving a minimization equation that minimizes a sum of square residuals of the reproduction light spectra to the desired illuminant spectra. The output reproduction light may be metamerically, matched with the desired illuminant, with respect to a particular camera's spectral response. One or more spectral reflectances of the subject may be measured and incorporated into the optimization process.