Abstract: In one embodiment, a method includes determining characteristics of one or more areas in an image by analyzing pixels in the image, computing a sampling density for each of the one or more areas in the image based on the characteristics of the one or more areas, generating samples corresponding to the image by sampling pixels in each of the one or more areas according to the associated sampling density, and providing the samples to a machine-learning model as an input, where the machine-learning model is configured to reconstruct the image by processing the samples.

Abstract: In one embodiment, a method includes accessing a pair of stereo images for a scene, where each image of the pair of stereo images has incomplete pixel information and k channels, stacking the pair of stereo images to form a stacked input image with 2k channels, processing the stacked input image using a machine-learning model to generate a stacked output image with 2k channels, and separating the stacked output image with 2k channels into a pair of reconstructed stereo images for the scene, where each image of the pair of reconstructed stereo images has complete pixel information and k channels.

Abstract: In one embodiment, a computing system may access a video including a first frame and a second frame. The computing system may determine first sampling locations for the first frame and determine second sampling locations for the second frame by transforming the first sampling locations to the second frame according to an optical flow between the first frame and the second frame. The computing system may detect one or more invalid second sampling locations based on determining pixels in the first frame corresponding to the first sampling locations do not match pixels in the second frame corresponding to the second sampling locations. The computing system may reject the one or more invalid second sampling locations to determine third sampling locations for the second frame. The computing system may generate a sample of the video.

Abstract: In one embodiment, a computing system accesses a three-dimensional (3D) model of an environment, the 3D model comprising a virtual representation of an object in the environment. The computing system accesses an image of the object captured by a camera from a camera pose. The computing system accesses light source parameters associated with a virtual representation of a light source in the environment. The computing system renders, using the 3D model, pixels associated with the virtual representation of the object based on the light source parameters, the pixels being rendered from a virtual perspective corresponding to the camera pose. The computing system determines updated light source parameters based on a comparison of the rendered pixels to corresponding pixels located in the image of the object.

Abstract: In one embodiment, a method includes accessing first-resolution images corresponding to frames of a video, computing a motion vector based on a first-resolution image of a first frame in the video and a first-resolution image of a second frame in the video, generating a second-resolution warped image associated with the second frame by using the motion vector to warp a second-resolution reconstructed image associated with the first frame, generating a second-resolution intermediate image associated with the second frame based on the first-resolution image associated with the second frame, computing adjustment parameters by processing the first-resolution image associated with the second frame and the second-resolution warped image associated with the second frame using a machine-learning model, and adjusting pixels of the second-resolution intermediate image associated with the second frame based on the adjustment parameters to reconstruct a second-resolution reconstructed image associated with the second fra

Abstract: Embodiments described herein pertain to a machine-learning approach for shading. A system may determine a number of pixels associated with a viewpoint of a viewer. The system may determine, for each of the pixels, (1) a view direction based on the viewpoint and a pixel position of that pixel and (2) and a surface orientation of a surface visible to that pixel. The system may generate, using a first machine-learning model, a latent space representation of ambient lighting information associated with the pixels based on respective view directions and surface orientations. The system may determine color values for the pixels by processing the latent space representation of ambient lighting information using a second machine-learning model.

Abstract: In one embodiment, a method includes receiving a pair of stereo images having a resolution lower than a target resolution, generating an initial first feature map for a first image of the pair based on first channels associated with the first image and generating an initial second feature map for a second image of the pair based on second channels associated with the second image, generating a first feature map based on combining the first channels with the initial first feature map, generating a second feature map based on combining the second channels with the initial second feature map, up-sampling the first feature map and the second feature map to the target resolution, warping the up-sampled second feature map, and generating a reconstructed image corresponding to the first image having the target resolution based on the up-sampled first feature map and the up-sampled and warped second feature map.

Abstract: In one embodiment, a computing system may access a video including a first frame and a second frame. The computing system may determine first sampling locations for the first frame and determine second sampling locations for the second frame by transforming the first sampling locations to the second frame according to an optical flow between the first frame and the second frame. The computing system may select a subset of the second sampling locations based on a comparison between pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the second sampling locations. The computing system may define one or more rejection areas in the second frame based on the subset of the second sampling locations to determine third sampling locations in areas outside of the rejection areas. The computing system may generate a sample of the video.

Abstract: In one embodiment, a computing system may receive a video including a sequence of frames. The computing system may access a three-dimensional mask that specifies pixel-sampling locations, the three-dimensional mask having a first dimension and a second dimension corresponding to a spatial domain and a third dimension corresponding to a temporal domain. Blue noise property may be present in the pixel-sampling locations that are associated with each of a plurality of two-dimensional spatial slices of the three-dimensional mask in the spatial domain and the pixel-sampling locations that are associated with each of a plurality of one-dimensional temporal slices of the three-dimensional mask in the temporal domain. The computing system may generate a sample of the video by sampling the sequence of frames using the three-dimensional mask.

Abstract: In one embodiment, a method includes receiving a first frame associated with a first time and one or more second frames of a video having a resolution lower than a target resolution, wherein each second frame is associated with a second time prior to the first time, generating a first feature map for the first frame and one or more second feature maps for the one or more second frames, up-sampling the first feature map and the one or more second feature maps to the target resolution, warping each of the up-sampled second feature maps according to a motion estimation between the associated second time and the first time, and generating a reconstructed frame having the target resolution corresponding to the first frame by using a machine-learning model to process the up-sampled first feature map and the one or more up-sampled and warped second feature maps.

Abstract: In one embodiment, a computing system may receive a video including a sequence of frames. The computing system may access a three-dimensional mask that specifies pixel-sampling locations, the three-dimensional mask having a first dimension and a second dimension corresponding to a spatial domain and a third dimension corresponding to a temporal domain. Blue noise property may be present in the pixel-sampling locations that are associated with each of a plurality of two-dimensional spatial slices of the three-dimensional mask in the spatial domain and the pixel-sampling locations that are associated with each of a plurality of one-dimensional temporal slices of the three-dimensional mask in the temporal domain. The computing system may generate a sample of the video by sampling the sequence of frames using the three-dimensional mask.

Abstract: In one embodiment, a computing system may access a video including a first frame and a second frame. The computing system may determine first sampling locations for the first frame and determine second sampling locations for the second frame by transforming the first sampling locations to the second frame according to an optical flow between the first frame and the second frame. The computing system may select a subset of the second sampling locations based on a comparison between pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the second sampling locations. The computing system may define one or more rejection areas in the second frame based on the subset of the second sampling locations to determine third sampling locations in areas outside of the rejection areas. The computing system may generate a sample of the video.

Abstract: In one embodiment, a method includes determining characteristics of one or more areas in an image by analyzing pixels in the image, computing a sampling density for each of the one or more areas in the image based on the characteristics of the one or more areas, generating samples corresponding to the image by sampling pixels in each of the one or more areas according to the associated sampling density, and providing the samples to a machine-learning model as an input, where the machine-learning model is configured to reconstruct the image by processing the samples.

Abstract: In one embodiment, a method includes accessing a pair of stereo images for a scene, where each image of the pair of stereo images has incomplete pixel information and k channels, stacking the pair of stereo images to form a stacked input image with 2k channels, processing the stacked input image using a machine-learning model to generate a stacked output image with 2k channels, and separating the stacked output image with 2k channels into a pair of reconstructed stereo images for the scene, where each image of the pair of reconstructed stereo images has complete pixel information and k channels.

Abstract: In one embodiment, a method includes accessing first-resolution images corresponding to frames of a video, computing a motion vector based on a first-resolution image of a first frame in the video and a first-resolution image of a second frame in the video, generating a second-resolution warped image associated with the second frame by using the motion vector to warp a second-resolution reconstructed image associated with the first frame, generating a second-resolution intermediate image associated with the second frame based on the first-resolution image associated with the second frame, computing adjustment parameters by processing the first-resolution image associated with the second frame and the second-resolution warped image associated with the second frame using a machine-learning model, and adjusting pixels of the second-resolution intermediate image associated with the second frame based on the adjustment parameters to reconstruct a second-resolution reconstructed image associated with the second fra

Abstract: In one embodiment, a method includes receiving a first frame associated with a first time and one or more second frames of a video having a resolution lower than a target resolution, wherein each second frame is associated with a second time prior to the first time, generating a first feature map for the first frame and one or more second feature maps for the one or more second frames, up-sampling the first feature map and the one or more second feature maps to the target resolution, warping each of the up-sampled second feature maps according to a motion estimation between the associated second time and the first time, and generating a reconstructed frame having the target resolution corresponding to the first frame by using a machine-learning model to process the up-sampled first feature map and the one or more up-sampled and warped second feature maps.

Abstract: In one embodiment, a computing system may determine an orientation in a three-dimensional (3D) space and generate a plurality of coordinates in the 3D space based on the determined orientation. The system may access pre-determined ray trajectory definitions associated with the plurality of coordinates. The system may determine visibility information of one or more objects defined within the 3D space by projecting rays through the plurality of coordinates, wherein trajectories of the rays from the plurality of coordinates are determined based on the pre-determined ray trajectory definitions. The system may then generate an image of the one or more objects based on the determined visibility information of the one or more objects.

Abstract: In one embodiment, a computing system may receive current eye-tracking data associated with a user of a head-mounted display. The system may dynamically adjust a focal length of the head-mounted display based on the current eye-tracking data. The system may generate an in-focus image of a scene and a corresponding depth map of the scene. The system may generate a circle-of-confusion map for the scene based on the depth map. The circle-of-confusion map encodes a desired focal surface in the scene. The system may generate, using a machine-learning model, an output image with a synthesized defocus-blur effect by processing the in-focus image, the corresponding depth map, and the circle-of-confusion map of the scene. The system may display the output image with the synthesized defocus-blur effect to the user via the head-mounted display having the adjusted focal length.

Abstract: A method, computer readable medium, and system are disclosed for performing spatiotemporal filtering. The method includes identifying image data to be rendered, reconstructing the image data to create reconstructed image data, utilizing a filter including a neural network having one or more skip connections and one or more recurrent layers, and returning the reconstructed image data.

Abstract: In one embodiment, a system may access a training sample that includes training images and corresponding training depth maps of a scene, with the training images being associated with different predetermined viewpoints of the scene. The system may generate elemental images of the scene by processing the training images and the training depth maps using a machine-learning model. The elemental images are associated with more viewpoints of the scene than the predetermined viewpoints associated with the training images. The system may update the machine-learning model based on a comparison between the generated elemental images of the scene and target elemental images that are each associated with a predetermined viewpoint. The updated machine-learning model is configured to generate elemental images of a scene of interest based on input images and corresponding depth maps of the scene of interest from different viewpoints.