Abstract: In particular embodiments, a computing system may capture a first image of a scene using a first camera of an artificial reality device. The system may capture a second image of the scene using a second camera and one or more optical elements of the artificial reality device. The second image may include an overlapping portion of multiple shifted copies of the scene. The system may generate an upsampled first image by applying a particular sampling technique to the first image. The system may generate a tiled image comprising a plurality of repeated second images by applying a tiling process to the second image. The system may generate an initial output image by processing the upsampled first image and the tiled image using a machine learning model. The system may generate a final output image by normalizing the initial output image using the upsampled first image.

Abstract: A distributed imaging system for augmented reality devices is disclosed. The system includes a computing module in communication with a plurality of spatially distributed sensing devices. The computing module is configured to process input images from the sensing devices based on performing a local feature matching computation to generate corresponding first output images. The computing module is further configured to process the input images based on performing an optical flow correspondence computation to generate corresponding second output images. The computing module is further configured to computationally combine first and second output images to generate third output images.

Abstract: In one embodiment, a method includes determining a viewing direction of a scene and rendering an image of the scene for the viewing direction, wherein the rendering comprises: for each pixel of the image, casting a view ray into the scene, and for a particular sampling point along the view ray, determining a pixel radiance associated with surface light field (SLF) and opacity, which comprises identifying multiple voxels within a threshold distance to the particular sampling point, wherein each of the voxels is associated with a respective local plane, for each the voxels computing a pixel radiance associated with SLF and opacity based on locations of the particular sampling point and the local plane associated with that voxel, and determining the pixel radiance associated with SLF and opacity for the particular sampling point based on interpolating the pixel radiances associated with SLF and opacity associated with the multiple voxels.

Abstract: In one embodiment, a method includes accessing a digital image captured by a camera that is connected to a machine-detectable object, detecting a reflection of the machine-detectable object in the digital image, computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues, and storing information associated with the reflective surface, where the information includes a pose of the reflective surface and the boundary of the reflective surface in a 3D model of a physical environment, and where the information associated with the reflective surface and the 3D model are configured to be used to render a reconstruction of the physical environment.

Abstract: In one embodiment, a method includes initializing latent codes respectively associated with times associated with frames in a training video of a scene captured by a camera. For each of the frames, a system (1) generates rendered pixel values for a set of pixels in the frame by querying NeRF using the latent code associated with the frame, a camera viewpoint associated with the frame, and ray directions associated with the set of pixels, and (2) updates the latent code associated with the frame and the NeRF based on comparisons between the rendered pixel values and original pixel values for the set of pixels. Once trained, the system renders output frames for an output video of the scene, wherein each output frame is rendered by querying the updated NeRF using one of the updated latent codes corresponding to a desired time associated with the output frame.

Abstract: In one embodiment, a method includes accessing a digital image captured by a camera that is connected to a machine-detectable object, detecting a reflection of the machine-detectable object in the digital image, computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues, and storing information associated with the reflective surface, where the information includes a pose of the reflective surface and the boundary of the reflective surface in a 3D model of a physical environment, and where the information associated with the reflective surface and the 3D model are configured to be used to render a reconstruction of the physical environment.

Abstract: In one embodiment, a method includes accessing a digital image captured by a camera that is connected to a machine-detectable object, detecting a reflection of the machine-detectable object in the digital image, computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues, and storing information associated with the reflective surface, where the information includes a pose of the reflective surface and the boundary of the reflective surface in a 3D model of a physical environment, and where the information associated with the reflective surface and the 3D model are configured to be used to render a reconstruction of the physical environment.