Abstract: Example embodiments relate to techniques for volumetric performance capture with neural rendering. A technique may involve initially obtaining images that depict a subject from multiple viewpoints and under various lighting conditions using a light stage and depth data corresponding to the subject using infrared cameras. A neural network may extract features of the subject from the images based on the depth data and map the features into a texture space (e.g., the UV texture space). A neural renderer can be used to generate an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view. The neural render may resample the features of the subject from the texture space to an image space to generate the output image.

Abstract: Example embodiments relate to techniques for volumetric performance capture with neural rendering. A technique may involve initially obtaining images that depict a subject from multiple viewpoints and under various lighting conditions using a light stage and depth data corresponding to the subject using infrared cameras. A neural network may extract features of the subject from the images based on the depth data and map the features into a texture space (e.g., the UV texture space). A neural renderer can be used to generate an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view. The neural render may resample the features of the subject from the texture space to an image space to generate the output image.

Abstract: Methods, systems, and media for generating compressed images are provided. In some embodiments, the method comprises: identifying a multi-plane image, MPI, that represents a three-dimensional image, wherein the MPI comprises a plurality of fronto-parallel planes; splitting the MPI into a plurality of sub-volumes, wherein each sub-volume in the plurality of sub-volumes includes a subset of the plurality of fronto-parallel planes; calculating, for each sub-volume of the MPI, a depthmap; converting each depthmap to a mesh, wherein each mesh corresponds to a layer of a plurality of layers associated with a multi-depth image, MDI, to be rendered; calculating, for each layer of the plurality of layers, an image; and storing the meshes corresponding to the plurality of layers of the MDI and the images corresponding to the plurality of layers of the MDI as the MDI.

Abstract: Methods, systems, and media for generating compressed images are provided. In some embodiments, the method comprises: identifying a multi-plane image, MPI, that represents a three-dimensional image, wherein the MPI comprises a plurality of fronto-parallel planes; splitting the MPI into a plurality of sub-volumes, wherein each sub-volume in the plurality of sub-volumes includes a subset of the plurality of fronto-parallel planes; calculating, for each sub-volume of the MPI, a depthmap; converting each depthmap to a mesh, wherein each mesh corresponds to a layer of a plurality of layers associated with a multi-depth image, MDI, to be rendered; calculating, for each layer of the plurality of layers, an image; and storing the meshes corresponding to the plurality of layers of the MDI and the images corresponding to the plurality of layers of the MDI as the MDI.

Abstract: Example embodiments relate to techniques for volumetric performance capture with neural rendering. A technique may involve initially obtaining images that depict a subject from multiple viewpoints and under various lighting conditions using a light stage and depth data corresponding to the subject using infrared cameras. A neural network may extract features of the subject from the images based on the depth data and map the features into a texture space (e.g., the UV texture space). A neural renderer can be used to generate an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view. The neural render may resample the features of the subject from the texture space to an image space to generate the output image.

Abstract: Methods, systems, and media for generating compressed images are provided. In some embodiments, the method comprises: identifying a multi-plane image, MPI, that represents a three-dimensional image, wherein the MPI comprises a plurality of fronto-parallel planes; splitting the MPI into a plurality of sub-volumes, wherein each sub-volume in the plurality of sub-volumes includes a subset of the plurality of fronto-parallel planes; calculating, for each sub-volume of the MPI, a depthmap; converting each depthmap to a mesh, wherein each mesh corresponds to a layer of a plurality of layers associated with a multi-depth image, MDI, to be rendered; calculating, for each layer of the plurality of layers, an image; and storing the meshes corresponding to the plurality of layers of the MDI and the images corresponding to the plurality of layers of the MDI as the MDI.

Abstract: Mechanisms for generating compressed images are provided. More particularly, methods, systems, and media for capturing, reconstructing, compressing, and rendering view-dependent immersive light field video with a layered mesh representation are provided.

Abstract: Mechanisms for generating compressed images are provided. More particularly, methods, systems, and media for capturing, reconstructing, compressing, and rendering view-dependent immersive light field video with a layered mesh representation are provided.

Abstract: Methods, systems, and media for generating compressed images are provided. In some embodiments, the method comprises: identifying a multi-plane image, MPI, that represents a three-dimensional image, wherein the MPI comprises a plurality of fronto-parallel planes; splitting the MPI into a plurality of sub-volumes, wherein each sub-volume in the plurality of sub-volumes includes a subset of the plurality of fronto-parallel planes; calculating, for each sub-volume of the MPI, a depthmap; converting each depthmap to a mesh, wherein each mesh corresponds to a layer of a plurality of layers associated with a multi-depth image, MDI, to be rendered; calculating, for each layer of the plurality of layers, an image; and storing the meshes corresponding to the plurality of layers of the MDI and the images corresponding to the plurality of layers of the MDI as the MDI.

Abstract: Mechanisms for generating compressed images are provided. More particularly, methods, systems, and media for capturing, reconstructing, compressing, and rendering view-dependent immersive light field video with a layered mesh representation are provided.

Abstract: Methods, systems, and media for generating compressed images are provided.

Abstract: Methods, systems, and media for generating compressed images are provided.

Abstract: A method for rendering a view from a lightfield includes identifying a ray associated with a portion of the view and selecting a set of camera views from a plurality of camera views representing the lightfield based on an intersection point of the ray with a plane. Each camera view has an associated contribution region disposed on the plane. The associated contribution region overlaps contribution regions associated with other camera views of the set of camera views at the intersection point. The method also includes determining a characteristic of the ray based on a contribution factor for each camera view of the set of camera views. The contribution factor is determined based on the relative position of the intersection point within the associated contribution region.

Abstract: A display system includes a first storage device to store a first lightfield and a second storage device to store a second lightfield. The first lightfield comprises a two-dimensional array of images, with each image comprising a corresponding two-dimensional array of image tiles, and the second lightfield comprises a compressed representation of the first lightfield. The system further includes at least one processor configured to selectively eliminate image tiles of images of the first lightfield from inclusion in the second lightfield based on identified redundancies between the eliminated image tiles and image tiles in the same tile positions of other images of the lightfield.

Abstract: A method for rendering a view from a lightfield includes identifying a ray associated with a portion of the view and selecting a set of camera views from a plurality of camera views representing the lightfield based on an intersection point of the ray with a plane. Each camera view has an associated contribution region disposed on the plane. The associated contribution region overlaps contribution regions associated with other camera views of the set of camera views at the intersection point. The method also includes determining a characteristic of the ray based on a contribution factor for each camera view of the set of camera views. The contribution factor is determined based on the relative position of the intersection point within the associated contribution region.

Abstract: A display system includes a first storage device to store a first lightfield and a second storage device to store a second lightfield. The first lightfield comprises a two-dimensional array of images, with each image comprising a corresponding two-dimensional array of image tiles, and the second lightfield comprises a compressed representation of the first lightfield. The system further includes at least one processor configured to selectively eliminate image tiles of images of the first lightfield from inclusion in the second lightfield based on identified redundancies between the eliminated image tiles and image tiles in the same tile positions of other images of the lightfield.

Abstract: Systems and methods are described for generating a virtual reality experience including generating a user interface with a plurality of regions on a display in a head-mounted display device. The head-mounted display device housing may include at least one pass-through camera device. The systems and methods can include obtaining image content from the at least one pass-through camera device and displaying a plurality of virtual objects in a first region of the plurality of regions in the user interface, the first region substantially filling a field of view of the display in the head-mounted display device. In response to detecting a change in a head position of a user operating the head-mounted display device, the methods and systems can initiate display of updated image content in a second region of the user interface.