Abstract: In one embodiment, a method includes a step of receiving a geometric representation of a virtual object and a texture atlas, the geometric representation comprising multiple geometric primitives defining a shape of the virtual object, the texture atlas comprises regions each of which is allocated to include shading information of a respective geometric primitive of the plurality of geometric primitives, and the shading information of the respective geometric primitive being scaled down to be smaller than the allocated region so as to create a buffer between the allocated region and adjacent regions of the plurality of regions on the texture atlas. The method further includes steps of identifying, based on a first viewpoint from which to view the virtual object, visible geometric primitives from the plurality of geometric primitives and rendering images of the visible geometric primitives using corresponding shading information included in the texture atlas.

Abstract: A method for rendering a frame includes receiving, from a server, a color texture and a depth mesh associated with an object to be rendered, the depth mesh including multiple vertices and metadata associated with the vertices. The method further includes receiving motion data associated with the object, warping the depth mesh using the metadata and the motion data, resulting in a warped depth mesh, and rendering one or more frames using the color texture and the warped depth mesh.

Abstract: In one embodiment, a method includes receiving a first viewpoint associated with a head-mounted device from the head-mounted device, accessing a 3D mesh of a virtual scene, selecting a portion of the 3D mesh based on the first viewpoint, generating an image and a corresponding depth map of the virtual scene based on the selected portion of the 3D mesh, generating a simplified 3D mesh based on the depth map, wherein the simplified 3D mesh has fewer primitives than the selected portion of 3D mesh of the virtual scene, generating a texture for the simplified 3D mesh based on the image, and sending the simplified 3D mesh and the texture to the head-mount device, wherein the simplified 3D mesh and the texture are configured to be used for rendering the virtual scene from one or more viewpoints different from the first viewpoint.

Abstract: Particular embodiments described herein present a technique for compressing a 3D mesh. A computing system may access a topology-coding list and a vertex list representing a 3D mesh. The vertex list may comprise X, Y, and Z coordinates for ordered vertices in the 3D mesh. The computing system may construct a predicted vertex list based on the vertex list. The computing system may generate X, Y, and Z coordinate bit streams. Each coordinate bit stream may comprise ordered coordinate values for a corresponding coordinate in the predicted vertex list. Each coordinate value in a coordinate bit stream may be represented in a corresponding number of bits. The corresponding number of bits may be stored in a memory-size list corresponding to the coordinate bit stream. The computing system may encode the topology-coding list and memory-size lists corresponding to the X, Y, and Z coordinate bit streams using Zstandard coder.

Abstract: In one embodiment, a method includes a server generating first shading information for visible portions of objects relative to a first viewpoint, storing the first shading information in a texture atlas, and sending the texture atlas to a client device. The method also includes determining a first subset of the visible portions of the objects for which shading information is to be re-generated and a second subset for which elements of the first shading information are to be reused, generating second shading information for the first subset relative to a second viewpoint, updating the texture atlas to include the second shading information for the first subset and the elements of the first shading information for the second subset, and sending the updated texture atlas to the client device. The updated texture atlas is configured for rendering images of the visible portions of the objects from multiple viewpoints.

Abstract: In one embodiment, a method includes a step of receiving a geometric representation of a virtual object and a texture atlas, the geometric representation comprising multiple geometric primitives defining a shape of the virtual object, the texture atlas comprises regions each of which is allocated to include shading information of a respective geometric primitive of the plurality of geometric primitives, and the shading information of the respective geometric primitive being scaled down to be smaller than the allocated region so as to create a buffer between the allocated region and adjacent regions of the plurality of regions on the texture atlas. The method further includes steps of identifying, based on a first viewpoint from which to view the virtual object, visible geometric primitives from the plurality of geometric primitives and rendering images of the visible geometric primitives using corresponding shading information included in the texture atlas.

Abstract: In one embodiment, a method includes the steps of receiving, from a client device, a first viewpoint from which to view a virtual object, the virtual object having a shape defined by multiple geometric primitives, identifying, relative to the first viewpoint, visible geometric primitives from multiple geometric primitives, allocating a region in a texture atlas for each of the visible geometric primitives, generating shading information for each of the visible geometric primitives, storing the shading information of each of the visible geometric primitives in a portion of the allocated region smaller than the allocated region to create a buffer around the portion of the allocated region where the shading information is stored, and sending, to the client device, the texture atlas and a list identifying the visible geometric primitives, the texture atlas being configured for rendering images of the visible geometric primitives from different viewpoints.

Abstract: A method includes the steps of receiving training data comprising images of an object and associated camera poses from which the images are captured, training, based on the training data, a machine-learning model to take as input a given viewpoint and synthesize an image of a virtual representation of the object viewed from the given viewpoint, generating, for each of predetermined viewpoints surrounding the virtual representation of the object, a view-dependent image of the object as viewed from that viewpoint using the trained machine-learning model, receiving, from a client device, a desired viewpoint from which to view the virtual representation of the object, selecting one or more of the predetermined viewpoints based on the desired viewpoint, and sending, to the client device, the view-dependent images associated with the selected one or more viewpoints for rendering an output image of the virtual representation of the object viewed from the desired viewpoint.

Abstract: In one embodiment, a method includes the steps of receiving, from a client device, a first viewpoint from which to view a virtual object, the virtual object having a shape defined by multiple geometric primitives, identifying, relative to the first viewpoint, visible geometric primitives from multiple geometric primitives, allocating a region in a texture atlas for each of the visible geometric primitives, generating shading information for each of the visible geometric primitives, storing the shading information of each of the visible geometric primitives in a portion of the allocated region smaller than the allocated region to create a buffer around the portion of the allocated region where the shading information is stored, and sending, to the client device, the texture atlas and a list identifying the visible geometric primitives, the texture atlas being configured for rendering images of the visible geometric primitives from different viewpoints.

Abstract: A method includes the steps of receiving training data comprising images of an object and associated camera poses from which the images are captured, training, based on the training data, a machine-learning model to take as input a given viewpoint and synthesize an image of a virtual representation of the object viewed from the given viewpoint, generating, for each of predetermined viewpoints surrounding the virtual representation of the object, a view-dependent image of the object as viewed from that viewpoint using the trained machine-learning model, receiving, from a client device, a desired viewpoint from which to view the virtual representation of the object, selecting one or more of the predetermined viewpoints based on the desired viewpoint, and sending, to the client device, the view-dependent images associated with the selected one or more viewpoints for rendering an output image of the virtual representation of the object viewed from the desired viewpoint.

Abstract: Disclosed herein includes a system, a method, and a device for rendering an image through adaptive reprojection. A first reprojection can be performed to generate a portion of a first image of a first view of a virtual space at a first rate. An amount of change from the first view to a second view of the virtual space can be determined. A portion of a second image of the second view of the virtual space can be determined to generate through a second reprojection. The second reprojection can be performed at a second rate to the portion of the second image according to the amount of change from the first view to the second view of the virtual space.

Abstract: Disclosed herein are related to a system and a method of remotely rendering an image. In one approach, a console device generates an image according to a gaze direction of a user of a head mounted display (HMD). In one aspect, the image includes a first area and a second area disposed along an axis, where the second area is located farther away from a foveated area of the image than the first area. In one aspect, the foveated area corresponds to the gaze direction of the user of the HMD. In one aspect, the console device compresses the image according to the axis, where the second area is compressed at a higher level than the first area. In one aspect, the compressed image is transmitted to the HMD. The HMD may decompress the compressed image according to the axis, and render the decompressed image.

Abstract: A method includes a computing system receiving a first-person image that captures an artificial-reality environment from a first-person perspective of a user of a head-mounted device, wherein the first-person image is rendered based on a pose of the head-mounted device. In response to a determination that a measured rotational movement of the pose exceeds a predetermined threshold, a dampened rotational movement is determined that is less than the measured rotational movement. A spectator perspective is determined wherein the spectator perspective and the first-person perspective are co-located in a three-dimensional space and a rotational orientation of the spectator perspective is determined based on the dampened rotational movement.

Abstract: Disclosed herein includes a system, a method, and a device for rendering an image through adaptive reprojection. A first reprojection can be performed to generate a portion of a first image of a first view of a virtual space at a first rate. An amount of change from the first view to a second view of the virtual space can be determined. A portion of a second image of the second view of the virtual space can be determined to generate through a second reprojection. The second reprojection can be performed at a second rate to the portion of the second image according to the amount of change from the first view to the second view of the virtual space.

Abstract: In one embodiment, a method includes the steps of generating, for a virtual object defined by a geometric representation, multiple viewpoints surrounding the virtual object, generating, for each of the multiple viewpoints, a simplified geometric representation of the virtual object based on the viewpoint, wherein the simplified geometric representation has a lower resolution than the geometric representation of the virtual object, receiving, from a client device, a desired viewpoint from which to view the virtual object, selecting one or more viewpoints from the multiple viewpoints based on the desired viewpoint, and sending, to the client device, rendering data including the simplified geometric representation and an associated view-dependent texture that are associated with each of the selected one or more viewpoints, the rendering data being configured for rendering an image of the virtual object from the desired viewpoint.

Abstract: A system can include a device and a graphics processing unit (GPU). The device can be configured to receive a first image from one or more cameras corresponding to a first view and a second image from the one or more cameras corresponding to a second view. The GPU can include a motion estimator and configured to receive the first image and the second image and be configured to receive the first image and the second image. The motion estimator can be configured to determine first disparity offsets for the first image and second disparity offsets for the second image. The device can be configured to generate, for rendering 3D image using the first image and the second image, a first depth buffer for the first image derived from the first disparity offsets and a second depth buffer for the second image derived from the second disparity offsets.

Abstract: A method includes a computing system receiving a first-person image that captures an artificial-reality environment from a first-person perspective of a user of a head-mounted device, wherein the first-person image is rendered based on a pose of the head-mounted device. Based on the pose of the head-mounted device, the computing system determines a spectator perspective wherein the spectator perspective and the first-person perspective are co-located in a three-dimensional space and the spectator perspective has a predetermined rotational orientation about a view axis. The computing system generates a spectator image that captures the first-person image from the spectator perspective, and causes the first-person image to be displayed by the head-mounted device and the spectator image to be displayed on a device separate from the head-mounted device.

Abstract: The disclosed computer-implemented method may include (1) receiving a first 2D frame depicting an evolving 3D scene and elements in the evolving 3D scene, (2) receiving a second 2D frame depicting the evolving 3D scene and the elements, (3) deriving 2D motion vectors from the first 2D frame and the second 2D frame that each include an estimated offset from coordinates of an element in the first 2D frame to coordinates of the element in the second 2D frame, (4) receiving depth information for the evolving 3D scene, (5) using the 2D motion vectors and the depth information to extrapolate a synthetic 2D frame, and (6) displaying the synthetic 2D frame to a user. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: Disclosed herein are related to a system and a method of remotely rendering an image. In one approach, a console device generates an image according to a gaze direction of a user of a head mounted display (HMD). In one aspect, the image includes a first area and a second area disposed along an axis, where the second area is located farther away from a foveated area of the image than the first area. In one aspect, the foveated area corresponds to the gaze direction of the user of the HMD. In one aspect, the console device compresses the image according to the axis, where the second area is compressed at a higher level than the first area. In one aspect, the compressed image is transmitted to the HMD. The HMD may decompress the compressed image according to the axis, and render the decompressed image.

Abstract: Disclosed herein includes a system, a method, and a device for rendering an image through adaptive reprojection. A first reprojection can be performed to generate a portion of a first image of a first view of a virtual space at a first rate. An amount of change from the first view to a second view of the virtual space can be determined. A portion of a second image of the second view of the virtual space can be determined to generate through a second reprojection. The second reprojection can be performed at a second rate to the portion of the second image according to the amount of change from the first view to the second view of the virtual space.