Abstract: Systems and methods for occluding virtual objects in an augmented reality environment are described. A method includes generating a distance map associated with a physical object, determining a location of a virtual object in an artificial-reality environment, determining, based at least in part upon the distance map and the location of the virtual object, a first portion of the physical object that is closer than a first portion of the virtual object from the viewpoint of the user, determining a second portion of the virtual object that is not obscured by the physical object from the viewpoint of the user, and rendering a display image. The display image comprises a first set of pixels corresponding to the first portion of the physical object in a first state, and the display image comprises a second set of pixels corresponding to the second portion of the virtual object in a second state.

Abstract: A display device includes a display panel having a first emission region and one or more second emission regions disposed adjacent to the first emission region. The display device includes a plurality of light emitters, arranged in the first emission region, corresponding to a first color gamut and a plurality of light emitters, arranged in the one or more second emission regions, corresponding to a second color gamut that is distinct from the first color gamut. A method for making a display device with a plurality of light emitters corresponding to a first color gamut in a first emission region and a plurality of light emitters corresponding to a second color gamut in a second emission region is also described.

Abstract: In one embodiment, a method includes by one or more computing devices, accessing an image including a hand of a user of a head-mounted display at a first time. The method includes generating, from at least the image, a virtual object representation of the hand, defined in a virtual environment that includes at least one other virtual object. The method includes rendering a first image of the virtual environment comprising a first portion of the hand of the user at a first frame rate, and determining a second viewpoint of the user at a second time. The method includes rendering a second image of the virtual environment comprising a second portion of the hand of the user at a second frame rate. The method includes providing, to a set of light emitters of the head-mounted display, instructions to display the second image.

Abstract: In one example, an apparatus for integrating sensing and display system includes a first semiconductor layer that includes an image sensor; a second semiconductor layer that includes a display; a third semiconductor layer that includes compute circuits configured to support an image sensing operation by the image sensor and a display operation by the display; and a semiconductor package that encloses the first, second, and third semiconductor layers, the semiconductor package further including a first opening to expose the image sensor and a second opening to expose the display. The first, second, and third semiconductor layers form a first stack structure along a first axis. The third semiconductor layer is sandwiched between the first semiconductor layer and the second semiconductor layer in the first stack structure.

Abstract: In an embodiment, a headset display device includes a central processor and multiple projector integrated circuits for eyes of a wearer of the headset display device. Each eye of the wear is associated with at least three projector integrated circuits. Each of the three projector integrated circuits is communicatively coupled to the central processor. Each projector integrated circuit includes a first integrated circuit including a light emitter array having monochrome light emitters of a single color, and a second integrated circuit coupled to the first integrated circuit. The second integrated circuit includes a graphics processor configured to generate transformed image data. The graphics processor is configured to provide the transformed image data to the first integrated circuit. The first integrated circuit is configured to output the transformed image data using the light emitter array.

Abstract: In one embodiment, a method includes accessing display timing information associated with a display, and determining, based on the display timing information, one or more rendering requirements for a plurality of portions of the display. For each portion of the display, the method may determine a current viewpoint of a user, render, based on the current viewpoint of the user, pixel values for that portion of the display in accordance with the associated rendering requirement, and send the rendered pixel values to be output by the display. The display timing information may include a waveform associated with an oscillation motion of a scanning assembly of the display, and/or a requested size for each portion of the plurality of portions of the display. The rendering requirements may include a timing interval for rendering and sending the pixel values for each of the plurality of portions of the display.

Abstract: In one embodiment, a computing system may access surfaces and texel data of an artificial reality scene. The surfaces may be generated based on a first viewing position of a viewer. The system may determine tiles on a display to test for a visibility of the surfaces from a second viewing position. The tiles may include first tiles that need more computational resources and second tiles that need less computational resources. The system may determine a tile order which interleaves the first and second tiles. The system may generate rays based on the tile order. The system may determine the visibility of the surfaces from the second viewing position based on the ray-surface intersections. The system may generate color values of a subframe based on the surface visibility and the texel data. The system may provide the color values to the display.

Abstract: In one embodiment, a method includes by one or more computing devices, accessing an image including a hand of a user of a head-mounted display at a first time. The method includes generating, from at least the image, a virtual object representation of the hand, defined in a virtual environment that includes at least one other virtual object. The method includes rendering a first image of the virtual environment comprising a first portion of the hand of the user at a first frame rate, and determining a second viewpoint of the user at a second time. The method includes rendering a second image of the virtual environment comprising a second portion of the hand of the user at a second frame rate. The method includes providing, to a set of light emitters of the head-mounted display, instructions to display the second image.

Abstract: In one embodiment, a computing system may access surfaces and texel data of an artificial reality scene. The surfaces may be generated based on a first viewing position of a viewer. The system may determine tiles on a display to test for a visibility of the surfaces from a second viewing position. The tiles may include first tiles that need more computational resources and second tiles that need less computational resources. The system may determine a tile order which interleaves the first and second tiles. The system may generate rays based on the tile order. The system may determine the visibility of the surfaces from the second viewing position based on the ray-surface intersections. The system may generate color values of a subframe based on the surface visibility and the texel data. The system may provide the color values to the display.

Abstract: In one embodiment, one or more computing devices access an image comprising at least a portion of a hand of a user of a head-mounted display and generate a planar representation of the hand and a height map associated with the planar representation. A first portion of the planar representation that is closer than a first portion of a virtual object to a viewpoint and a second portion of the planar representation that is farther than a second portion of the virtual object from the viewpoint is determined based on the height map and the virtual object. A display image is rendered from the viewpoint for display, the display image comprising a first set of pixels corresponding to the first portion of the planar representation and a second set of pixels corresponding to the second portion of the virtual object.

Abstract: In one embodiment, a computing system may access surfaces and texel data of an artificial reality scene. The surfaces may be generated based on a first viewing position of a viewer. The system may determine tiles on a display to test for a visibility of the surfaces from a second viewing position. The tiles may include first tiles that need more computational resources and second tiles that need less computational resources. The system may determine a tile order which interleaves the first and second tiles. The system may generate rays based on the tile order. The system may determine the visibility of the surfaces from the second viewing position based on the ray-surface intersections. The system may generate color values of a subframe based on the surface visibility and the texel data. The system may provide the color values to the display.

Abstract: In one embodiment, a method includes accessing one or more surfaces of an artificial reality scene. The one or more surfaces are generated based on one or more images of the artificial reality scene rendered at a first rate and based on a first viewing position. The method includes generating subframes at a second rate higher than the first rate. Each of the subframes is generated by determining a second viewing position, determining a visibility of the one or more surfaces of the artificial reality scene from the second viewing position, generating, based on the determined visibility of the one or more surfaces, color values of the subframe corresponding to output positions of a display, and providing the color values of the subframe for display.

Abstract: A display device includes a display panel having a first emission region and one or more second emission regions disposed adjacent to the first emission region. The display device includes a plurality of light emitters, arranged in the first emission region, corresponding to a first color gamut and a plurality of light emitters, arranged in the one or more second emission regions, corresponding to a second color gamut that is distinct from the first color gamut. A method for making a display device with a plurality of light emitters corresponding to a first color gamut in a first emission region and a plurality of light emitters corresponding to a second color gamut in a second emission region is also described.

Abstract: In one embodiment, a method includes accessing a first distortion definition for a first color component and a second distortion definition for a second color component; generating, for a pixel, a first ray for the first color component of the pixel and a second ray for the second component of the pixel; defining trajectories for the first and second rays using the first and second distortion definitions, respectively; casting, from a viewpoint in a 3D space, the first and second rays into the 3D space using the defined trajectories; computing, for the first and second rays, a first intersection location and a second intersection location with one or more objects in the 3D space; and determining a color value for the pixel based on the first and second intersection locations.

Abstract: In an embodiment, a headset display device includes a central processor and multiple projector integrated circuits each coupled to the central processor and configured to process image data. Each projector integrated circuit includes multiple first integrated circuits, each including a light emitter array. Each projector integrated circuit includes a second integrated circuit coupled to the multiple first integrated circuits. The second integrated circuit includes a graphics processor configured to generate transformed image data correcting for geometrical or brightness distortions and (2) is configured to provide the transformed image data to the multiple first integrated circuits for display.

Abstract: In one embodiment, one or more computing devices access an image comprising at least a portion of a hand of a user of a head-mounted display and generate a planar representation of the hand and a height map associated with the planar representation. A first portion of the planar representation that is closer than a first portion of a virtual object to a viewpoint and a second portion of the planar representation that is farther than a second portion of the virtual object from the viewpoint is determined based on the height map and the virtual object. A display image is rendered from the viewpoint for display, the display image comprising a first set of pixels corresponding to the first portion of the planar representation and a second set of pixels corresponding to the second portion of the virtual object.

Abstract: A system for content upsampling comprises a console and a head-mounted display (HMD). The console can select content for presentation and provide the content at a first frame rate. The HMD outputs fast calibration data comprising one or more intermediate estimated positions of a reference point on the HMD. The HMD estimates future positions of the HMD using the fast calibration data, and generates synthetic frames using frames from the content and the future positions of the HMD. The HMD then upsamples the content from the first frame rate to a second frame using the synthetic frames to generate augmented content, wherein the second frame rate is faster than the first frame rate. The HMD presents the augmented content at the second frame rate via an electronic display.

Abstract: In one embodiment, a computing system may access a first image that is generated at a first frame rate. The system may determine whether a change of a user viewpoint with respect to one or more display contents satisfies a threshold criterion. The system may select an operation mode from a first operation mode and a second operation mode based on the determination whether the change of the user viewpoint satisfies the threshold criterion. The system may generate a number of second images at a second frame rate higher than the first frame rate. When the selected operation mode is the first operation mode, the second images may be generated using a resampling process. When the selected operation mode is the second operation mode, the second images may be generated by transforming one or more previously generated second images that are generated based on the first image.

Abstract: In an embodiment, a method includes accessing a first rendered frame generated based on a first viewing direction of a user. The first rendered frame may be generated at a first frame rate. The method includes generating, based on the first rendered frame, one or more sub-frames at a second frame rate that is higher than the first frame rate. A first sub-frame of the one or more sub-frames is generated by determining a second viewing direction of the user based on sensor data and applying one or more transformations to the first frame based on the second viewing direction. The method includes outputting the one or more sub-frames for display at the second frame rate.

Abstract: In one embodiment, a computing system may determine an orientation in a three-dimensional (3D) space and generate a plurality of coordinates in the 3D space based on the determined orientation. The system may access pre-determined ray trajectory definitions associated with the plurality of coordinates. The system may determine visibility information of one or more objects defined within the 3D space by projecting rays through the plurality of coordinates, wherein trajectories of the rays from the plurality of coordinates are determined based on the pre-determined ray trajectory definitions. The system may then generate an image of the one or more objects based on the determined visibility information of the one or more objects.