Abstract: A method to identify positions of fingers of a hand is described. The method includes capturing images of a first hand using a plurality of cameras that are part of a wearable device. The wearable device is attached to a wrist of a second hand and the plurality of cameras of the wearable device is disposed around the wearable device. The method includes repeating capturing of additional images of the first hand, the images and the additional images captured to produce a stream of captured image data during a session of presenting the virtual environment in a head mounted display (HMD). The method includes sending the stream of captured image data to a computing device that is interfaced with the HMD. The computing device is configured to process the captured image data to identify changes in positions of the fingers of the first hand.

Abstract: A method for implementing a graphics pipeline. The method includes determining a plurality of light sources affecting a virtual scene. Geometries of objects of an image of the scene is projected onto a plurality of pixels of a display from a first point-of-view. The pixels are partitioned into a plurality of tiles. A foveal region of highest resolution is defined for the image as displayed, wherein a first subset of pixels is assigned to the foveal region, and wherein a second subset of pixels is assigned to a peripheral region that is outside of the foveal region. A first set of light sources is determined from the plurality of light sources that affect one or more objects displayed in a first tile that is in the peripheral region. At least two light sources from the first set is clustered into a first aggregated light source affecting the first tile when rendering the image in pixels of the first tile.

Abstract: A method for implementing a graphics pipeline. The method includes building a first shadow map of high resolution, and building a second shadow map based on the first shadow map of lower resolution. The method includes determining a light source affecting a virtual scene, and projecting geometries of objects of an image of the virtual scene onto a plurality of pixels of a display from a first point-of-view. The method includes determining a foveal region when rendering the image, wherein the foveal region corresponds to where an attention of a user is directed. The method includes determining a first set of geometries is drawn to a first pixel, determining the first set of geometries is in shadow based on the light source, and determining the first set of geometries is outside of the foveal region. The method includes rendering the first set of geometries using the second shadow map.

Abstract: Methods, systems, and computer programs are presented for rendering images on a head mounted display (HMD). One method includes operations for tracking, with one or more first cameras inside the HMD, the gaze of a user and for tracking motion of the HMD. The motion of the HMD is tracked by analyzing images of the HMD taken with a second camera that is not in the HMD. Further, the method includes an operation for predicting the motion of the gaze of the user based on the gaze and the motion of the HMD. Rendering policies for a plurality of regions, defined on a view rendered by the HMD, are determined based on the predicted motion of the gaze. The images are rendered on the view based on the rendering policies.

Abstract: A head mounted display is provided. The head mounted display includes an optics block having a display screen. The optics block has a forward-facing region, a lower-facing region, a top-facing region, an inside-facing region, and side regions that extend from the forward-facing region toward the inside-facing region. The display screen is viewable from the inside-facing region. A band adjustment unit is located opposite the optics block. A headband has a front portion and an adjustable portion. The adjustable portion of the headband is connected to the band adjustment unit to contract a size of the headband or expand the size of the headband. The front portion of the headband has a connection to the top-facing region of the optics block that is proximate to the inside-facing region. A pad is disposed inside of the front portion of the headband. The pad provides for at least partial support of the optics block when the head mounted display is worn by a user.

Abstract: Methods and systems are provided for using temporal supersampling to increase a displayed resolution associated with peripheral region of a foveated rendering view. A method for enabling reconstitution of higher resolution pixels from a low resolution sampling region for fragment data is provided. The method includes an operation for receiving a fragment from a rasterizer of a GPU and for applying temporal supersampling to the fragment with the low resolution sampling region over a plurality of prior frames to obtain a plurality of color values. The method further includes an operation for reconstituting a plurality of high resolution pixels in a buffer that is based on the plurality of color values obtained via the temporal supersampling. Moreover, the method includes an operation for sending the plurality of high resolution pixels for display.

Abstract: Systems and methods for facilitating secret communication between players are described. One of the methods includes receiving image data of a first player to identify one or more gestures and receiving input data indicating a response of a second player to the one or more gestures. The method further includes training a model using the image data and the input data to generate an inference of communication between the first user and the second user. The method also includes generating additional inferences of communication between the first user and the second user. The method includes generating a recommendation to the second user based on the inference of communication and the additional inferences of communication.

Abstract: Gaze tracking data representing a user's gaze is analyzed to determine one or more regions of interest. One or more gaze tracking error parameters are determined from the gaze tracking data including a rate of rotation of a user's eye with respect to one or more axes. An error in a fixation of the user's eye on a particular region of interest is determined. Adjusted foveation data is determined representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters. The one or more transmitted images are selectively compressed so that fewer bits are needed to transmit data for portions of an image outside the one or more regions interest than for portions of the image within the one or more regions of interest.

Abstract: A method for implementing a graphics pipeline. The method includes generating a plurality of bones for an object defining positioning relationships between the bones. The method includes determining a subsystem of bones from the plurality of bones for the object. The method includes determining a foveal region when rendering an image of the virtual scene including the object, wherein the foveal region corresponds to where an attention of a user is directed. The method includes determining that at least one portion of the object is located in a peripheral region for the image. The method includes animating the object by applying a transformation to the subsystem of bones.

Abstract: Methods and systems are provided for using temporal supersampling to increase a displayed resolution associated with peripheral region of a foveated rendering view. A method for enabling reconstitution of higher resolution pixels from a low resolution sampling region for fragment data is provided. The method includes an operation for receiving a fragment from a rasterizer of a GPU and for applying temporal supersampling to the fragment with the low resolution sampling region over a plurality of prior frames to obtain a plurality of color values. The method further includes an operation for reconstituting a plurality of high resolution pixels in a buffer that is based on the plurality of color values obtained via the temporal supersampling. Moreover, the method includes an operation for sending the plurality of high resolution pixels for display.

Abstract: A head mounted display is provided. The head mounted display includes an optics block having a display screen. The optics block has a forward-facing region, a lower-facing region, a top-facing region, an inside-facing region, and side regions that extend from the forward-facing region toward the inside-facing region. The display screen is viewable from the inside-facing region. A band adjustment unit is located opposite the optics block. A headband has a front portion and an adjustable portion. The adjustable portion of the headband is connected to the band adjustment unit to contract a size of the headband or expand the size of the headband. The front portion of the headband has a connection to the top-facing region of the optics block that is proximate to the inside-facing region. A pad is disposed inside of the front portion of the headband. The pad provides at least partial support of the optics block.

Abstract: Methods and systems for filtering content include identifying content for presenting in a rendering space defined by one or more mechanisms of a head mounted display (HMD), wherein the content is interactive, streaming content. A rating score for the content is identified. The content is dynamically adjusted to selectively filter a portion of the content based on a user profile of the user, rating score of the content, and ongoing interactions detected at the content. The adjusted content is formatted for rendering on the HMD. The formatted content is automatically transmitted to the HMD for rendering, in response to the request.

Abstract: In one implementation, a portable device is provided, including: a sensor configured to generate sensor data for determining and tracking a position and orientation of the portable device during an interactive session of a virtual reality program presented on a main display, the interactive session being defined for interactivity between a user and the virtual reality program; a communications module configured to send the sensor data to a computing device, the communications module being further configured to receive from the computing device an ancillary video stream of the interactive session that is generated based on a state of the virtual reality program and the tracked position and orientation of the portable device.

Abstract: Methods and systems are provided for enabling foveal adaption of temporal anti-aliasing within a foveated rendering presentation. A method for rendering a multi-resolution scene with anti-aliasing within a head mounted display (HMD) is provided. The method includes an operation for rendering a scene that includes a series of video frames. The method also includes operations for applying anti-aliasing to a first region of the scene using a first jitter offset and applying anti-aliasing to a second region of the scene using a second jitter offset. Further, the method provides for generating a foveated scene that includes the first region and the second region and for sending the foveated scene to be displayed on a display associated with the HMD, wherein the first region is associated with a higher resolution than the second scene and the first jitter offset is smaller than the second jitter offset, according to certain embodiments.

Abstract: Methods and systems are provided for using temporal supersampling to increase a displayed resolution associated with peripheral region of a foveated rendering view. A method for enabling reconstitution of higher resolution pixels from a low resolution sampling region for fragment data is provided. The method includes an operation for receiving a fragment from a rasterizer of a GPU and for applying temporal supersampling to the fragment with the low resolution sampling region over a plurality of prior frames to obtain a plurality of color values. The method further includes an operation for reconstituting a plurality of high resolution pixels in a buffer that is based on the plurality of color values obtained via the temporal supersampling. Moreover, the method includes an operation for sending the plurality of high resolution pixels for display.

Abstract: Gaze tracking data may representing a user's gaze with respect to one or more images transmitted to a user are analyzed to determine one or more regions of interest. The one or more transmitted images are selectively compressed so that fewer bits are needed to transmit data for portions of an image outside the one or more regions interest than for portions of the image within the one or more regions of interest.

Abstract: A robot, including: a controller that communicates with a computing device, wherein the computing device executes a video game and renders a primary video feed of the video game to a display device, the primary video feed providing a first view into a virtual space that is defined by the executing video game; a camera that captures images of a local environment in which the robot is disposed; a projector; wherein the controller processes the images of the local environment to identify a projection surface in the local environment; wherein the computing device generates a secondary video feed of the video game and transmits the secondary video feed of the video game to the robot, the secondary video feed providing a second view into the virtual space; wherein the controller of the robot activates the projector to project the secondary video feed onto the projection surface in the local environment.

Abstract: Methods and systems for filtering content include identifying content for presenting in a rendering space defined by one or more mechanisms of a head mounted display (HMD), wherein the content is interactive, streaming content. A rating score for the content is identified. The content is dynamically adjusted to selectively filter a portion of the content based on a user profile of the user, rating score of the content, and ongoing interactions detected at the content. The adjusted content is formatted for rendering on the HMD. The formatted content is automatically transmitted to the HMD for rendering, in response to the request.

Abstract: Gaze tracking data may be analyzed to determine the onset and duration of a vision interrupting event, such as a blink or saccade. Presentation of images to a viewer may then be suspended during the vision interrupting event and resumed in sufficient time to ensure that the viewer sees the image at the time the vision interrupting event has concluded.

Abstract: A method for implementing a graphics pipeline. The method includes building a first shadow map of high resolution, and building a second shadow map based on the first shadow map of lower resolution. The method includes determining a light source affecting a virtual scene, and projecting geometries of objects of an image of the virtual scene onto a plurality of pixels of a display from a first point-of-view. The method includes determining a foveal region when rendering the image, wherein the foveal region corresponds to where an attention of a user is directed. The method includes determining a first set of geometries is drawn to a first pixel, determining the first set of geometries is in shadow based on the light source, and determining the first set of geometries is outside of the foveal region. The method includes rendering the first set of geometries using the second shadow map.