Abstract: In one embodiment, a method includes identifying, in each image of a stereoscopic pair of images of a scene at a particular time, every pixel as either a static pixel corresponding to a portion of a scene that does not have local motion at that time or a dynamic pixel corresponding to a portion of a scene that has local motion at that time. For each static pixel, the method includes comparing each of a plurality of depth calculations for the pixel, and when the depth calculations differ by at least a threshold amount, then re-labeling that pixel as a dynamic pixel. For each dynamic pixel, the method includes comparing a geometric 3D calculation for the pixel with a temporal 3D calculation for that pixel, and when the geometric 3D calculation and the temporal 3D calculation are within a threshold amount, then re-labeling the pixel as a static pixel.

Abstract: In one embodiment, a method includes accessing an audio input that includes a mixture of vocal sounds and non-vocal sounds and separating, by a trained audio source separation model, the audio input into a first audio output representing the vocal sounds and a second audio output representing the non-vocal sounds. The method further includes determining, by one or more trained avatar animation models and by separately encoding the first audio output representing the vocal sounds and the second audio output representing the non-vocal sounds, an avatar animation temporally corresponding to the audio input; and rendering, in real time and temporally coincident with the audio input, the determined avatar animation.

Abstract: A method includes receiving, from a camera, one or more frames of image data of a scene comprising a background and one or more three-dimensional objects, wherein each frame comprises a raster of pixels of image data; detecting layer information of the scene, wherein the layer information is associated with a depth-based distribution of the pixels in the one or more frames; and determining a multi-layer model for the scene, the multi-layer model comprising a plurality of discrete layers comprising first and second discrete layers, wherein each discrete layer is associated with a unique depth value relative to the camera. The method further includes mapping the pixels to the layers of the plurality of discrete layers; rendering the pixels as a first image of the scene as viewed from a first perspective; and rendering the pixels as a second image of the scene as viewed from a second perspective.

Abstract: In one embodiment, a method includes identifying, in each image of a stereoscopic pair of images of a scene at a particular time, every pixel as either a static pixel corresponding to a portion of a scene that does not have local motion at that time or a dynamic pixel corresponding to a portion of a scene that has local motion at that time. For each static pixel, the method includes comparing each of a plurality of depth calculations for the pixel, and when the depth calculations differ by at least a threshold amount, then re-labeling that pixel as a dynamic pixel. For each dynamic pixel, the method includes comparing a geometric 3D calculation for the pixel with a temporal 3D calculation for that pixel, and when the geometric 3D calculation and the temporal 3D calculation are within a threshold amount, then re-labeling the pixel as a static pixel.

Abstract: A method includes receiving, from a camera, one or more frames of image data of a scene comprising a background and one or more three-dimensional objects, wherein each frame comprises a raster of pixels of image data; detecting layer information of the scene, wherein the layer information is associated with a depth-based distribution of the pixels in the one or more frames; and determining a multi-layer model for the scene, the multi-layer model comprising a plurality of discrete layers comprising first and second discrete layers, wherein each discrete layer is associated with a unique depth value relative to the camera. The method further includes mapping the pixels to the layers of the plurality of discrete layers; rendering the pixels as a first image of the scene as viewed from a first perspective; and rendering the pixels as a second image of the scene as viewed from a second perspective.

Abstract: A method for hand pose tracking in a virtual reality (VR) environment by a wearable device is provided. The method includes calibrating a plurality of sensors configured to detect an orientation of fingers with respect to a hand of a user, identifying sensor data obtained by the plurality of sensors, and tracking a pose of the hand based on the sensor data.

Abstract: An apparatus that modifies display settings of image data is provided. The apparatus stores one or more spectral mappings and receives display data for an image. The received display data includes a plurality of colors and each of the colors includes a plurality of attributes. A spectral mapping, of the one or more spectral mappings, is identified based on a detected user visual limitation. Each of the spectral mappings maps a set of spectral colors to an adjusted set of spectral colors and provides attribute adjustments for the set of spectral colors of image data so an image can be displayed in a manner that mitigates the user's visual limitation. The mapped and adjusted plurality of colors of the image data that mitigate the visual limitation when displayed on a display may mitigate, for example, viewer color blindness, eye strain, biometric measurements and/or other visual limitations.

Abstract: A system and method enable an electronic device to determine six-degree of freedom (6DOF) of a hand-held controller. The device includes one or more tracking devices and a processor. The one or more tracking devices are configured to indicate a first three-degree of freedom (3DOF) orientation information regarding an orientation tracker device, and indicate a second 3DOF orientation information regarding the hand-held controller. The processor is configured to determine a six-degree of freedom (6DOF) orientation information of the hand-held controller as a function of the first 3DOF orientation information and the second 3DOF orientation information, wherein the orientation tracker device is different than the hand-held controller.

Abstract: An electronic device, method, and computer readable medium for synchronizing frame-based and event-based tracking points are provided. The electronic device includes a frame-based camera, an event-based camera, and a processor. The processor is coupled to the frame-based camera and the event-based camera. The processor receives frame-based data captured by the frame-based camera and event-based data captured by the event-based camera; identifies a frame-based tracking point from the frame-based data; identifies an event-based tracking point from the event-based data corresponding to the frame-based tracking point; and synchronize the frame-based camera and the event-based camera based on the identified frame-based tracking point and the identified event-based tracking point.

Abstract: A head mounted augmented reality apparatus is provided having a frame, a band structure removably coupled to the frame, and a first cartridge removably coupled to a portion of the frame. First cartridge can hold the display such that when the first cartridge is coupled to the portion of the frame and holding the display, the display is oriented to face away from the frame at a selected angle. They cover structure is provided that removably attaches to a portion of either the frame or the first cartridge. A visor removably attaches to the cover structure and is disposed such that a surface of the visor reflects an image displayed on the display back toward the frame.

Abstract: Systems, methods, and computer readable media are provided for automatically personalizing virtual reality content ratings using biometric data on a head-mountable display (HMD) device. A server includes a transceiver configured to transmit and receive data to and from a first electronic device, a memory, and at least one processor operably connected to the memory and the transceiver. The processor(s) is configured to receive application data and biometric data, wherein the biometric data corresponds to the application data from the first electronic device, generate a rating for an application related to the application data, wherein the generated rating is based on the biometric data, and store the generated rating to a database corresponding to the application in an online store.

Abstract: A method for hand pose tracking in a virtual reality (VR) environment by a wearable device is provided. The method includes calibrating a plurality of sensors configured to detect an orientation of fingers with respect to a hand of a user, identifying sensor data obtained by the plurality of sensors, and tracking a pose of the hand based on the sensor data.

Abstract: A head mounted augmented reality apparatus is provided having a frame, a band structure removably coupled to the frame, and a first cartridge removably coupled to a portion of the frame. First cartridge can hold the display such that when the first cartridge is coupled to the portion of the frame and holding the display, the display is oriented to face away from the frame at a selected angle. They cover structure is provided that removably attaches to a portion of either the frame or the first cartridge. A visor removably attaches to the cover structure and is disposed such that a surface of the visor reflects an image displayed on the display back toward the frame.

Abstract: A system and method enable an electronic device to determine six-degree of freedom (6DOF) of a hand-held controller. The device includes one or more tracking devices and a processor. The one or more tracking devices are configured to indicate a first three-degree of freedom (3DOF) orientation information regarding an orientation tracker device, and indicate a second 3DOF orientation information regarding the hand-held controller. The processor is configured to determine a six-degree of freedom (6DOF) orientation information of the hand-held controller as a function of the first 3DOF orientation information and the second 3DOF orientation information, wherein the orientation tracker device is different than the hand-held controller.

Abstract: An electronic device, method, and computer readable medium for synchronizing frame-based and event-based tracking points are provided. The electronic device includes a frame-based camera, an event-based camera, and a processor. The processor is coupled to the frame-based camera and the event-based camera. The processor receives frame-based data captured by the frame-based camera and event-based data captured by the event-based camera; identifies a frame-based tracking point from the frame-based data; identifies an event-based tracking point from the event-based data corresponding to the frame-based tracking point; and synchronize the frame-based camera and the event-based camera based on the identified frame-based tracking point and the identified event-based tracking point.

Abstract: Systems, methods, and computer readable media are provided for automatically personalizing virtual reality content ratings using biometric data on a head-mountable display (HMD) device. A server includes a transceiver configured to transmit and receive data to and from a first electronic device, a memory, and at least one processor operably connected to the memory and the transceiver. The processor(s) is configured to receive application data and biometric data, wherein the biometric data corresponds to the application data from the first electronic device, generate a rating for an application related to the application data, wherein the generated rating is based on the biometric data, and store the generated rating to a database corresponding to the application in an online store.

Abstract: An apparatus that modifies display settings of image data is provided. The apparatus stores one or more spectral mappings and receives display data for an image. The received display data includes a plurality of colors and each of the colors includes a plurality of attributes. A spectral mapping, of the one or more spectral mappings, is identified based on a detected user visual limitation. Each of the spectral mappings maps a set of spectral colors to an adjusted set of spectral colors and provides attribute adjustments for the set of spectral colors of image data so an image can be displayed in a manner that mitigates the user's visual limitation. The mapped and adjusted plurality of colors of the image data that mitigate the visual limitation when displayed on a display may mitigate, for example, viewer color blindness, eye strain, biometric measurements and/or other visual limitations.

Abstract: The present invention relates to a system and method to allow multiple users to collaborate on tasks and interact in a shared space session within a network in real-time; using a media application to manage media-layers. Each media-layer serves as a container for multimedia programs or plug-ins. The invention allows which media-layer to display via organization metaphors and filtering criteria. When multiple users are logged into the same shared space, each user can invoke and observe modifications to media-layers with the browser based or client based application. All events are synchronized among all users in that shared space, where the system is a communication conduit. The media-layers in the shared space maintains spatial and temporal correlation by a media application stage manager tool and described as a collection file descriptor such as an XML file. The ability to invoke events that affect media-layers can be supported in a synched or non synched mode on demand.