Abstract: A system that efficiently estimates an object's orientation using magnetic, angular rate, and gravity sensors. The object may be for example a virtual reality headset or a user in a virtual reality environment. Magnetic and gravity data are used to correct errors that accumulate from integrating angular velocity. Unlike systems that use Kalman filter approaches, embodiments of the system apply a simple, highly efficient technique to generate magnetic and gravity error vectors; these error vectors are added directly to the angular velocity prior to integration. Error calculations are performed in the sensor reference frame rather than in the Earth reference frame. Magnetic error correction uses only the horizontal component of the magnetic field, which is efficiently calculated by subtracting off the projection of the magnetic field onto the measured gravity vector. Sensors and processors for calculating orientation may be integrated into a low-latency virtual reality display system.

Abstract: A modular virtual reality headset that may be operated in multiple modes. Embodiments enable a mount with modular receivers having electronic and mechanical interfaces that accept swappable modules. Swappable modules may include swappable display modules, swappable audio modules, and swappable sensor modules. Embodiments enable multiple modes of operation, including for example a virtual mode to display virtual reality environments, a real mode to display images of the real environment surrounding the user, and an augmented reality mode that overlays real scenes with other information. Embodiments may also provide multiple modes of operation for audio.

Abstract: A display system for a head mounted device that illuminates the peripheral regions of the user's field of view to enhance an immersive experience. The system may use peripheral light emitters to the left and right of one or more central displays. Peripheral light emitters may provide lower resolution images, or only with vertical resolution, corresponding to the user's lower resolution vision in these peripheral regions. Reflective surfaces and lenses may be used to direct peripheral light into desired shapes and patterns. Rendering of peripheral light colors and intensities at each peripheral pixel may use approximations for improved performance since users may not be sensitive to precise color values in the peripheral regions.

Abstract: A virtual reality display system that generates display images in two phases: the first phase renders images based on a predicted pose at the time the display will be updated; the second phase re-predicts the pose using recent sensor data, and corrects the images based on changes since the initial prediction. The second phase may be delayed so that it occurs just in time for a display update cycle, to ensure that sensor data is as accurate as possible for the revised pose prediction. Pose prediction may extrapolate sensor data by integrating differential equations of motion. It may incorporate biomechanical models of the user, which may be learned by prompting the user to perform specific movements. Pose prediction may take into account a user's tendency to look towards regions of interest. Multiple parallel pose predictions may be made to reflect uncertainty in the user's movement.

Abstract: A virtual reality display system that renders images at different resolutions in different parts of a display. Reduces rendering latency by rendering at a lower resolution in selected regions, for example on the sides of a display where human vision has lower resolution than in the center. Pixels in low resolution regions are combined into grid elements, and rendering may generate grid element values rather than individual pixel values. Rendering may use ray casting, rasterization, or both. Variable resolution rendering may be combined with variable level of detail geometry models to further reduce rendering time. Selected objects may be designed as high resolution objects that are rendered at a high resolution even in low resolution display regions.

Abstract: A system that efficiently estimates an object's orientation using magnetic, angular rate, and gravity sensors. The object may be for example a virtual reality headset or a user in a virtual reality environment. Magnetic and gravity data are used to correct errors that accumulate from integrating angular velocity. Unlike systems that use Kalman filter approaches, embodiments of the system apply a simple, highly efficient technique to generate magnetic and gravity error vectors; these error vectors are added directly to the angular velocity prior to integration. Error calculations are performed in the sensor reference frame rather than in the Earth reference frame. Magnetic error correction uses only the horizontal component of the magnetic field, which is efficiently calculated by subtracting off the projection of the magnetic field onto the measured gravity vector. Sensors and processors for calculating orientation may be integrated into a low-latency virtual reality display system.

Abstract: A virtual reality system that uses gestures to obtain commands from a user. Embodiments may use sensors mounted on a virtual reality headset to detect head movements, and may recognize selected head motions as gestures associated with commands. Commands associated with gestures may modify the user's virtual reality experience, for example by selecting or modifying a virtual world or by altering the user's viewpoint within the virtual world. Embodiments may define specific gestures to place the system into command mode or user input mode, for example to temporarily disable normal head tracking within the virtual environment. Embodiments may also recognize gestures of other body parts, such as wrist movements measured by a smart watch.

Abstract: Method, computer storage device, and system that establish communication with at least first and second sensors, receive information from the sensors, and determine whether the information from the first sensor satisfies a first alert threshold. It is also determined whether the information from the second sensor satisfies a second alert threshold. Responsive to a determination that the information from the first sensor satisfies the first alert threshold and that the information from the second sensor does not satisfy the second alert threshold, an alert signal is not generated. On the other hand, responsive to a determination that the information from the first sensor satisfies the first alert threshold and that the information from the second sensor satisfies the second alert threshold, the alert signal is generated for output thereof to a display device.

Abstract: Method, computer storage device, and system that establish communication with at least first and second sensors, receive information from the sensors, and determine whether the information from the first sensor satisfies a first alert threshold. It is also determined whether the information from the second sensor satisfies a second alert threshold. Responsive to a determination that the information from the first sensor satisfies the first alert threshold and that the information from the second sensor does not satisfy the second alert threshold, an alert signal is not generated. On the other hand, responsive to a determination that the information from the first sensor satisfies the first alert threshold and that the information from the second sensor satisfies the second alert threshold, the alert signal is generated for output thereof to a display device.

Abstract: A virtual reality headset that tracks motion of a user relative to a moving vehicle in which the user is located. The system updates the headset's display based on relative motion, rather than on absolute motion due to vehicle movement. Data from sensors on the headset, the vehicle, or both is analyzed to determine the pose (orientation, position, or both) of the headset relative to the vehicle. Sensors may include headset-mounted cameras that observe the vehicle's interior from the user's viewpoint. Analysis of camera images may use optical flow or it may compare images to reference images captured from known poses. The system may also combine data from headset sensors and vehicle sensors to calculate the relative pose. Headset-mounted cameras may function both as relative pose sensors for virtual reality, and as image capture devices for display of the real environment on the headset.

Abstract: A display system for a head mounted device that provides a wide field-of-view image to the user. The system may use a pair of displays angled relative to one another, and a lens or lenses between the displays and the user's eyes to generate a wide field-of-view image. Lenses may be for example gradient index lenses, Fresnel lenses, or holographic optical elements in order to provide significant and complex bending of light across the field of view with relatively thin lenses. The system may provide lower resolution images at the periphery of the user's field of view, using for example light emitting elements at the periphery that are directed by the lens or lenses towards the outside edges of the field of view.

Abstract: A virtual reality virtual theater system that generates or otherwise displays a virtual theater, for example in which to view videos, such as movies or television. Videos may be for example 2D or 3D movies or television programs. Embodiments create a virtual theater environment with elements that provide a theater-like viewing experience to the user. Virtual theaters may be for example indoor movie theaters, drive-in movie theaters, or home movie theaters. Virtual theater elements may include for example theater fixtures, décor, and audience members; these elements may be selectable or customizable by the user. One or more embodiments also render sound by placing virtual speakers in a virtual theater and projecting sounds from these virtual speakers onto virtual microphones corresponding to the position and orientation of the user's ears. Embodiments may also employ minimal latency processing to improve the theater experience.

Abstract: A low-latency virtual reality display system that provides rapid updates to virtual reality displays in response to a user's movements. Display images generated by rendering a 3D virtual world are modified using an image warping approximation, which allows updates to be calculated and displayed quickly. Image warping is performed using various rerendering approximations that use simplified 3D models and simplified 3D to 2D projections. An example of such a rerendering approximation is a shifting of all pixels in the display by a pixel translation vector that is calculated based on the user's movements; pixel translation may be done very quickly, possibly using hardware acceleration, and may be sufficiently realistic particularly for small changes in a user's position and orientation. Additional features include techniques to fill holes in displays generated by image warping, synchronizing full rendering with approximate rerendering, and using prediction of a user's future movements to reduce apparent latency.

Abstract: Methods and systems for producing ambient light effects based on video content are provided. A method of producing an ambient light effect includes providing a receiving device including a processor configured to parse incoming video content and receiving the video content including a plurality of scenes at the receiving device. The method further includes parsing the incoming video content and detecting at least one scene in the video content for association with at least one ambient light effect. The method also includes generating a command specifying at least one ambient light effect to be associated with at least one scene and sending the command from the receiving device to at least one lighting device in communication with the receiving device to generate at least one ambient light effect associated with at least one scene when the scene is displayed to a user.

Abstract: Method, computer storage device, and system that establish communication with at least first and second sensors, receive information from the sensors, and determine whether the information from the first sensor satisfies a first alert threshold. It is also determined whether the information from the second sensor satisfies a second alert threshold. Responsive to a determination that the information from the first sensor satisfies the first alert threshold and that the information from the second sensor does not satisfy the second alert threshold, an alert signal is not generated. On the other hand, responsive to a determination that the information from the first sensor satisfies the first alert threshold and that the information from the second sensor satisfies the second alert threshold, the alert signal is generated for output thereof to a display device.

Abstract: Methods and systems for producing ambient light effects based on video content are provided. A method of producing an ambient light effect includes providing a receiving device including a processor configured to parse incoming video content and receiving the video content including a plurality of scenes at the receiving device. The method further includes parsing the incoming video content and detecting at least one scene in the video content for association with at least one ambient light effect. The method also includes generating a command specifying at least one ambient light effect to be associated with at least one scene and sending the command from the receiving device to at least one lighting device in communication with the receiving device to generate at least one ambient light effect associated with at least one scene when the scene is displayed to a user.

Abstract: A system for controlling ambient light effects comprises a receiving device including at least one processor programmed to parse incoming video content to detect at least one portion of the incoming video content and determine at least one ambient light effect to be associated with the portion of the video content. The system further comprises a home automation controller in communication with the processor and at least one lighting device in communication with the receiving device and the home automation controller. The processor of the receiving device is programmed to send a command to the home automation controller specifying the ambient light effect determined by the processor to be associated with the portion of the video content. The lighting device generates the ambient light effect specified in the command when the portion of the video content is displayed to a user.

Abstract: An audio video display device (AVDD) system includes a display, a processor controlling the display, and a computer readable storage medium accessible to the processor and programmed with instructions. The instructions cause the processor to establish communication with at least one sensor. The instructions then cause the processor to receive information from the sensor conforming to an application programming interface (API) provided by a manufacturer of the AVDD to an entity affiliated with the sensor, or sent from the AVDD to the sensor. Thereafter, the instructions cause the processor to present the information from the sensor on the display in accordance with the API.