Abstract: A first device coupled with a first display and an image sensor receives output data from a second device having a second display different from the first display. The output data represents content displayable by the second device on the second display. The first device determines, using the image sensor, a position of the second display relative to the first device and causes the first display to display content based on the output data received from the second device and the determined position of the second display relative to the first device.

Abstract: In an exemplary process for interacting with user interface objects using an eye gaze, an affordance associated with a first object is displayed. A gaze direction or a gaze depth is determined. While the gaze direction or the gaze depth is determined to correspond to a gaze at the affordance, a first input representing user instruction to take action on the affordance is received, and the affordance is selected responsive to receiving the first input.

Abstract: A device implementing a system for managing multi-modal rendering of application content includes at least one processor configured to receive content, provided by an application running on a device, for displaying in a three-dimensional display mode. The at least one processor is further configured to determine that the content corresponds to two-dimensional content. The at least one processor is further configured to identify a portion of the two-dimensional content for enhancement by a three-dimensional render. The at least one processor is further configured to enhance, in response to the determining, the portion of the two-dimensional content by the three-dimensional renderer. The at least one processor is further configured to provide for display of the enhanced portion of the two-dimensional content on a display of the device in the three-dimensional display mode.

Abstract: A mixed reality system that includes a device and a base station that communicate via a wireless connection The device may include sensors that collect information about the user's environment and about the user. The information collected by the sensors may be transmitted to the base station via the wireless connection. The base station renders frames or slices based at least in part on the sensor information received from the device, encodes the frames or slices, and transmits the compressed frames or slices to the device for decoding and display. The base station may provide more computing power than conventional stand-alone systems, and the wireless connection does not tether the device to the base station as in conventional tethered systems. The system may implement methods and apparatus to maintain a target frame rate through the wireless link and to minimize latency in frame rendering, transmittal, and display.

Abstract: In an exemplary process for interacting with user interface objects using an eye gaze, an affordance associated with a first object is displayed. A gaze direction or a gaze depth is determined. While the gaze direction or the gaze depth is determined to correspond to a gaze at the affordance, a first input representing user instruction to take action on the affordance is received, and the affordance is selected responsive to receiving the first input.

Abstract: A mixed reality system that includes a device and a base station that communicate via a wireless connection The device may include sensors that collect information about the user's environment and about the user. The information collected by the sensors may be transmitted to the base station via the wireless connection. The base station renders frames or slices based at least in part on the sensor information received from the device, encodes the frames or slices, and transmits the compressed frames or slices to the device for decoding and display. The base station may provide more computing power than conventional stand-alone systems, and the wireless connection does not tether the device to the base station as in conventional tethered systems. The system may implement methods and apparatus to maintain a target frame rate through the wireless link and to minimize latency in frame rendering, transmittal, and display.

Abstract: An apparatus includes a transceiver and a controller in communication with the transceiver. The controller is configured to determine a target lighting condition in a room relative to a current lighting condition in the room. The controller is also configured to generate a control signal with instructions to adjust an ambient lighting peripheral in the room based on the determined target lighting condition. The control signal is provided to the transceiver for transmission.

Abstract: Implementations use a first device (e.g., an HMD) to provide a CGR environment that augments the input and output capabilities of a second device, e.g., a laptop, smart speaker, etc. In some implementations, the first device communicates with a second device in its proximate physical environment to exchange input or output data. For example, an HMD may capture an image of a physical environment that includes a laptop. The HMD may detect the laptop, send a request the laptop's content, receive content from the laptop (e.g., the content that the laptop is currently displaying and additional content), identify the location of the laptop, and display a virtual object with the received content in the CGR environment on or near the laptop. The size, shape, orientation, or position of the virtual object (e.g., a virtual monitor or monitor extension) may also be configured to provide a better user experience.

Abstract: A device implementing a system for managing multi-modal rendering of application content includes at least one processor configured to receive content, provided by an application running on a device, for displaying in a three-dimensional display mode. The at least one processor is further configured to determine that the content corresponds to two-dimensional content. The at least one processor is further configured to identify a portion of the two-dimensional content for enhancement by a three-dimensional render. The at least one processor is further configured to enhance, in response to the determining, the portion of the two-dimensional content by the three-dimensional renderer. The at least one processor is further configured to provide for display of the enhanced portion of the two-dimensional content on a display of the device in the three-dimensional display mode.

Abstract: The present disclosure relates to providing a computer-generated reality (CGR) platform for generating CGR environments including virtual and augmented reality environments. In some embodiments, the platform includes an operating-system-level (OS-level) process that simulates and renders content in the CGR environment, and one or more application-level processes that provide information related to the content to be simulated and rendered to the OS-level process.

Abstract: The disclosure pertains to the operation of graphics systems and to a variety of architectures for design and/or operation of a graphics system spanning from the output of an application program and extending to the presentation of visual content in the form of pixels or otherwise. In general, many embodiments of the invention envision the processing of graphics programming according to an on-the-fly decision made regarding how best to use the specific available hardware and software. In some embodiments, a software arrangement may be used to evaluate the specific system hardware and software capabilities, then make a decision regarding what is the best graphics programming path to follow for any particular graphics request. The decision regarding the best path may be made after evaluating the hardware and software alternatives for the path in view of the particulars of the graphics program to be processed.

Abstract: A first device coupled with a first display and an image sensor receives output data from a second device having a second display different from the first display. The output data represents content displayable by the second device on the second display. The first device determines, using the image sensor, a position of the second display relative to the first device and causes the first display to display content based on the output data received from the second device and the determined position of the second display relative to the first device.

Abstract: In an exemplary process for interacting with user interface objects using an eye gaze, an affordance associated with a first object is displayed. A gaze direction or a gaze depth is determined. While the gaze direction or the gaze depth is determined to correspond to a gaze at the affordance, a first input representing user instruction to take action on the affordance is received, and the affordance is selected responsive to receiving the first input.

Abstract: An apparatus includes a transceiver and a controller in communication with the transceiver. The controller is configured to determine a target lighting condition in a room relative to a current lighting condition in the room. The controller is also configured to generate a control signal with instructions to adjust an ambient lighting peripheral in the room based on the determined target lighting condition. The control signal is provided to the transceiver for transmission.

Abstract: Various implementations disclosed herein include devices, systems, and methods that enable a device to provide a view of virtual elements and a physical environment where the presentation of the virtual elements is based on positioning relative to the physical environment. In one example, a device is configured to detect a change in positioning of a virtual element, for example, when a virtual element is added, moved, or the physical environment around the virtual element is changed. The location of the virtual element in the physical environment is used to detect an attribute of the physical environment upon which the presentation of the virtual element depends. Thus, the device is further configured to detect an attribute (e.g., surface, table, mid-air, etc.) of the physical environment based on the placement of the virtual element and present the virtual element based on the detected attribute.

Abstract: Systems and techniques for generating an artificial terrain map can select a plurality of component terrains for each of several terrain types. Values of a selection noise map ranging between a lower bound and an upper bound can be computed on a tile-by-tile basis. One or more noise bands within the range of selection-noise-map values can correspond to each terrain type. The noise map can be sampled on a tile-by-tile basis to determine a tile value for each tile. Each respective tile can be assigned to the noise band in which the tile value falls. A terrain value can be assigned to each respective tile in the selection noise map based on the noise band assigned to the respective tile. Generated maps in machine-readable form can be converted to a human-perceivable form, and/or to a modulated signal form conveyed over a communication connection.

Abstract: A mixed reality system that includes a device and a base station that communicate via a wireless connection The device may include sensors that collect information about the user's environment and about the user. The information collected by the sensors may be transmitted to the base station via the wireless connection. The base station renders frames or slices based at least in part on the sensor information received from the device, encodes the frames or slices, and transmits the compressed frames or slices to the device for decoding and display. The base station may provide more computing power than conventional stand-alone systems, and the wireless connection does not tether the device to the base station as in conventional tethered systems. The system may implement methods and apparatus to maintain a target frame rate through the wireless link and to minimize latency in frame rendering, transmittal, and display.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide a CGR environment in which virtual objects from one or more apps are included. User interactions with the virtual objects are detected and interpreted by a system that is separate from the apps that provide the virtual objects. The system detects user interactions received via one or more input modalities and interprets those user interactions as events. These events provide a higher-level, input modality-independent, abstractions of the lower-level input-modality dependent user interactions that are detected. The system uses UI capability data provided by the apps to interpret user interactions with respect to the virtual object provided by the apps. For example, the UI capability data can identify whether a virtual object is moveable, actionable, hover-able, etc. and the system interprets user interactions at or near the virtual object accordingly.

Abstract: A method of assembling a tile map can include assigning each tile in a plurality of tiles to one or more color groups in correspondence with a measure of a color profile of the respective tile: A position of each tile in relation to one or more neighboring tiles can be determined from a position of a silhouette corresponding to each respective tile in relation to one or more neighboring silhouettes within a set containing a plurality of silhouettes. The plurality of tiles can be automatically assembled into a tile map, with a position of each tile in the tile map being determined from the color group to which the respective tile belongs and the determined position of the respective tile in relation to the one or more neighboring tiles. Tangible, non-transitory computer-readable media can include computer executable instructions that, when executed, cause a computing environment to implement disclosed methods.

Abstract: The refresh rate of a display of a portable display device is dependent on the degree of device motion detected by one or more motion sensors included in the portable display device, according to an embodiment of the invention. In an embodiment, when no device motion is detected by the one or more sensors, the display of the portable display device is refreshed at an initial refresh rate. When the one or more motion sensors detects a degree of device motion above a motion threshold, the refresh rate of the display is decreased to a motion-based refresh rate, according to an embodiment. In an embodiment, the degree of motion of moving content on the display is also taken into account when determining the display refresh rate.