Abstract: A method of generating at least one image of a real environment comprises providing at least one environment property related to at least part of the real environment, providing at least one virtual object property related to a virtual object, determining at least one imaging parameter according to the at least one provided virtual object property and the at least one provided environment property, and generating at least one image of the real environment representing information about light leaving the real environment according to the determined at least one imaging parameter, wherein the light leaving the real environment is measured by at least one camera.

Abstract: Various implementations disclosed herein include devices, systems, and methods that use external device data to improve user data on a wearable device. For example, an example process may include receiving first sensor data, from a sensor of a wearable electronic device, depicting a user of the wearable electronic device. The process may further include receiving second sensor data depicting the user from a second device separate from the wearable electronic device. At least a part of the user depicted in the second sensor data is absent in the first sensor data. The process may further include determining that a portion of the second sensor data depicts the user based on a similarity between the first sensor data and second sensor data. The process may further include updating a user model based on the second sensor data.

Abstract: Various implementations disclosed herein include devices, systems, and methods that predict a proper light seal fit for a head mounted device (HMD) to reduce external light leakage into the HMD. For example, a process may include obtaining first images of a portion of a face of a user while the user is wearing the HMD and initial light seal contacts at least some perimeter regions around the portion of the face such that the HMD forms an enclosed area between the HMD and the portion of the face. Based on the first images illumination characteristics may be identified on the portion of the face corresponding to external light entering the enclosed area via one or more light source leakage regions between the face and the initial light seal. Based on the identified illumination characteristics one or more parameters for an adjusted light seal for the user may be determined.

Abstract: Various implementations disclosed herein include devices, systems, and methods that dynamically selects particular frames of image data for generating three-dimensional (3D) representations of a physical environment. For example, an example process may include acquiring frames of image data from one or more sensors in a physical environment. The process may include determining one or more deterrent properties associated with the frames of image data. The process may include selecting a subset of the frames of image data by determining that at least one of the determined deterrent properties satisfies at least one condition of one or more conditions. The process may include determining a scene understanding of the physical environment based on the selected subset of the frames. The scene understanding may determine geometric properties of the physical environment.

Abstract: Methods and apparatus for specular surface mapping in which a camera detects reflections of a light source from a specular surface. The detected light sources may be projected onto a celestial sphere as virtual point sources. True positive observations should be tightly clustered on the celestial sphere; thus, false positives may be identified and removed. Specular surface information may then be determined from clusters of the virtual point sources on the celestial sphere. The clusters of virtual point sources on the celestial sphere may be identified and used to identify a surface as a specular surface. The clusters may also be used to extract other information regarding the specular surface, including but not limited to distance to and extent of the specular surface.

Abstract: Methods and apparatus for specular surface mapping in which a camera detects reflections of a light source from a specular surface. The detected light sources may be projected onto a celestial sphere as virtual point sources. True positive observations should be tightly clustered on the celestial sphere; thus, false positives may be identified and removed. Specular surface information may then be determined from clusters of the virtual point sources on the celestial sphere. The clusters of virtual point sources on the celestial sphere may be identified and used to identify a surface as a specular surface. The clusters may also be used to extract other information regarding the specular surface, including but not limited to distance to and extent of the specular surface.

Abstract: A VR system for vehicles that may implement methods that address problems with vehicles in motion that may result in motion sickness for passengers. The VR system may provide virtual views that match visual cues with the physical motions that a passenger experiences. The VR system may provide immersive VR experiences by replacing the view of the real world with virtual environments. Active vehicle systems and/or vehicle control systems may be integrated with the VR system to provide physical effects with the virtual experiences. The virtual environments may be altered to accommodate a passenger upon determining that the passenger is prone to or is exhibiting signs of motion sickness.

Abstract: Various implementations record a shared event by generating a temporal-based three-dimensional (3D) representation for use in providing/rendering views of the recorded event. For example, a method may include determining that a second device is currently at an event at a physical environment. The method may further include providing a notification to the second device for an option to authorize use of sensor data based on determining that the second device is currently at the event. The method may further include receiving authorization to use the sensor data. The method may further include obtaining the sensor data from the second device in accordance with the authorization. The method may further include generating a temporal-based 3D representation of the event based on the sensor data obtained from the second device and additional sensor data obtained from one or more other devices at the event.

Abstract: In one implementation, a method includes emitting light with modulating intensity from a plurality of light sources towards an eye of a user. The method includes receiving light intensity data indicative of an intensity of the plurality of glints reflected by the eye of the user in the form of a plurality of glints. The method includes determining an eye tracking characteristic of the user based on the light intensity data. In one implementation, a method includes generating, using an event camera comprising a plurality of light sensors at a plurality of respective locations, a plurality of event messages, each of the plurality of event messages being generated in response to a particular light sensor detecting a change in intensity of light and indicating a particular location of the particular light sensor. The method includes determining an eye tracking characteristic of a user based on the plurality of event messages.

Abstract: Various implementations disclosed herein include devices, systems, and methods that use external device data to improve user data on a wearable device. For example, an example process may include receiving first sensor data, from a sensor of a wearable electronic device, depicting a user of the wearable electronic device. The process may further include receiving second sensor data depicting the user from a second device separate from the wearable electronic device. At least a part of the user depicted in the second sensor data is absent in the first sensor data. The process may further include determining that a portion of the second sensor data depicts the user based on a similarity between the first sensor data and second sensor data. The process may further include updating a user model based on the second sensor data.

Abstract: Identifying a specular surface, such as a mirror, in a captured scene includes extracting, from one or more images of the scene, a set of natural features and generating, from the image, a set of synthesized “mirrored” features. One or more correspondences may be determined between the set of natural features in the image and the set of synthesized mirrored features. A first set of features are identified based on the determined one or more correspondences as representing a specular surface (e.g., a mirror) located in the scene, and then a geometry and/or location of the specular surface within the scene may be determined. For example, in some embodiments, the feature from a determined pair of corresponding features in a scene that is determined to be farther away from the device that captured the image(s) of the scene may be determined to be the feature lying on the specular surface.

Abstract: The invention is related to a method of presenting a digital information related to a real object, comprising determining a real object, providing a plurality of presentation modes, wherein the plurality of presentation modes comprises an augmented reality mode, and at least one of a virtual reality mode and an audio mode, providing at least one representation of a digital information related to the real object, determining a spatial relationship between a camera and a reference coordinate system under consideration of an image captured by the camera, selecting a presentation mode from the plurality of presentation modes according to the spatial relationship, and presenting the at least one representation of the digital information using the selected presentation mode.

Abstract: Various implementations disclosed herein include devices, systems, and methods that use light probes to facilitate the display of virtual objects in 3D environments. A light probe provides lighting information that describes light incident on a point in space in a 3D environment. For example, a light probe may describe such incident light using an environment map. Such lighting information can be used to provide realistic appearances for objects placed at or near light probe locations in the 3D environment. Implementations disclosed herein determine the light probe locations in real-time or other 3D environments that are generated based on a live physical environment. A digital representation of the live physical environment is used to determine where to position the light probes, how many light probes to use, and/or various light probe attributes.

Abstract: Various implementations disclosed herein include devices, systems, and methods that use a marking on a transparent surface (e.g., a prescription lens insert for an HMD) to identify information (e.g., prescription parameters) about the transparent surface. In some implementations, the markings do not interfere with eye tracking through the transparent surface or using the transparent surface to view virtual content or a physical environment. In some implementations, image data is obtained from an image sensor of an electronic device, the image data corresponding to a transparent surface attached to the electronic device. Then, a code is identified in the image data, wherein the code is detectable on the transparent surface by the image sensor without interfering with a function of the electronic device involving the transparent surface. In some implementations, content is provided at the electronic device based on the identified code, wherein the content is viewable through the transparent surface.

Abstract: A method of generating at least one image of a real environment comprises providing at least one environment property related to at least part of the real environment, providing at least one virtual object property related to a virtual object, determining at least one imaging parameter according to the at least one provided virtual object property and the at least one provided environment property, and generating at least one image of the real environment representing information about light leaving the real environment according to the determined at least one imaging parameter, wherein the light leaving the real environment is measured by at least one camera.

Abstract: Various implementations disclosed herein include devices, systems, and methods that determine an orientation of a three-dimensional (3D) model with respect to a global reference direction. For example, an example process may include obtaining an image from an image sensor, determining an orientation of the image sensor with respect to the direction of a celestial element, determining an orientation of the image sensor with respect to a global reference direction based on the orientation of the image sensor with respect to the direction of the celestial element and a relationship between the direction of the celestial element and the global reference direction, obtaining a 3D model of the physical environment, determining an orientation of the image sensor with respect to the 3D model, and determining an orientation of the 3D model with respect to the global reference direction.

Abstract: A VR system for vehicles that may implement methods that address problems with vehicles in motion that may result in motion sickness for passengers. The VR system may provide virtual views that match visual cues with the physical motions that a passenger experiences. The VR system may provide immersive VR experiences by replacing the view of the real world with virtual environments. Active vehicle systems and/or vehicle control systems may be integrated with the VR system to provide physical effects with the virtual experiences. The virtual environments may be altered to accommodate a passenger upon determining that the passenger is prone to or is exhibiting signs of motion sickness.

Abstract: The present disclosure is related to a method and device for detecting a touch between at least part of a first object and at least part of a second object, wherein the at least part of the first object has a different temperature than the at least part of the second object. The method includes providing at least one thermal image of a portion of the second object, determining in at least part of the at least one thermal image a pattern which is indicative of a particular value or range of temperature or a particular value or range of temperature change, and using the determined pattern for detecting a touch between the at least part of the first object and the at least part of the second object.

Abstract: Various implementations disclosed herein include devices, systems, and methods that detect surfaces and reflections in such surfaces. Some implementations involve providing a CGR environment that includes virtual content that replaces the appearance of a user or the user's device in a mirror or other surface providing a reflection. For example, a CGR environment may be modified to include a reflection of the user that does not include the device that the user is holding or wearing. In another example, the CGR environment is modified so that virtual content, such as a newer version of the electronic device or a virtual wand, replaces the electronic device in the reflection. In another example, the CGR environment is modified so that virtual content, such as a user avatar, replaces the user in the reflection.

Abstract: Various implementations disclosed herein include devices, systems, and methods that detect surfaces and reflections in such surfaces. Some implementations involve providing a CGR environment that includes virtual content that replaces the appearance of a user or the user's device in a mirror or other surface providing a reflection. For example, a CGR environment may be modified to include a reflection of the user that does not include the device that the user is holding or wearing. In another example, the CGR environment is modified so that virtual content, such as a newer version of the electronic device or a virtual wand, replaces the electronic device in the reflection. In another example, the CGR environment is modified so that virtual content, such as a user avatar, replaces the user in the reflection.