Abstract: In some implementations, the techniques may include performing a first readout of an image sensor to generate a first image frame, where the first readout is a first type of readout of the image sensor. In addition, the techniques may include identifying, by the first processor, one or more first regions of interest in the first image frame. The techniques may include performing a second readout of the image sensor to generate a second image frame, where the second readout is a second type of readout of the image sensor of a device, and the second image frame may include the one or more first regions of interest, where the electronic device has a display and the second image frame is smaller than the display. Moreover, the techniques may include presenting the second image frame on the display.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide detecting and decoding of visual markers that encode data. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, detecting the visual marker is based on a pre-determined portion of the visual marker. In some implementations, detecting the visual marker is based on a predefined shape. In some implementations, a visual marker interpretation scheme is determined for decoding the visual marker. In some implementations, the visual marker is decoded based on one or more colors being attributed a same value. In some implementations, the visual marker is decoded based on one or more other colors being attributed a different value.

Abstract: Various implementations disclosed herein include devices, systems, and methods that are capable of adjusting the appearance of computer-generated objects (e.g., augmentations) that are predicted to be outside the current focus of an optical-see-through HMD user's attention. In some implementations, a method includes displaying a first computer-generated object and a second computer-generated object on an optical-see-though display of a HMD. In some implementations, the first computer-generated object is identified based on a prediction that the attention of a user is directed to the first computer-generated object, and an appearance of the second computer-generated object is adjusted to change from a first state to a second state.

Abstract: Various implementations provide passthrough video based on adjusting camera parameters based on environment modeling. An environment characteristic may be determined based on modeling the physical environment based on sensor data captured via one or more sensors. For example, this may involve determining environment light source optical characteristics, environment surfaces optical characteristics, a 3D mapping of the environment, user behavior, a prediction of optical characteristics of light coming in the camera, and the like. The method may involve, based on the environment characteristic, determining a camera parameter for an image captured via the image sensor. For example, the method may determine exposure, gain, tone mapping, color balance, noise reduction, sharpness enhancement. The method may determine the camera parameter based on user information, e.g., user preferences, user activity, etc.

Abstract: Predicting and counting repetitions of a physical activity includes capturing first sensor data, by a first magnetometer on a wearable device, a change in a magnetic field indicative of a ferromagnetic object moving in relation to the wearable device. One or more characteristics of a user motion are determined based on the first sensor data. A count of repetitions of the user motion are determined based on the one or more characteristics of the user motion, and a notification of the count of repetitions is generated.

Abstract: A method includes sensing a plurality of light superposition characteristic values associated with ambient light from a physical environment. The ambient light emanates from the physical environment towards one side of a translucent display. The plurality of light superposition characteristic values quantifies interactions with the ambient light. The method includes determining a plurality of display correction values associated with the electronic device based on a function of the plurality of light superposition characteristic values and predetermined display characteristics of a computer-generated reality (CGR) object. The method includes changing one or more display operating parameters associated with the electronic device in accordance with the plurality of display correction values in order to satisfy the predetermined display characteristics of the CGR object within a performance threshold.

Abstract: An apparatus includes a primary camera sensor configured to capture images having a first resolution, a primary processing circuit configured to process images captured by the primary camera sensor, a secondary camera sensor configured to capture images having a second resolution, and a secondary processing circuit configured to process images captured by the secondary camera sensor. In response to a determination that a particular object of interest is included in a particular image, the secondary processing circuit may be further configured to cause the primary processing circuit and the primary camera sensor to exit a reduced power mode. The primary camera sensor may be further configured, in response to the exiting, to capture a different image. The primary processing circuit may also be configured to process the different image to validate the particular object of interest.

Abstract: In one implementation, a method of controlling a dimming level of dimmable optical element based on a predicted ambient light level is performed at a device including one or more processors, non-transitory memory, and a dimmable optical element. The method includes predicting a change, in a first direction, in an ambient light level at a future time. The method includes changing, at a first time in advance of the future time and in the first direction, a transmission coefficient of the dimmable optical element based on the predicted change in the ambient light level. The method includes changing, at a second time after the first time and in a second direction opposite the first direction, the transmission coefficient of the dimmable optical element based on the ambient light level at the second time.

Abstract: Various implementations disclosed herein include devices, systems, and methods that determine a type of environment based on light assessment. For example, an example process may include obtaining a first set of light data from a first sensor (e.g., a multiwavelength ambient light sensor) in a physical environment that includes a light source. The first set of light data may identify diffuse light in each of multiple light wavelength ranges that is received by the first sensor. The process may further include obtaining a second set of data from the second sensor in the physical environment. The process may further include determining, based on the first set of light data and the second set of data, that the physical environment is a first type of environment (e.g., indoor) or a second type of environment (e.g., outdoor) based on a set of criteria (e.g., artificial light vs. sunlight).

Abstract: Various implementations disclosed herein include devices, systems, and methods that obtain sensor data from one or more sensors in a physical environment, and a context is determined based on the sensor data, where the context includes a location of the physical environment and an occurrence of an activity in the physical environment. Camera parameters are selected based on historical parameter data identified based on the context, where the historical parameter data is identified based on camera parameters previously used in the location during prior occurrences of the activity. Then, a camera is configured to capture an image using the selected camera parameters in the location during the occurrence of the activity. In some implementations, the adjusted camera parameters are selected based on context and shared information from a different electronic device regarding camera parameters that were used in the same physical location and during the same activity.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide XR in which virtual objects are positioned based on the accuracy of localizing an electronic device in a physical environment. In some implementations, the technique assesses the accuracy of localization (e.g., centimeter-level accuracy, room-level accuracy, and building-level accuracy) and dynamically adjusts a display strategy. In some implementations, the technique determines a condition causing inaccuracy (e.g., a semantic condition such as “too fast”, “too far”, “too dark”), and provides a notification (e.g., “too fast-slow down”, “too far-move closer”, “too dark-turn on a light”) at the electronic device based on the condition causing the inaccuracy in the localization.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide point identification techniques for electronic devices such as optical see-through head mounted devices. In some implementations, a line of sight technique is used to identify a 3D position of a point. In some implementations, a touching technique is used to identify a 3D position of a point. In some implementations, different point identification techniques are automatically selected and used to identify a 3D position of a point. In some implementations, a 3D position of a point is associated with user input. In some implementations, a 3D position of a point is identified to determine distances, surface areas, or volumes.

Abstract: An apparatus includes a primary camera sensor configured to capture images having a first resolution, a primary processing circuit configured to process images captured by the primary camera sensor, a secondary camera sensor configured to capture images having a second resolution, and a secondary processing circuit configured to process images captured by the secondary camera sensor. In response to a determination that a particular object of interest is included in a particular image, the secondary processing circuit may be further configured to cause the primary processing circuit and the primary camera sensor to exit a reduced power mode. The primary camera sensor may be further configured, in response to the exiting, to capture a different image. The primary processing circuit may also be configured to process the different image to validate the particular object of interest.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide a visual marker including a plurality of markings arranged in a corresponding plurality of shapes. In some implementations, each marking is formed of a set of sub-markings separated by gaps and arranged according to a respective shape, and the gaps of the plurality of markings are configured to encode data and indicate orientation of the visual marker. In some implementations, the plurality of markings are arranged in a plurality of concentric rings of increasing size. In some implementations, the orientation is encoded in a first set of gaps and data in a second set of gaps of the gaps in the plurality of markings.

Abstract: Various implementations disclosed herein assess the blurriness of portions of images depicting shapes such as codes or text that have known structural elements. This may involve determining whether a portion of an image of a code or text is sufficiently clear (not blurry) to be accurately interpreted. Blur may be assessed based on spatial frequency of statistical analysis. Blur may be assessed using a machine learning model that is trained using target blur metrics determined based on spatial frequency (e.g., analysis of high frequency portions of discrete cosine transforms of image portions) or statistical analysis (e.g., based on corner/edge detection in image portions).

Abstract: Various implementations disclosed herein include devices, systems, and methods that estimate a location of a light source based on ambient light data. For example, an example process may include acquiring ambient light data from an ambient light sensor (ALS) during movement of a device in a physical environment, acquiring motion data from a motion sensor during the movement of the device, determining, based on the ambient light data and the motion data, estimates of three-dimensional (3D) locations of a light source with respect to the device during the movement of the device, and tracking a location of the device in a 3D coordinate system during the movement of the device based on the estimates of the 3D locations of the light source with respect to the device during the movement of the device.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide XR in which virtual objects are positioned based on the accuracy of localizing an electronic device in a physical environment. In some implementations, the technique assesses the accuracy of localization (e.g., centimeter-level accuracy, room-level accuracy, and building-level accuracy) and dynamically adjusts a display strategy. In some implementations, the technique determines a condition causing inaccuracy (e.g., a semantic condition such as “too fast”, “too far”, “too dark”), and provides a notification (e.g., “too fast-slow down”, “too far-move closer”, “too dark-turn on a light”) at the electronic device based on the condition causing the inaccuracy in the localization.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide measurement techniques for electronic devices such as optical see-through head mounted devices. In some implementations, a line of sight technique is used to identify a 3D position of a measurement point to enable measurement of an object in a 3D environment. In some implementations, different measurement point identification techniques are automatically selected and used to identify a 3D position of a measurement point to enable measurement of an object in a 3D environment. In some implementations, a 3D position of a measurement point is identified to enable measurement of an object in a 3D environment, where the measurement point is identified by selecting from multiple candidates that are determined using different measurement point selection techniques.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide a visual marker including a plurality of markings arranged in a corresponding plurality of shapes. In some implementations, each marking is formed of a set of sub-markings separated by gaps and arranged according to a respective shape, and the gaps of the plurality of markings are configured to encode data and indicate orientation of the visual marker. In some implementations, the plurality of markings are arranged in a plurality of concentric rings of increasing size. In some implementations, the orientation is encoded in a first set of gaps and data in a second set of gaps of the gaps in the plurality of markings.

Abstract: Identifying a specular surface, such as a mirror, in a captured scene includes extracting, from one or more images of the scene, a set of natural features and generating, from the image, a set of synthesized “mirrored” features. One or more correspondences may be determined between the set of natural features in the image and the set of synthesized mirrored features. A first set of features are identified based on the determined one or more correspondences as representing a specular surface (e.g., a mirror) located in the scene, and then a geometry and/or location of the specular surface within the scene may be determined. For example, in some embodiments, the feature from a determined pair of corresponding features in a scene that is determined to be farther away from the device that captured the image(s) of the scene may be determined to be the feature lying on the specular surface.