Abstract: Various implementations disclosed herein include devices, systems, and methods that interpret user activity as user interactions with user interface (UI) elements positioned within a three-dimensional (3D) space such as an extended reality (XR) environment. Some implementations enable user interactions with virtual elements displayed in 3D environments that utilize alternative input modalities, e.g., XR environments that interpret user activity as either direct interactions or indirect interactions with virtual elements.

Abstract: Various implementations provide views of 3D environments (e.g., extended reality (XR) environments). Non-eye-based user activity, such as hand gestures, is associated with some types of eye-based activity, such as the user gazing at a particular user interface component displayed within a view of a 3D environment. For example, a user's pinching hand gesture may be associated with the user gazing at a particular user interface component, such as a button, at around the same time as the pinching hand gesture is made. These associated behaviors (e.g., the pinch and gaze at the button) may then be interpreted as user input, e.g., user input selecting or otherwise acting upon that user interface component. In some implementations, non-eye-based user activity is only associated with types of eye-based user activity that are likely to correspond to a user perceiving what they are seeing and/or intentionally looking at something.

Abstract: In some embodiments, a computer system enables a user to invoke display of transport controls (and/or other controls associated with controlling playback of content) using gaze inputs, gesture inputs, or a combination of these. In some embodiments, in response to detecting a first user input, the computer system displays a first set of controls in a reduced-prominence state (e.g., in a manner that is not unduly distracting to the user), and in response to detecting a second user input, the computer system displays a second set of controls in an increased-prominence state (e.g., in a more visually prominent state). The second set of controls optionally includes more controls than the first set of controls.

Abstract: Various implementations provide views of 3D environments (e.g., extended reality (XR) environments). Non-eye-based user activity, such as hand gestures, is associated with some types of eye-based activity, such as the user gazing at a particular user interface component displayed within a view of a 3D environment. For example, a user's pinching hand gesture may be associated with the user gazing at a particular user interface component, such as a button, at around the same time as the pinching hand gesture is made. These associated behaviors (e.g., the pinch and gaze at the button) may then be interpreted as user input, e.g., user input selecting or otherwise acting upon that user interface component. In some implementations, non-eye-based user activity is only associated with types of eye-based user activity that are likely to correspond to a user perceiving what they are seeing and/or intentionally looking at something.

Abstract: Techniques for mapping a user input motion includes detecting an input motion by a user, determining an origin for an input motion in a user-centric spherical coordinate system, determining an arc length for the input motion based on the determined origin, mapping the arc length of the input motion to a 2D plane of a user input component, and presenting a movement of a user input component on the 2D plane in accordance with the mapping.

Abstract: Various implementations disclosed herein facilitate interactions with a user interface in 3D environment in which a user interface element is moved based on a user movement in a way that the user interface element appears to lag behind or follow a portion of the user (e.g., the user's fingertip). The user interface element may be moved in a way that it converges with and thus catches up to the portion of the user. Such convergence may be based on the speed of the movement of the portion of the user. No convergence may occur when the portion of the user is not moving or is moving below a threshold speed. When the portion of the user is moving (e.g., above a threshold speed), the user interface component may converge with the portion of the user and the rate of convergence may be increased with faster speeds.

Abstract: Various implementations provide views of 3D environments (e.g., extended reality (XR) environments). Non-eye-based user activity, such as hand gestures, is associated with some types of eye-based activity, such as the user gazing at a particular user interface component displayed within a view of a 3D environment. For example, a user's pinching hand gesture may be associated with the user gazing at a particular user interface component, such as a button, at around the same time as the pinching hand gesture is made. These associated behaviors (e.g., the pinch and gaze at the button) may then be interpreted as user input, e.g., user input selecting or otherwise acting upon that user interface component. In some implementations, non-eye-based user activity is only associated with types of eye-based user activity that are likely to correspond to a user perceiving what they are seeing and/or intentionally looking at something.

Abstract: Suppressing a hand gesture upon detecting peripheral events on a peripheral device includes determining a first hand pose for a first hand a second hand pose for a second hand in response to a detected peripheral device peripheral event, determining, based on the first hand pose and the second hand pose, at least one hand of the first hand and the second hand in a peripheral use mode, detecting an input gesture from a hand of the at least one hand determined to be in the peripheral use mode, and rejecting the input gesture by a user input pipeline in accordance with the determination that the hand is in the peripheral use mode. The presence of a peripheral device is confirmed by activating a computer vision system in response to determining that a peripheral use condition is satisfied.

Abstract: Aspects of the subject technology provide improved techniques for gesture recognition. Improved techniques may include detecting and/or classifying an interaction between the body part and another object in a scan of the body part, and then controlling recognition of a gesture based on the interaction. In an aspect, recognition parameters may be selected based on the interaction classification that disable recognition of one or more gestures while not disabling recognition of other gestures.

Abstract: While a view of an environment is visible via a display generation component of a computer system, the computer system detects a gaze input directed to a first location, corresponding to a first user interface element, in the environment. In response to detecting the gaze input: if a user's hand is in a predefined configuration during the gaze input, the computer system: provides, to the first user interface element, information about the gaze input; and then, in response to detecting the gaze input moving to a different, second location in the environment while the user's hand is maintained in the predefined configuration, provides, to a second user interface element that corresponds to the second location, information about the gaze input. If the user's hand is not in the predefined configuration during the gaze input, the computer system forgoes providing, to the first user interface element, information about the gaze input.

Abstract: Techniques for managing an engagement zone include tracking, by a system, a hand of a user and determining that a height of the hand of the user satisfies a first threshold height. In accordance with determining that the height of the hand of the user satisfies the first threshold height, the techniques also include initiating a UI engagement state, wherein the system monitors the user for user input during the UI engagement state, and determining user input into the system based on a user motion detected while the hand is tracked. The threshold height is associated with a boundary of a UI engagement zone and is modifiable based on user activity.

Abstract: An electronic device detects an input sequence that includes an increase in intensity of an input that corresponds to a first input event. If the input sequence includes a second input event, including a decrease in intensity of the input after the first input event, a long press is not detected and a first operation is performed. If the second input event is not detected within a second time period that is longer than the first time period and the input had a characteristic intensity above a respective intensity threshold between when the first input event was detected and when the second time period elapsed, a long press is detected on an accelerated basis, and a second operation is performed. Finally, a long press is detected if the second input event is not detected within a longer, default time period, and the second operation is performed.

Abstract: An electronic device detects an input sequence that includes an increase in intensity of an input that corresponds to a first input event. If the input sequence includes a second input event, including a decrease in intensity of the input after the first input event, a long press is not detected and a first operation is performed. If the second input event is not detected within a second time period that is longer than the first time period and the input had a characteristic intensity above a respective intensity threshold between when the first input event was detected and when the second time period elapsed, a long press is detected on an accelerated basis, and a second operation is performed. Finally, a long press is detected if the second input event is not detected within a longer, default time period, and the second operation is performed.

Abstract: An electronic device detects a change in intensity of an input on an input element that includes detecting an increase in intensity followed by a decrease in intensity, and determines whether the first decrease in intensity of the input meets up-click detection criteria, which require that the intensity of the input decrease below a first up-click intensity threshold in order for the up-click detection criteria to be met. The first up-click intensity threshold is time varying, in accordance with the detected intensity of the input during the first decrease in intensity of the input. If the first decrease in intensity of the input meets the up-click detection criteria, first feedback is provided, indicating that the first decrease in intensity of the input was recognized as an up-click, and if the first decrease in intensity of the input does not meet the up-click detection criteria, the first feedback is not provided.

Abstract: Thermal compensation can be applied to force measurements of a force-sensitive button. A temperature differential between an object and the force-sensitive button can result in changes in the reconstructed force by the force sensor due to thermal effects rather than actual user force, which in turn can result in degraded performance of the force sensor (e.g., false positive or inconsistent activation force). In some examples, a force-sensitive button can include a force sensor configured to measure an amount of force applied to the force-sensitive button, and a temperature sensor configured to measure a temperature associated with the force sensor. The measured temperature can be used to compensate the amount of force measured by the force sensor based on the temperature associated with the force sensor. In some examples, the thermal compensation can be applied when an object is detected contacting the force-sensitive button (i.e., when rapid temperature differentials can occur).

Abstract: An electronic device detects a change in intensity of an input on an input element that includes detecting an increase in intensity followed by a decrease in intensity, and determines whether the first decrease in intensity of the input meets up-click detection criteria, which require that the intensity of the input decrease below a first up-click intensity threshold in order for the up-click detection criteria to be met. The first up-click intensity threshold is time varying, in accordance with the detected intensity of the input during the first decrease in intensity of the input. If the first decrease in intensity of the input meets the up-click detection criteria, first feedback is provided, indicating that the first decrease in intensity of the input was recognized as an up-click, and if the first decrease in intensity of the input does not meet the up-click detection criteria, the first feedback is not provided.

Abstract: Thermal compensation can be applied to force measurements of a force-sensitive button. A temperature differential between an object and the force-sensitive button can result in changes in the reconstructed force by the force sensor due to thermal effects rather than actual user force, which in turn can result in degraded performance of the force sensor (e.g., false positive or inconsistent activation force). In some examples, a force-sensitive button can include a force sensor configured to measure an amount of force applied to the force-sensitive button, and a temperature sensor configured to measure a temperature associated with the force sensor. The measured temperature can be used to compensate the amount of force measured by the force sensor based on the temperature associated with the force sensor. In some examples, the thermal compensation can be applied when an object is detected contacting the force-sensitive button (i.e., when rapid temperature differentials can occur).

Abstract: An electronic device detects a change in intensity of an input on an input element that includes detecting an increase in intensity followed by a decrease in intensity, and recognizes at least a portion of the change in intensity of the input as a first input event that is associated with a first operation, for example a single click operation. After recognizing the first input event, the device delays performance of the first operation while monitoring subsequent changes in intensity of the input for a second input event, wherein the delay is limited by a default delay time period. If the second event is recognized before default delay time period has elapsed, a second operation is performed and the first operation is not performed. However, if early-confirmation criteria for the first input event are met before the default delay period elapses, the first operation is performed.

Abstract: An electronic device detects a change in intensity of an input on an input element that includes detecting an increase in intensity followed by a decrease in intensity, and recognizes at least a portion of the change in intensity of the input as a first input event that is associated with a first operation, for example a single click operation. After recognizing the first input event, the device delays performance of the first operation while monitoring subsequent changes in intensity of the input for a second input event, wherein the delay is limited by a default delay time period. If the second event is recognized before default delay time period has elapsed, a second operation is performed and the first operation is not performed. However, if early-confirmation criteria for the first input event are met before the default delay period elapses, the first operation is performed.

Abstract: An electronic device can include gain-based error tracking for improved force sensing performance. The electronic device can comprise a plurality of force sensors (e.g., coupled to a touch sensor panel configured to detect an object touching the touch sensor panel). The plurality of force sensors can be configured to detect an amount of force with which the object touches the touch sensor panel. A processor can be coupled to the plurality of force sensors, and the processor can be configured to: in accordance with a determination that an acceleration characteristic of the electronic device is less than a threshold, determine an error metric for one or more of the plurality of force sensors, and in accordance with a determination that the acceleration characteristic of the electronic device is not less than the threshold, forgo determining the error metric for one or more of the plurality of force sensors.