Abstract: Processing gesture input includes obtaining hand tracking data based on a set of camera frames, determining a hand pose based on the hand tracking data, and determining an intentionality classification for a gesture based on the hand pose. An input action corresponding to the gesture is enabled based on the hand pose and the intentionality classification. An occlusion classification is determined for the hand based on the hand pose and the input gesture can be determined based on the occlusion classification.

Abstract: Techniques for managing an engagement zone include tracking, by a system, a hand of a user and determining that a height of the hand of the user satisfies a first threshold height. In accordance with determining that the height of the hand of the user satisfies the first threshold height, the techniques also include initiating a UI engagement state, wherein the system monitors the user for user input during the UI engagement state, and determining user input into the system based on a user motion detected while the hand is tracked. The threshold height is associated with a boundary of a UI engagement zone and is modifiable based on user activity.

Abstract: Techniques for mapping a user input motion includes detecting an input motion by a user, determining an origin for an input motion in a user-centric spherical coordinate system, determining an arc length for the input motion based on the determined origin, mapping the arc length of the input motion to a 2D plane of a user input component, and presenting a movement of a user input component on the 2D plane in accordance with the mapping.

Abstract: Various implementations disclosed herein include devices, systems, and methods that interpret user activity as user interactions with user interface (UI) elements positioned within a three-dimensional (3D) space such as an extended reality (XR) environment. Some implementations enable user interactions with virtual elements displayed in 3D environments that utilize alternative input modalities, e.g., XR environments that interpret user activity as either direct interactions or indirect interactions with virtual elements.

Abstract: A movement of a user input component is presented in accordance with a mapping of a user input motion from a user-centric spherical coordinate system to a 2D plane. An input motion by a user is detected. An origin for the input motion is identified in a user-centric spherical coordinate system. An arc length for the input motion is determined based on the identified origin. The input motion is mapped from the user-centric spherical coordinate system to a 2D plane of a user input component, and a movement of the user input component is presented on the 2D plane in accordance with the mapping.

Abstract: Suppressing a hand gesture upon detecting peripheral events on a peripheral device includes determining a first hand pose for a first hand a second hand pose for a second hand in response to a detected peripheral device peripheral event, determining, based on the first hand pose and the second hand pose, at least one hand of the first hand and the second hand in a peripheral use mode, detecting an input gesture from a hand of the at least one hand determined to be in the peripheral use mode, and rejecting the input gesture by a user input pipeline in accordance with the determination that the hand is in the peripheral use mode. The presence of a peripheral device is confirmed by activating a computer vision system in response to determining that a peripheral use condition is satisfied.

Abstract: While a view of an environment is visible, a computer system detects that attention of a user is directed toward a location of a hand of the user, and in response: in accordance with a determination that the attention of the user is directed toward the location of the hand while first criteria are met, wherein the first criteria include a requirement that the hand is in a respective pose and oriented with a palm of the hand facing toward a viewpoint of the user in order for the first criteria to be met, the computer system displays a control corresponding to the location of the hand; and in accordance with a determination that the attention of the user is directed toward the location of the hand while the first criteria are not met, the computer system forgoes displaying the control.

Abstract: While a view of an environment is visible via one or more display generation components of a computer system, and while the view of the environment includes a respective object that moves as a hand of a user moves, the computer system detects, via one or more input devices, a respective input. In response to detecting the respective input: in accordance with a determination that first criteria are met, the first criteria including a requirement that the hand of the user is holding a controller, the computer system displays, via the one or more display generation components, a first user interface object at a first location relative to the respective object; and, in accordance with a determination that the first criteria are not met, the computer system forgoes displaying the first user interface object at the first location.

Abstract: Input gestures having a particular palm orientation are detected based on geometric characteristics of a hand relative to a head. Gaze information is used to determine a hand gesture state. The gesture state refers to a palm-up gesture or a palm-flip gesture. A hand orientation state machine is used to determine a hand orientation state based on the geometric characteristics. A gesture detection state machine is used to determine a hand gesture based on a hand orientation state and the gaze vector. An action is invoked based on the hand gesture state.

Abstract: While a view of an environment is visible, a computer system detects a selection input performed by a hand of a user. In response to detecting the selection input and while the hand is in a second orientation with the palm of the hand facing away from the viewpoint of the user: in accordance with a determination that the selection input was detected after detecting a change in orientation of the hand from a first orientation with the palm facing toward the viewpoint of the user to the second orientation with the palm facing away from the viewpoint of the user, and that the change in orientation of the hand was detected while attention of the user was directed toward a location of the hand, the computer system displays a control user interface that provides access to a plurality of controls corresponding to different functions of the computer system.

Abstract: Enabling gesture recognition and input based on hand tracking data and occlusion information is described. A determination is made as to whether a hand or a portion of a hand is occluded by a physical object or by the hand itself, and filters and consolidate the occlusion scores for each portion of the hand to determine whether to invoke or dismiss an input action associated with an input gesture. In doing so, hand tracking data can be used to obtain occlusion data and pose data from which input gesture invocation and gating can be implemented.

Abstract: Various implementations disclosed interpret direct touch-based gestures, such as drag and scroll gestures, made by a user virtually touching one position of a user interface and moving their hand to another position of the user interface. For example, such gestures may be made relative to a user interface presented in an extended reality (XR) environment. In some implementations, a user movement is interpreted using one or more techniques that avoid unexpected gain or loss of user-interface-associated motion. Some implementations determine which segments of a movement to associate with user interface content based on characteristics of the movement. Some implementations determine that a break occurs when a user movement leaves a break volume that is adjusted dynamically.

Abstract: Various implementations disclosed herein include devices, systems, and methods that interpret user activity as user interactions with user interface (UI) elements positioned within a three-dimensional (3D) space such as an extended reality (XR) environment. Some implementations enable user interactions with virtual elements displayed in 3D environments that utilize alternative input modalities, e.g., XR environments that interpret user activity as either direct interactions or indirect interactions with virtual elements.

Abstract: Various implementations disclosed interpret direct touch-based gestures, such as drag and scroll gestures, made by a user virtually touching one position of a user interface and moving their hand to another position of the user interface. For example, such gestures may be made relative to a user interface presented in an extended reality (XR) environment. In some implementations, a user movement is interpreted using one or more techniques that avoid unexpected gain or loss of user-interface-associated motion. Some implementations determine which segments of a movement to associate with user interface content based on characteristics of the movement. Some implementations determine that a break occurs when a user movement leaves a break volume that is adjusted dynamically.

Abstract: Methods, devices, and systems, in some implementations, stabilize user interface element using a stabilization property that is determined based on assessing the user's movement, e.g., assessing a hand movement characteristic and a head movement characteristic. In one example, the stabilization property restricts movement of user interface elements in certain circumstances, e.g., based on based on head movement (e.g., angular speed), hand movement (e.g., pinch centroid angular speed), and/or hand translation (e.g., pinch centroid speed).

Abstract: Suppressing a hand gesture upon detecting peripheral events on a peripheral device includes determining a first hand pose for a first hand a second hand pose for a second hand in response to a detected peripheral device peripheral event, determining, based on the first hand pose and the second hand pose, at least one hand of the first hand and the second hand in a peripheral use mode, detecting an input gesture from a hand of the at least one hand determined to be in the peripheral use mode, and rejecting the input gesture by a user input pipeline in accordance with the determination that the hand is in the peripheral use mode. The presence of a peripheral device is confirmed by activating a computer vision system in response to determining that a peripheral use condition is satisfied.

Abstract: Enabling gesture input includes obtaining hand tracking data based on one or more camera frames, detecting a contact event between a first finger and a second finger based on the hand tracking data, and determining a first contact location on the first finger and a second contact location on the second finger. In accordance with a determination that the first contact location and the second contact location are within a first predefined gesture zone for a first gesture, an input action is enabled corresponding to the first gesture.

Abstract: While a view of an environment is visible via a display generation component, a computer system displays a user interface including one or more user interface objects. The computer system detects, via one or more input devices, one or more inputs; and, if the one or more inputs include a first input performed using a first input manipulator and a second input performed using a second input manipulator distinct from the first input manipulator, where the first input and the second input meet concurrency criteria, the computer system: provides a first input event for the first input to a first application and a second input event for the second input to the first application. The first input includes information identifying a target location and the second input event also includes information identifying the target location.

Abstract: Enabling gesture input includes obtaining hand tracking data based on one or more camera frames, detecting a contact event between a first finger and a second finger based on the hand tracking data, and determining a first contact location on the first finger and a second contact location on the second finger. In accordance with a determination that the first contact location and the second contact location are within a first predefined gesture zone for a first gesture, an input action is enabled corresponding to the first gesture.

Abstract: Various implementations track hand motion to interpret a scroll movement. For example, an example process may include obtaining sensor data associated with a hand via one or more sensors in a physical environment. The process may further include determining positional data corresponding to three-dimensional (3D) positions of two points on the hand based on the sensor data. The process may further include determining whether a movement of the hand includes a wrist rotation associated with the hand rotating about the wrist, an arm motion associated with movement of the arm, or a combination of the wrist rotation and the arm motion. The process may further include determining the movement of the hand as a user interaction event based on the positional data and whether the movement of the hand includes the wrist rotation, the arm motion, or a combination of the wrist rotation and the arm motion.