Abstract: Controlling an avatar in an XR environment can be based on input(s) from external, non-XR computing devices, such as mobile devices. An external device can capture sensor data, audio data, and/or image data, which are processed by an avatar control system to determine a gesture performed by the user, gauge the level of intensity or excitement of the user, and/or infer the user's sentiment or emotional state. The avatar control system can generate a motion plan based on these inferences drawn from the external device sensor data based on a baseline animation or motion template. The system can generate avatar animations that have exaggerated or understated expressions relative to the user's real-world gestures, possibly exceeding limitations of a kinematic model that otherwise constrain the avatar's animations. The technology can enable users to control an avatar in an XR environment without specialized XR equipment.