Abstract: A method and apparatus for using physical movement, e.g. associated with a physical therapy, for virtual gameplay control. A selection of an exercise is received from a user interface. After the selection, data associated with the exercise can be read from memory. The data includes a plurality of joints to detect and expected positions of each of the joints. A plurality of frames of a performance of a user performing the exercise are received, from sensors such as a video camera. Each of the joints within each of the frames can be detected, and a change in positions of joints can be determined. Subsequently, an indication of success that the exercise was performed correctly is sent to the user interface. A character in the game then takes an action, thus in this manner, the user can reliably control their character's in-game actions.