Abstract: A system may track a user's motions or gestures performed in a physical space and map them to a visual representation of the user. The user's gestures may be translated to a control in a system or application space, such as to open a file or to execute a punch in a punching game. Similarly, the user's gestures may be translated to a control in the system or application space for making modifications to a visual representation. A visual representation may be a display of a virtual object or a display that maps to a target in the physical space. In another example embodiment, the system may track the target in the physical space over time and apply modifications or updates to the visual representation based on the history data.