Abstract: A system for immersive physical interaction includes a hand position and orientation tracker that tracks a first hand position and a first hand orientation of a first hand of a user and tracks a second hand position and a second hand orientation of a second hand of the user; a physical criteria generator comprising a body physics simulator that simulates avatar motion and a constrained object interaction system that manages virtual object coupling; and an input translator that translates the first hand position and the first hand orientation into a first virtual hand position and a first virtual hand orientation of a user avatar in a virtual environment and translates the second hand position and the second hand orientation into a second virtual hand position and a second virtual hand orientation of the user avatar.