Abstract: Systems and methods are disclosed for performing three-dimensional (3D) gesture based interaction with a user interface rendered from a 3D object model based upon the viewpoint of a user. Gesture based interactive systems in accordance with many embodiments of the invention utilize a spatial model of how user interface objects, such as icons and a cursor, spatially relate to one another. The user interface model can be constructed as a 3D object model. With a 3D object model, the operating system can use depth information to render a user interface display appropriate to the requirements of a specific display technology. In many embodiments, head tracking is used to determine a viewpoint from which to render a user interface display from a 3D object model maintained by the operating system and the user interface can be updated in response to the detection of 3D gestures.

Abstract: A method for operating a real-time gesture based interactive system includes: obtaining a sequence of frames of data from an acquisition system; comparing successive frames of the data for portions that change between frames; determining whether any of the portions that changed are part of an interaction medium detected in the sequence of frames of data; defining a 3D interaction zone relative to an initial position of the part of the interaction medium detected in the sequence of frames of data; tracking a movement of the interaction medium to generate a plurality of 3D positions of the interaction medium; detecting movement of the interaction medium from inside to outside the 3D interaction zone at a boundary 3D position; shifting the 3D interaction zone relative to the boundary 3D position; computing a plurality of 2D positions based on the 3D positions; and supplying the 2D positions to control an application.