Abstract: Systems and methods for providing assistance for manipulating objects using virtual proxies and virtual replicas are provided. Disclosed systems enable a remote or co-located subject matter expert (SME) to provide guidance to a local user on how to manipulate physical or virtual objects in the local user's environment. The local user views an augmented reality (AR) display of the local environment. The SME views a virtual reality (VR) or AR display of the local environment. The SME manipulates virtual proxies and virtual replicas to demonstrate to the local user how physical or virtual objects in the local environment should be manipulated. In other scenarios, a local user is provided instruction by using a display from which the local user can view the local environment. Manipulations of objects are tracked and the system provides the user feedback on whether the objects are properly oriented.

Abstract: A computer interaction system includes an augmented interaction device and a computer. The augmented interaction device includes a display device that displays augmented reality or virtual reality images, a first tracking mechanism that tracks a position of a physical object relative to the first tracking mechanism and an orientation of the physical object relative to the first tracking mechanism, and a second tracking mechanism that tracks a position of the second tracking mechanism relative to a reference and an orientation of the second tracking mechanism relative to the reference. The computer includes a processor and a memory, and processes position and orientation information received from the first tracking mechanism, and position and orientation information received from the second tracking mechanism.