Abstract: Systems and methods for providing assistance for manipulating objects using virtual proxies and virtual replicas are provided. Disclosed systems enable a remote or co-located subject matter expert (SME) to provide guidance to a local user on how to manipulate physical or virtual objects in the local user's environment. The local user views an augmented reality (AR) display of the local environment. The SME views a virtual reality (VR) or AR display of the local environment. The SME manipulates virtual proxies and virtual replicas to demonstrate to the local user how physical or virtual objects in the local environment should be manipulated. In other scenarios, a local user is provided instruction by using a display from which the local user can view the local environment. Manipulations of objects are tracked and the system provides the user feedback on whether the objects are properly oriented.