Abstract: A non-transitory machine-readable media may store instructions that, when run by one or more processors, cause the one or more processors to generate stereo endoscopic image data of a medical field and define a medical field volume from the stereo endoscopic image data. The processors may also project a 3D representation of the medical field volume to an extended-reality display device in a remote volume. The 3D representation may include a stereoscopic image and a 3D scene generated from the stereo endoscopic image data. The processors may also generate visual guidance in the remote volume to augment the stereo endoscopic image, map the visual guidance from the remote volume to the medical field volume to generate an augmented image of the medical field volume, and project the augmented image of the medical field volume to a display device viewed by an operator of instruments in the medical field.

Abstract: A grounded virtual portal to interface with a robotic medical system is provided. The system can include an input controller coupled to a first structure via a plurality of linkages. The first structure can be mechanically coupled to a base structure. The input controller can control an object of a medical environment. A head mounted display can be coupled to a second structure extending the base structure. The head mounted display can be configured to render an extended reality environment corresponding to the medical environment. A processor can establish a grounded reference frame for the input controller to be maintained in the extended reality environment rendered via the head mounted display, and configure, based on the plurality of linkages and the grounded reference frame, a force scaler function for bidirectional force delivery between the input controller and the object of the medical environment.