Abstract: A robotic system for remote operation of human-controllable equipment includes a remotely operable robotic subsystem configured to be at least one of mounted on or arranged adjacent to the human-controllable equipment to be controlled, and a remote controller that is configured to be operated by a human. The remote controller emulates a control representation of the human-controllable equipment such that a human can provide commands using the control representation. The remotely operable robotic subsystem is configured to receive commands from the remote controller to thereby operate the human-controllable equipment.

Abstract: A minimally invasive surgical system is disclosed including a processor coupled to a stereoscopic endoscope and a stereoscopic video display device. The processor generates an operative image of an anatomic structure in the surgical site, overlays the operative image onto the captured stereo video images for display on the stereoscopic video display device, generates and overlays a pointer onto the operative image or the captured stereo video images to display the pointer on the stereoscopic video display device with a three dimensional appearance, and switches between a first mode for input devices of a surgeon console used to couple motion into surgical instruments and a second mode used to control an interactive graphical user interface to allow interactions with the pointer and the operative image in three dimensions using input devices having at least three degrees of freedom.

Abstract: A computer-implemented method for displaying augmented reality (AR) content within an AR device coupled to one or more loupe lenses comprising: obtaining calibration parameters defining a magnified display portion within a display of the AR device, wherein the magnified display portion corresponds to boundaries encompassing the one or more loupe lenses; receiving the AR content for display within the AR device; and rendering the AR content within the display, wherein the rendering the AR content comprises: identifying a magnified portion of the AR content to be displayed within the magnified display portion, and rendering the magnified portion of the AR content within the magnified display portion.

Abstract: A computer-implemented method includes: receiving, by an augmented reality device, a medical image of a surgical site, generating, by the augmented reality device, a virtual surgical site model based on the medical image; presenting, by the augmented reality device, the virtual surgical site model; receiving, by the augmented reality device, user calibration input; aligning, by the augmented reality device, the virtual surgical site model with a real-life surgical site based on the user calibration input; and displaying, by the augmented reality device and after the aligning, a virtual insertion path between an incision point and a target point to aid in inserting a tool as part of performing a surgical procedure.

Abstract: A calibration platform may obtain measurements for aligning a real-world coordinate system and a display coordinate system. For example, the calibration platform may display, via an optical see-through head-mounted display (OST-HMD), a three-dimensional virtual object and receive, from a positional tracking device, information that relates to a current pose of a three-dimensional real-world object to be aligned with the three-dimensional virtual object. The calibration platform may record a three-dimensional position of a plurality of points on the three-dimensional real-world object based on the current pose of the three-dimensional real-world object, based on an indication that the plurality of points on the three-dimensional real-world object respectively corresponds with a plurality of points on the three-dimensional virtual object.

Abstract: A system includes: a phantom object; one or more sensors within the phantom object; and a processor, a computer readable memory, a non-transitory computer readable storage medium associated with a computing device in communication with the phantom object, and program instructions executable by the computing device to cause the computing device to perform operations including: detecting a medical instrument within the phantom object based on sensor data captured by the one or more sensors; measuring a distance between the medical instrument and a target point based on the sensor data; and storing or outputting information identifying the distance between the medical instrument and the target point.

Abstract: A calibration platform may display, via an optical see-through head-mounted display (OST-HMD), a virtual image having at least one feature. The calibration platform may determine, based on information relating to a gaze of a user wearing the OST-HMD, that the user performed a voluntary eye blink to indicate that the at least one feature of the virtual image appears to the user to be aligned with at least one point on the three-dimensional real-world object. The calibration platform may record an alignment measurement based on a position of the at least one point on the three-dimensional real-world object in a real-world coordinate system based on a time when the user performed the voluntary eye blink. Accordingly, the alignment measurement may be used to generate a function providing a mapping between three-dimensional points in the real-world coordinate system and corresponding points in a display space of the OST-HMD.

Abstract: A computer-implemented method includes: receiving, by an augmented reality device, a medical image of a surgical site, generating, by the augmented reality device, a virtual surgical site model based on the medical image; presenting, by the augmented reality device, the virtual surgical site model; receiving, by the augmented reality device, user calibration input; aligning, by the ugmented reality device, the virtual surgical site model with a real-life surgical site based on the user calibration input; and displaying, by the augmented reality device and after the aligning, a virtual insertion path between an incision point and a target point to aid in inserting a tool as part of performing a surgical procedure.

Abstract: A computer-implemented method for displaying augmented reality (AR) content within an AR device coupled to one or more loupe lenses comprising: obtaining calibration parameters defining a magnified display portion within a display of the AR device, wherein the magnified display portion corresponds to boundaries encompassing the one or more loupe lenses; receiving the AR content for display within the AR device; and rendering the AR content within the display, wherein the rendering the AR content comprises: identifying a magnified portion of the AR content to be displayed within the magnified display portion, and rendering the magnified portion of the AR content within the magnified display portion.

Abstract: A calibration platform may obtain measurements for aligning a real-world coordinate system and a display coordinate system. For example, the calibration platform may display, via an optical see-through head-mounted display (OST-HMD), a three-dimensional virtual object and receive, from a positional tracking device, information that relates to a current pose of a three-dimensional real-world object to be aligned with the three-dimensional virtual object. The calibration platform may record a three-dimensional position of a plurality of points on the three-dimensional real-world object based on the current pose of the three-dimensional real-world object, based on an indication that the plurality of points on the three-dimensional real-world object respectively corresponds with a plurality of points on the three-dimensional virtual object.

Abstract: A calibration platform may obtain measurements for aligning a real-world coordinate system and a display coordinate system. For example, the calibration platform may display, via an optical see-through head-mounted display (OST-HMD), a three-dimensional virtual object and receive, from a positional tracking device, information that relates to a current pose of a three-dimensional real-world object to be aligned with the three-dimensional virtual object. The calibration platform may record a three-dimensional position of a plurality of points on the three-dimensional real-world object based on the current pose of the three-dimensional real-world object, based on an indication that the plurality of points on the three-dimensional real-world object respectively corresponds with a plurality of points on the three-dimensional virtual object.

Abstract: In one embodiment of the invention, a a minimally invasive surgical system is disclosed. The system configured to capture and display camera images of a surgical site on at least one display device at a surgeon console; switch out of a following mode and into a masters-as-mice (MaM) mode; overlay a graphical user interface (GUI) including an interactive graphical object onto the camera images; and render a pointer within the camera images for user interactive control. In the following mode, the input devices of the surgeon console may couple motion into surgical instruments. In the MaM mode, the input devices interact with the GUI and interactive graphical objects. The pointer is manipulated in three dimensions by input devices having at least three degrees of freedom. Interactive graphical objects are related to physical objects in the surgical site or a function thereof and are manipulatable by the input devices.

Abstract: A device may determine a view of a user of a head mounted display. The device may obtain tracking data relating to a surgical procedure. The device may obtain, from an imaging device, surgical imaging relating to the surgical procedure. The device may orient models of objects based on the tracking data and the view of the user of the head mounted display, wherein the objects includes the imaging device. The device may augment, by providing output to the head mounted display for display, the view of the user with contextual information relating to the objects based on orienting the models based on the tracking data and the view of the user, wherein the contextual information includes the surgical imaging captured by an imaging device.

Abstract: A minimally invasive surgical system is disclosed including a processor coupled to a stereoscopic endoscope and a stereoscopic video display device. The processor generates an operative image of an anatomic structure in the surgical site, overlays the operative image onto the captured stereo video images for display on the stereoscopic video display device, generates and overlays a pointer onto the operative image or the captured stereo video images to display the pointer on the stereoscopic video display device with a three dimensional appearance, and switches between a first mode for input devices of a surgeon console used to couple motion into surgical instruments and a second mode used to control an interactive graphical user interface to allow interactions with the pointer and the operative image in three dimensions using input devices having at least three degrees of freedom.

Abstract: A calibration platform may obtain measurements for aligning a real-world coordinate system and a display coordinate system. For example, the calibration platform may display, via an optical see-through head-mounted display (OST-HMD), a three-dimensional virtual object and receive, from a positional tracking device, information that relates to a current pose of a three-dimensional real-world object to be aligned with the three-dimensional virtual object. The calibration platform may record a three-dimensional position of a plurality of points on the three-dimensional real-world object based on the current pose of the three-dimensional real-world object, based on an indication that the plurality of points on the three-dimensional real-world object respectively corresponds with a plurality of points on the three-dimensional virtual object.

Abstract: A device may determine a view of a user of a head mounted display. The device may obtain tracking data relating to a surgical procedure. The device may obtain, from an imaging device, surgical imaging relating to the surgical procedure. The device may orient models of objects based on the tracking data and the view of the user of the head mounted display, wherein the objects includes the imaging device. The device may augment, by providing output to the head mounted display for display, the view of the user with contextual information relating to the objects based on orienting the models based on the tracking data and the view of the user, wherein the contextual information includes the surgical imaging captured by an imaging device.

Abstract: A calibration platform may display, via an optical see-through head-mounted display (OST-HMD), a virtual image having at least one feature. The calibration platform may determine, based on information relating to a gaze of a user wearing the OST-HMD, that the user performed a voluntary eye blink to indicate that the at least one feature of the virtual image appears to the user to be aligned with at least one point on the three-dimensional real-world object. The calibration platform may record an alignment measurement based on a position of the at least one point on the three-dimensional real-world object in a real-world coordinate system based on a time when the user performed the voluntary eye blink. Accordingly, the alignment measurement may be used to generate a function providing a mapping between three-dimensional points in the real-world coordinate system and corresponding points in a display space of the OST-HMD.

Abstract: The present invention is directed to a method and system for photoacoustic imaging for guiding medical procedures. A transducer is placed near the site of the procedure. The optical fiber, coupled to an electromagnetic source, such as a laser, is attached to a medical device. During the procedure, the device and optical fiber are inserted into the procedure site where the optical fiber illuminates the procedure site, which has a thickness of approximately 2 mm. Photoacoustic images are acquired to visualize the procedure site as the procedure is proceeding in order to provide real-time guidance. This system is applicable to multiple surgical and interventional procedures, such as transsphenoidal surgery.

Abstract: In one embodiment of the invention, a a minimally invasive surgical system is disclosed. The system configured to capture and display camera images of a surgical site on at least one display device at a surgeon console; switch out of a following mode and into a masters-as-mice (MaM) mode; overlay a graphical user interface (GUI) including an interactive graphical object onto the camera images; and render a pointer within the camera images for user interactive control. In the following mode, the input devices of the surgeon console may couple motion into surgical instruments. In the MaM mode, the input devices interact with the GUI and interactive graphical objects. The pointer is manipulated in three dimensions by input devices having at least three degrees of freedom. Interactive graphical objects are related to physical objects in the surgical site or a function thereof and are manipulatable by the input devices.

Abstract: According to some embodiments of the present invention, a cooperatively controlled robot includes a robotic actuator assembly comprising a tool holder and a force sensor, a control system adapted to communicate with the robotic actuator assembly and the force sensor, and an output system in communication with the control system. The tool holder is configured to receive a tool to be manipulated by a user. The control system is configured to receive an instruction from a user to switch from a robot control mode into a user interface control mode. The force sensor is configured to measure at least one of a force and a torque applied to the tool, and the control system is configured to receive an indication of the at least one of a force and a torque applied to the tool and manipulate the output system based on the indication.