Abstract: Mobile virtual reality system for simulation, training or demonstration of a surgical robotic system can include a virtual reality processor. The processor can generate a virtual surgical robot and render the virtual surgical robot on a display. The virtual surgical robot can include a virtual surgical tool. A handheld user input device (UID) can sense a hand input from a hand. A foot input device can sense a foot input from a foot. The virtual reality processor can be configured to control a movement or action of the virtual surgical robot based on the hand input, and change which of the virtual surgical instruments is controlled by the one or more handheld UIDs based on the foot input. Other embodiments and aspects are disclosed and claimed.

Abstract: A surgical robotic system including a surgical table, a surgical robotic manipulator coupled to the surgical table and comprising a plurality of links coupled together by a plurality of joints that are operable to move with respect to one another to move the surgical robotic manipulator, at least one of the plurality of links or the plurality of joints having a portion that faces another of the plurality of links or the plurality of joints, a proximity sensing assembly coupled to the portion of the at least one of the plurality of links or the plurality of joints, the proximity sensing assembly operable to detect an object prior to the surgical robotic manipulator colliding with the object and to output a corresponding detection signal, and a processor operable to receive the corresponding detecting signal and cause the manipulator or the object to engage in a collision avoidance operation.

Abstract: A surgical robotic system has a tool drive coupled to a distal end of a robotic arm that has a plurality of actuators. The tool drive has a docking interface to receive a trocar. The system also includes one or more sensors that are operable to visually sense a surface feature of the trocar. One or more processors determine a position and orientation of the trocar, based on the visually sensed surface feature. In response, the processor controls the actuators to orient the docking interface to the determined orientation of the trocar and to guide the robotic arm toward the determined position of the trocar. Other aspects are also described and claimed.

Abstract: A surgical robotic system including a surgical table, a surgical robotic manipulator coupled to the surgical table and comprising a plurality of links coupled together by a plurality of joints that are operable to move with respect to one another to move the surgical robotic manipulator, at least one of the plurality of links or the plurality of joints having a portion that faces another of the plurality of links or the plurality of joints, a proximity sensing assembly coupled to the portion of the at least one of the plurality of links or the plurality of joints, the proximity sensing assembly operable to detect an object prior to the surgical robotic manipulator colliding with the object and to output a corresponding detection signal, and a processor operable to receive the corresponding detecting signal and cause the manipulator or the object to engage in a collision avoidance operation.

Abstract: An augmented reality display system used during a surgical procedure is disclosed. The system includes an imaging device to capture a real image of a surgical area during the surgical procedure. An augmented reality display presents functional overlay data associated with critical operations of a surgical instrument being actively visualized and interactions of the surgical instrument with tissue in the surgical area. The functional data is overlaid onto the real image of the surgical area. The functional data overlay is a combination of aspects of the critical operations of the surgical instrument and the interaction of the surgical instrument with the tissue in the surgical area. A processor receives functional data from the surgical instrument, determines the overlaid data related to the functional aspect of the surgical instrument, and combines the aspect of the tissue in the surgical area with the functional data received from the surgical instrument.

Abstract: An augmented reality display system used during a surgical procedure is disclosed. The augmented reality display system includes an imaging device to capture a real image of a surgical area during the surgical procedure. An augmented reality display presents an overlay of an operational aspect of a surgical instrument being actively visualized onto the real image of the surgical area. The overlay combines aspects of tissue interaction in the surgical area with functional data received from the surgical instrument. A processor receives the functional data from the surgical instrument, determines the overlay related to the operational aspect of the surgical instrument, and combines the aspect of the tissue in the surgical area with the functional data received from the surgical instrument.

Abstract: A virtual operating room (OR) is generated that includes virtual surgical equipment based on existing OR models and existing equipment models. Sensor feedback is received that defines a physical OR having physical equipment, the physical equipment including a surgical robotic system within the physical OR. The virtual OR and the virtual surgical equipment is updated based on the sensor feedback. A layout of the virtual surgical equipment is optimized in the virtual OR. The virtual OR and the virtual surgical equipment are rendered on a display.

Abstract: An autostereoscopic three-dimensional display system for surgical robotics has an autostereoscopic three-dimensional display configured to receive and display video from a surgical robotics camera, and a first sensor assembly and a second sensor assembly. A processor is configured to detect and track an eye position or a head position of a user relative to the display based on processing output data of the first sensor assembly, and to detect and track a gaze of the user based on processing output data of the second sensor assembly. The processor further is configured to modify or control an operation of the display system based on the detected gaze of the user. A spatial relationship of the display also can be automatically adjusted in relation to the user based on the detected eye or head position of the user to optimize the user's visualization of three-dimensional images on the display.

Abstract: A virtual operating room (OR) is generated that includes virtual surgical equipment based on existing OR models and existing equipment models. Sensor feedback is received that defines a physical OR having physical equipment, the physical equipment including a surgical robotic system within the physical OR. The virtual OR and the virtual surgical equipment is updated based on the sensor feedback. A layout of the virtual surgical equipment is optimized in the virtual OR. The virtual OR and the virtual surgical equipment are rendered on a display.

Abstract: Planning surgical robotic workflow with a surgical robotic system can include generating a virtual surgical environment, the virtual surgical environment including a virtual surgical robotic arm and a virtual patient. A workspace can be determined in the virtual patient. A position of a virtual tool, attached to the virtual surgical robotic arm, can be determined. A position of the virtual surgical robotic arm can be determined to maintain a reach of the virtual tool in the workspace.

Abstract: A surgical robotic system comprising: a surgical robotic arm having a plurality of robotic arm links and a plurality of joints operable to move according to multiple degrees of freedom; a proximity sensor coupled to the surgical robotic arm, the proximity sensor comprising a plurality of sensing pads operable to detect a movement of a nearby controlling object prior to contact with the surgical robotic arm; and a processor configured to determine a desired position of the surgical robotic arm based on the detected movement of the nearby controlling object and drive a movement of more than one of the plurality of robotic arm links or the plurality of joints to achieve the desired position of the surgical robotic arm.

Abstract: A system for modelling a portion of a patient includes a processing unit, a manipulator, a sensor, and a display device. The processing unit is configured to receive patient data and to process the patient data to generate a model of the portion of the patient. The sensor is configured to capture manipulator data. The processing unit is configured to receive the manipulator data from the sensor and to process the manipulator data to determine a position of the manipulator, an orientation of the manipulator, or both. The display device is configured to display the model on or in the manipulator.

Abstract: An autostereoscopic three-dimensional display system for surgical robotics has an autostereoscopic three-dimensional display configured to receive and display video from a surgical robotics camera, and a first sensor assembly and a second sensor assembly. A processor is configured to detect and track an eye position or a head position of a user relative to the display based on processing output data of the first sensor assembly, and to detect and track a gaze of the user based on processing output data of the second sensor assembly. The processor further is configured to modify or control an operation of the display system based on the detected gaze of the user. A spatial relationship of the display also can be automatically adjusted in relation to the user based on the detected eye or head position of the user to optimize the user's visualization of three-dimensional images on the display.

Abstract: Disclosed is an augmented reality (AR) headset that provides a wearer with spatial, system, and temporal contextual information of a surgical robotic system to guide the wearer in configuring, operating, or troubleshooting the surgical robotic system prior to, during, or after surgery. The spatial context information may be rendered to display spatially-fixed 3D-generated virtual models of the robotic arms, instruments, bed, and other components of the surgical robotic system that match the actual position or orientation of the surgical robotic system in the AR headset's coordinate frame. The AR headset may communicate with the surgical robotic system to receive real-time state information of the components of the surgical robotic system. The AR headset may use the real-time state information to display context-sensitive user interface information such as tips, suggestions, visual or audio cues on maneuvering the robotic arms and table to their target positions and orientations or for troubleshooting purpose.

Abstract: A control system for surgical robots based on proximity sensing, the control system including a proximity sensor coupled to a component of a surgical robot, the surgical robot component including a table, robotic arms coupled to the table, and surgical tools mounted on the robotic arms, the proximity sensor configured to sense a movement of a nearby controlling object in one or more degrees of freedom; and a processor configured to drive the component of the surgical robot to follow the movement of the controlling object.

Abstract: A virtual surgical robot being built from kinematic data is rendered to a display. A user input is received to effect a movement or a configuration of the virtual surgical robot. The kinematic data is modified based on evaluation of the movement or the configuration of the virtual surgical robot.

Abstract: A surgical exercise performed with a surgical robotic system is sensed by depth cameras, generating 3D point cloud data. Robot system data associated with the surgical robotic system is logged. Object recognition is performed on image data produced by the one or more depth cameras, to recognized objects, including surgical equipment and people, in the operating room (OR). The surgical exercise is digitized by storing the 3D point cloud data of unrecognized objects, a position and orientation associated with the recognized objects, and c) the robot system data.

Abstract: A virtual representation of an operating room is generated based on robot information and sensing of the OR with depth cameras. One of the depth cameras is integrated with a portable electronic device, operated by a local user in the operating room. The virtual representation of the OR is communicated to the virtual reality headset, with three-dimensional point cloud data. A virtual reality environment is rendered to a display of the virtual reality headset, operated by a remote user. A virtual representation of the remote user is rendered in augmented reality to a display of the portable electronic device.

Abstract: A surgical robotic system has a tool drive coupled to a distal end of a robotic arm that has a plurality of actuators. The tool drive has a docking interface to receive a trocar. One or more sensors in the docking interface sense a magnetic field generated by the trocar. One or more processors are configured to determine a position and orientation of the trocar based on the sensed magnetic field, and then drive the actuators to orient the docking interface to the determined orientation of the trocar, or otherwise guide the robotic arm toward the determined position of the trocar. Other aspects are also described and claimed.

Abstract: A virtual surgical robot being built from kinematic data is rendered to a display. A user input is received to effect a movement or a configuration of the virtual surgical robot. The kinematic data is modified based on evaluation of the movement or the configuration of the virtual surgical robot.