METHOD AND SYSTEM FOR CONTROLLING AND DISPLAYING VIDEO STREAMS
A method performed by a video controller. The method receives a first video stream captured by an endoscope of a surgical system, and receives a second video stream that comprises surgical data. The method displays the second video stream superimposed above an area of the first video stream, and determines that the second video stream is to cease being superimposed. Responsive to determining that the second video stream is to cease being superimposed, the method continues to display the first video stream.
This application claims the benefit of and priority to U.S. Provisional Patent Application Ser. No. 63/256,514 filed Oct. 15, 2021, which is hereby incorporated by this reference in its entirety.
FIELDVarious aspects of the disclosure relate generally to surgical systems, and more specifically to a surgical system that controls and displays video streams. Other aspects are also described.
BACKGROUNDMinimally-invasive surgery, MIS, such as laparoscopic surgery, uses techniques that are intended to reduce tissue damage during a surgical procedure. Laparoscopic procedures typically call for creating a number of small incisions in the patient, e.g., in the abdomen, through which several surgical tools such as an endoscope, a blade, a grasper, and a needle, are then inserted into the patient. A gas is injected into the abdomen which insufflates the abdomen thereby providing more space around the tips of the tools, making it easier for the surgeon to see (via the endoscope) and manipulate tissue at the surgical site. MIS can be performed faster and with less surgeon fatigue using a surgical robotic system in which the surgical tools are operatively attached to the distal ends of robotic arms, and a control system actuates the arm and its attached tool. The tip of the tool will mimic the position and orientation movements of a handheld user input device (UID) as the latter is being manipulated by the surgeon. The surgical robotic system may have multiple surgical arms, one or more of which has an attached endoscope and others have attached surgical instruments for performing certain surgical actions.
Control inputs from a user (e.g., surgeon or other operator) are captured via one or more user input devices and then translated into control of the robotic system. For example, in response to user commands, a tool drive having one or more motors may actuate one or more degrees of freedom of a surgical tool when the surgical tool is positioned at the surgical site in the patient.
SUMMARYVisualization systems are often used intraoperatively for imaging tissue, performing biopsies, surgery, diagnostic, and/or other medical procedures. The term “intraoperative” as used herein refers to actions performed or events that occur during any medical procedure (invasive or non-invasive). A surgical system, such as a surgical robotic system (e.g., a system that may use one or more robotic components to perform a surgical procedure), may include a single computing device (e.g., a desktop computer) that controls various aspects of the surgical system and provides real-time analysis and feedback to operators. As an example, a computing device of a surgical system may receive control signals from an input device (e.g., in response to receiving operator input), and may control (e.g., manipulate) one or more components (e.g., in the case of a surgical robotic system, the control signals may manipulate one or more robotic arms) based on the received control signals. In addition, the computing device may provide feedback by displaying surgical information on one or more displays. For example, the computing device may be coupled to one or more surgical cameras (e.g., endoscopes) from which image data may be received and displayed during a surgical procedure. In particular, the computing device may include a graphical processing unit (GPU) that is tasked to receive the image data and render it on the display. In addition to rendering the image data, the GPU may also receive other surgical information for display along with the image data. For instance, the computing device may receive surgical information from one or more other devices of the system, such as energy and smart surgical tools, ultrasound scanner systems, etc., The computing device may also receive notifications based on the status of the other devices (e.g., based on operating room event detection) for display to the operators. As a result, the GPU may render the received information (e.g., along with the image data) in a graphical user interface (GUI) that is presented to the operators (e.g., as overlaid information on the image data captured by the one or more surgical cameras) on the one or more displays.
Having a centralized computer system, however, has several drawbacks. In situations where a computing device mixes and blends multiple streams (e.g., to overlay a GUI on one or more video streams), unexpected failures may result in no image (or a wrong image) being shown to medical personnel. Because visualization is often central to the performance of many medical procedures, failures can detrimentally impact performance of the procedure causing delays, increasing the likelihood of errors, interrupting flow, lowering device utilization, and in some cases could lead to the procedure being aborted. Thus, such a system in which an integrated GPU that receives video data from one or more endoscopes and surgical information, and blends the surgical information (e.g., as a GUI) with the video data, as described herein, has an inherent risk that any failure (e.g., blocking process) on the single computer device may potentially freeze up the display, and therefore would disable any image/surgical information updates that would otherwise be displayed on the surgical system's display. As an example, since resources (e.g., memory, processing power, etc.) are shared between surgical software applications that are being executed by the computing device during a surgical procedure, a failure or blocked process of one software application may adversely affect other software applications, including applications that are using (and sharing) the GPU. In which case, the image data from the endoscope would become unreliable (e.g., freezing, being obscured by the overlaid surgical information, etc.) during a surgical procedure when the failure occurs at (or affects) the GPU. For example, a system failure (or fault) may result in the GUI, which is overlaid on top of the endoscopic video, obscuring some or all of the critical endoscopic video. Therefore, there is a need for a video control device that controls video streams to ensure that necessary video (e.g., critical video during an intraoperative procedure) is displayed reliably and continuously.
The present disclosure provides a video control device that controls and displays video streams for a surgical system using (at least) two video controllers. Specifically, the video device includes a first video controller, such as a field-programmable gate array (FPGA) that (e.g., directly) receives clinically critical video streams (e.g., video captured by an endoscope) and a second video controller, such as a GPU that receives surgical data (e.g., a GUI that includes surgical information), which provides the surgical data as another video stream to the first video controller, where the first video controller displays (e.g., renders for display) the surgical data overlaid on top of (or superimposed above) an area of the critical video stream on a display of the surgical system. Essentially, the first video controller blends the video stream from the GPU with the critical video stream from the endoscope and passes through the blended video stream for display, such that the surgical data is superimposed above a (e.g., predefined) area of the endoscopic video. If a failure were to occur, resulting the video stream from the GPU no longer being superimposed over the area (e.g., but instead would result in the video stream from the GPU covering a larger area of (e.g., adversely obscuring) the critical endoscopic video), the video control device may blend the endoscopic video with the blended video (e.g., producing another blended video stream) in order to superimpose the endoscopic video over the faulty blended stream. As a result, any potential issue that may occur with the GPU (and/or other resources within the surgical system, e.g., due to a blocking process) would not prohibit the clinically critical video streams from being displayed (and/or would not obscure the critical video streams). Thus, this greatly reduces (or eliminates) the issue of the video stream from the endoscope freezing and/or being obscured from an operator's view, thereby increasing viewing reliability of the video stream from the system's camera(s) (e.g., when the surgical system experiences performance issues or a failure (or fault)).
The present disclosure provides a surgical system that controls and displays video streams. Specifically, the system may include a video controller that receives a first video stream captured by a camera (e.g., an endoscope) of the surgical system and receives a second video stream that includes surgical data (e.g., surgical overlay information). The video controller displays the second video stream superimposed above a portion (e.g., an area) of the first video stream, determines that the second video stream is no longer being superimposed above the area of the first video stream (e.g., the second video stream may obscure a larger area of the first video stream), and, responsive to determining that the second video stream is no longer being superimposed above the area of the first video stream, continue to display the first video stream. In particular, the video controller may superimpose the first video stream above (at least a portion of) the second video stream, such that a user may continue to view the critical video stream from the endoscope without interruption.
In one aspect, the surgical data includes a graphical user interface (GUI) that includes at least one of a notification associated with a surgical procedure, image data image data captured by one or more cameras of the surgical system and a user interface (UI) item for allowing a user to interact with the GUI through a (e.g., touch sensitive) display. In another aspect, a display includes several of lines of pixels, wherein displaying the second video stream superimposed above an area of the first video stream includes: producing a blended video stream by blending the first video stream with the second video stream; storing, in a line buffer, selected pixels for a set of lines of the several of lines based on the blended video stream; and providing, from the line buffer, the stored selected pixels to the display.
In one aspect, the area is a first area, wherein, while the second video stream is superimposed above the area of the first video stream, the surgical system is in a blended mode in which a blended video stream of the first and second video streams is displayed, wherein the method further comprises presenting a notification that provides a recommendation for a user of the surgical system to switch to a failover mode in which the first video stream is superimposed above a second area of the blended video stream, wherein the first video stream is continued to be displayed responsive to receiving user input via a user input device for switching from the blended mode to the failover mode.
In one aspect, determining that the second video stream is no longer being superimposed includes determining that the area of the first video stream over which the second video stream is displayed exceeds a threshold area. In another aspect, determining that the second video stream is no longer being superimposed is based on a content analysis of the second video stream.
In one aspect, the video controller is a first video controller, where the second video stream is received from a second video controller of the surgical system. In some aspects, the first video controller is a field-programmable gate array (FPGA), and the second video controller is a graphics processing unit (GPU). In one aspect, the FPGA draws power from a first power supply, and the GPU draws power from a second power supply of the surgical system.
In one aspect, displaying the second video stream superimposed above an area of the first video stream comprises providing a first blended video stream that includes the first and second video streams to a display, wherein continuing to display the first video stream comprises producing a second blended video stream by retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set video frames with a second set of one or more video frames of the first blended video stream; and providing the second blended video stream to a display
In one aspect, the second video stream and the first video stream are displayed on a display of the surgical system, the display includes a plurality of lines of pixels (e.g., that extend along a length of the display), where the displaying includes: storing, in a line buffer, selected pixels for a set of lines of the plurality of lines based on a composite (or blended) video stream including the first video stream and the second video stream; and providing, from the line buffer, the stored selected pixels to the display.
The above summary does not include an exhaustive list of all aspects of the disclosure. It is contemplated that the disclosure includes all systems and methods that can be practiced from all suitable combinations of the various aspects summarized above, as well as those disclosed in the Detailed Description below and particularly pointed out in the claims. Such combinations may have particular advantages not specifically recited in the above summary.
The aspects are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” or “one” aspect of this disclosure are not necessarily to the same aspect, and they mean at least one. Also, in the interest of conciseness and reducing the total number of figures, a given figure may be used to illustrate the features of more than one aspect, and not all elements in the figure may be required for a given aspect.
Several aspects of the disclosure with reference to the appended drawings are now explained. Whenever the shapes, relative positions and other aspects of the parts described in a given aspect are not explicitly defined, the scope of the disclosure here is not limited only to the parts shown, which are meant merely for the purpose of illustration. Also, while numerous details are set forth, it is understood that some aspects may be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description. Furthermore, unless the meaning is clearly to the contrary, all ranges set forth herein are deemed to be inclusive of each range's endpoints.
Each surgical tool 7 may be manipulated manually, robotically, or both, during the surgery. For example, the surgical tool 7 may be a tool used to enter, view, or manipulate an internal anatomy of the patient 6. In an aspect, the surgical tool 7 is a grasper that can grasp tissue of the patient. The surgical tool 7 may be controlled manually, by a bedside operator 8; or it may be controlled robotically, via actuated movement of the surgical robotic arm 4 to which it is attached. For example, when manually controlled an operator may (e.g., physically) hold a portion of the tool (e.g., a handle), and may manually control the tool by moving the handle and/or pressing one or more input controls (e.g., buttons) on the (e.g., handle of the) tool. In another aspect, when controlled robotically, the surgical system may manipulate the surgical tool based user input (e.g., received via the user console 2, as described herein).
Generally, a remote operator 9, such as a surgeon or other operator, may use the user console 2 to remotely manipulate the arms 4 and/or the attached surgical tools 7, e.g., during a teleoperation. The user console 2 may be located in the same operating room as the rest of the system 1, as shown in
In another aspect, the display 15 may be configured to display at last one graphical user interface (GUI) that may provide informative and/or interactive content, to thereby assist a user in performing a surgical procedure with one or more instruments in the surgical system 1. For example, some of the content displayed may include image data captured by one or more endoscopic cameras, as described herein. In another aspect, the GUI may include selectable UI items, which when manipulated by the user may cause the system to perform one or more operations. For instance, the GUI may include a UI item as interactive content to switch control between robotic arms. In one aspect, to interact with the GUI, the system may include input devices, such as a keyboard, a mouse, etc. In another aspect, the user may interact with the GUI using the UID 14. For instance, the user may manipulate the UID to navigate through the GUI, (e.g., with a cursor), and to make a selection, may hover the cursor over a UI item and manipulate the UID (e.g., selecting a control or button). In some aspects, the display may be a touch-sensitive display screen. In this case, the user may perform a selection by navigating and selecting through touching the display. In some aspects, any method may be used to navigate and/or select a UI item.
In some aspects, the surgical system may be capable of displaying an augmented view, which may comprise, for example, an endoscopic view (based on the video stream captured by an endoscope) augmented with one or more additional overlays, on display 15. As examples, the overlays may be simple (such as a toolkit for instrument use), or complex (such as a fusion of pre-operative images, which may be displayed in addition to toolkits, etc.). As another example, the overlays may include one or more GUIs that is superimposed on one or more areas of the image data captured by one or more cameras (e.g., endoscopes). More about this augmented view is described herein.
As shown, the remote operator 9 is sitting in the seat 10 and viewing the user display 15 while manipulating a foot-operated control 13 and a handheld UID 14 in order to remotely control one or more of the arms 4 and the surgical tools 7 (that are mounted on the distal ends of the arms 4.)
In some variations, the bedside operator 8 may also operate the system 1 in an “over the bed” mode, in which the beside operator 8 (user) is now at a side of the patient 6 and is simultaneously manipulating a robotically-driven tool (end effector as attached to the arm 4), e.g., with a handheld UID 14 held in one hand, and a manual laparoscopic tool. For example, the bedside operator's left hand may be manipulating the handheld UID to control a robotic component, while the bedside operator's right hand may be manipulating a manual laparoscopic tool. Thus, in these variations, the bedside operator 8 may perform both robotic-assisted minimally invasive surgery and manual laparoscopic surgery on the patient 6.
During an example procedure (surgery), the patient 6 is prepped and draped in a sterile fashion to achieve anesthesia. Initial access to the surgical site may be performed manually while the arms of the system 1 are in a stowed configuration or withdrawn configuration (to facilitate access to the surgical site.) Once access is completed, initial positioning or preparation of the system 1 including its arms 4 may be performed. Next, the surgery proceeds with the remote operator 9 at the user console 2 utilizing the foot-operated controls 13 and the UIDs 14 to manipulate the various end effectors and perhaps an imaging system, to perform the surgery. Manual assistance may also be provided at the procedure bed or table, by sterile-gowned bedside personnel, e.g., the bedside operator 8 who may perform tasks such as retracting tissues, performing manual repositioning, and tool exchange upon one or more of the robotic arms 4. Non-sterile personnel may also be present to assist the remote operator 9 at the user console 2. When the procedure or surgery is completed, the system 1 and the user console 2 may be configured or set in a state to facilitate post-operative procedures such as cleaning or sterilization and healthcare record entry or printout via the user console 2.
In one aspect, the remote operator 9 holds and moves the UID 14 to provide an input command to drive (move) one or more robotic arm actuators 17 (or driving mechanism) in the system 1 for teleoperation. The UID 14 may be communicatively coupled to the rest of the system 1, e.g., via a console computer system 16 (or host). The UID 14 can generate spatial state signals corresponding to movement of the UID 14, e.g., position and orientation of the handheld housing of the UID, and the spatial state signals may be input signals to control motions of the robotic arm actuators 17. The system 1 may use control signals derived from the spatial state signals, to control proportional motion of the actuators 17. In one aspect, a console processor of the console computer system 16 receives the spatial state signals and generates the corresponding control signals. Based on these control signals, which control how the actuators 17 are energized to drive a segment or link of the arm 4, the movement of a corresponding surgical tool that is attached to the arm may mimic the movement of the UID 14. Similarly, interaction between the remote operator 9 and the UID 14 can generate for example a grip control signal that causes a jaw of a grasper of the surgical tool 7 to close and grip the tissue of patient 6.
The system 1 may include several UIDs 14, where respective control signals are generated for each UID that control the actuators and the surgical tool (end effector) of a respective arm 4. For example, the remote operator 9 may move a first UID 14 to control the motion of an actuator 17 that is in a left robotic arm, where the actuator responds by moving linkages, gears, etc., in that arm 4. Similarly, movement of a second UID 14 by the remote operator 9 controls the motion of another actuator 17, which in turn drives other linkages, gears, etc., of the system 1. The system 1 may include a right arm 4 that is secured to the bed or table to the right side of the patient, and a left arm 4 that is at the left side of the patient. An actuator 17 may include one or more motors that are controlled so that they drive the rotation of a joint of the arm 4, to for example change, relative to the patient, an orientation of an endoscope or a grasper of the surgical tool 7 that is attached to that arm. Motion of several actuators 17 in the same arm 4 can be controlled by the spatial state signals generated from a particular UID 14. The UIDs 14 can also control motion of respective surgical tool graspers. For example, each UID 14 can generate a respective grip signal to control motion of an actuator, e.g., a linear actuator that opens or closes jaws of the grasper at a distal end of surgical tool 7 to grip tissue within patient 6.
In some aspects, the communication between the surgical robotic table 5 and the user console 2 may be through a control tower 3, which may translate user commands that are received from the user console 2 (and more particularly from the console computer system 16) into robotic control commands that transmitted to the arms 4 on the surgical table 5. The control tower 3 may also transmit status and feedback from the surgical table 5 back to the user console 2. The communication connections between the surgical table 5, the user console 2, and the control tower 3 may be via wired (e.g., optical fiber) and/or wireless links, using any suitable one of a variety of wireless data communication protocols, such as BLUETOOTH protocol. Any wired connections may be optionally built into the floor and/or walls or ceiling of the operating room. The system 1 may provide video output to one or more displays, including displays within the operating room as well as remote displays that are accessible via the Internet or other networks. The video output or feed may also be encrypted to ensure privacy and all or portions of the video output may be saved to a server or electronic healthcare record system.
The input device 19 may be any electronic device that is configured to determine (generate or produce) digital (e.g., surgical) data that has (e.g., surgical overlay information) associated with (relating to) the surgical system, and is configured to provide the surgical data to the video control device 20. For example, the input device 19 may be configured to produce surgical data that relates to a teleoperation that is being performed by an operator (e.g., operator 9 of
In another aspect, the input device may be a device of a surgical robotic system, such as a robotic arm 4 that is being manipulated by an operator, where the surgical data may indicate a status (e.g., a position) of the robotic arm. In another aspect, the input device may be any computing device of the surgical system 1 of
In one aspect, the user input device 21 may be any electronic device that is configured to receive user input and provide one or more control signals (generated by the user input device based on user input) to another electronic device to the video control device 20. For instance, the user input device may be a peripheral computer device, such as a keyboard or a mouse. In another aspect, the device may be (e.g., an electronic device with) a touch-sensitive display screen, which may display a GUI with one or more user interface (UI) items, where the device 21 may produce one or more control signals based on a user touching a portion of the display screen that is presenting a UI item. For instance, the user device 21 may be a tablet computer, a laptop computer, or a smart phone.
In one aspect, the user input device may be a part of the video control device 20. For instance, the user input device may be (or include) one or more (e.g., physical) buttons (e.g., toggle buttons, push buttons, etc.) that are disposed on the (e.g., housing 39 of the) video control device 20. In another aspect, the user input device may be a touch-screen (not shown) that is a part of the video control device.
The video control device 20 is an electronic device that is configured to control and display one or more video streams on the display 22. As shown, the video control device is communicatively coupled one or more other electronic devices of the surgical system in order to exchange digital data, such as video streams (e.g., video data). In particular, the video device is coupled to the camera 18, the input device 19, the user input device 21, and the display 22. In one aspect, the video control device may be configured to establish a wireless connection with one or more of the devices via a wireless communication protocol (e.g., BLUETOOTH protocol or any other wireless communication protocol). During the established wireless connection, the video control device may exchange (e.g., transmit and/or receive) data packets (e.g., Internet Protocol (IP) packets) with one or more of the other devices, which may include video data in any video format.
In another aspect, the video control device may communicatively couple with one or more electronic devices, via other methods. For example, the video control device may couple via a wired connection (e.g., a High-Definition Multimedia Interface (HDMI) connection, etc.) with the camera 18 to receive image data (as a video stream) that is captured by the camera, and may couple to the display 22 via another wired connection (e.g., another HDMI connection) in order to provide one or more video streams to the display to be displayed. In one aspect, the video controller may include one or more input connections (e.g., for coupling to the camera 18 , to the input device 19, and to the user input device 21), and one or more output connectors (e.g., the display 22) for (e.g., removably) coupling to one or more electronic devices. For example, the video control device may include one or more video input connectors and video output connectors (e.g., HDMI connectors, Digital Video Interface (DVI), Serial Digital Interface (SDI) connectors, composite video connectors) for coupling to the camera and to the display, respectively.
The video control device 20 includes a housing 39, a first video controller 23, a second video controller 24, a first power supply 38, and a second power supply 37. The first power supply 38 is configured to provide power to the first video controller 23 and the second power supply 37 is configured to provide power to the second video controller 24. In one aspect, both power supplies are configured to draw power from a power source (e.g., an external power source, such as an AC mains, one or more internal batteries, etc.), and each of the video controllers is configured to draw power from their respective power supplies (which may be configured to condition the input power drawn from the external power source for use by their respective controllers). In one aspect, the first power supply 38 may be arranged to only provide power to the first video controller, while the second power supply 37 may be arranged to power the second video controller and one or more other components of the video control device, such as a motherboard (not shown) to which the second video controller may be coupled. By coupling only, the first video controller to the first power supply 38 ensures that the controller 23 is always provided sufficient power to execute one or more video processing operations, as described herein. In another aspect, both controllers may couple to both power supplies, where only one power supply provides power at any given time. As a result, the other power supply may be a redundant power supply (e.g., used in cases in which the other power supply is not operational). In another aspect, the video control device may only include one power supply (e.g., only the first power supply 38), which is arranged to provide power to each (or at least some) of the components of the control device.
In one aspect, each of the components of the video control device 20 may be disposed (e.g., housed) within the housing 39. In which case, the video control device may be designed such that it may be integrated within an existing surgical system. For example, in an existing surgical system, the display may couple (e.g., directly) to a computing device, which is coupled to one or more cameras. In which case, the computing device receives video from the camera and provides (e.g., using an onboard GPU) image data (which may include surgical information overlaid on the image data) to the display. In one aspect, the video control device may be designed to be disposed (e.g., removably coupled) between 1) one or more cameras and/or a computing device (e.g., the input device 19) and 2) one or more displays, as shown herein, in order to control video streams provided to the display 22. In another aspect, the video control device may be designed to be removably received within another computing device, such as the control tower 3. More about the video control device controlling video streams is described herein.
As shown herein, the video control device includes the two video controllers 23 and 24. In one aspect, either of the controllers may be a special-purpose processor such as an application-specific integrated circuit (ASIC), a general purpose microprocessor, a field-programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and dedicated state machines). In particular, the first video controller 23 may be a FPGA and the second video controller 24 may be a GPU. In one aspect, each of the controllers may be coupled to a same printed circuit board (PCB), such as a motherboard (not shown) of the video control device, e.g., via a physical interface (such as peripheral component interconnect express (PCIe)). In another aspect, each of the controllers may be coupled (or a part of) separate PCBs disposed in the housing 39 of the video control device. In some aspects, the first video controller may be a FPGA that includes onboard software (e.g., firmware) stored within memory (e.g., memory 27) of the FPGA, which when used to program (configure) hardware elements (e.g., configurable logic blocks (CLBs)) of the FPGA, perform one or more operations to control one or more video streams that are being received and display the one or more video streams, as described herein. In one aspect, the FPGA may be a stand-alone hardware-based solution (e.g., with respect to other components of the surgical system, such as the second video controller 24 of the video device 20) such that operations performed by the FPGA use only resources (e.g., logic gates) that are a part of the FPGA, and the FPGA may not share resources with other electronic components of the surgical system (and/or may not perform operations other than those pre-programmed into the FPGA). In which case, other operations performed by the surgical system may not be performed by the FPGA. Thus, the FPGA may be a more reliable solution to ensure that critical video streams (e.g., video streams received from the endoscope 18) are less susceptible to blocking processes that may occur due to other operations performed by other components (e.g., the second video controller 24) of the surgical system.
The second video controller 24 is configured to receive surgical data from one or more input devices 19, and is configured to produce a (e.g., second) video stream that comprises the surgical data (e.g., produces one or more video streams, each having at least some of the received surgical data). In one aspect, the second video controller may receive the data and produce a graphical user interface (GUI) that includes at least some of the surgical data (e.g., as one or more video frames). For example, the second video controller may aggregate and arrange at least some of the surgical data as (e.g., portions of) the GUI. In one aspect, the second video controller may arrange the data based on a predefined layout (or may be a user-defined layout). For example, the surgical data may include one or more notifications associated with a surgical procedure that is being performed by one or more operators of the surgical system. Such notifications may indicate a status of a patient, status updates of one or more electronic devices (e.g., a surgical tool of a robotic arm of the surgical system), etc., and/or information relating to the surgical procedure. In another aspect, the GUI may include one or more user interface (UI) items for interacting with the GUI through a display (e.g., through touch, when the display is a touch-sensitive display, and/or through one or more peripherals that are coupled to the video control device, such as a keyboard). For instance, a UI item may control what the GUI displays and/or may control the surgical system. For example, the (e.g., one or more processors of the) surgical system may be executing one or more software application programs. In which case, the GUI may include UI items associated with those programs, where upon user interaction may cause the programs to perform one or more operations. In another aspect, the second video controller may receive one or more video streams from the input device. For instance, the input device 19 may include one or more cameras. In which case, the GUI may include captured image data of the cameras. In another aspect, the second video controller 24 may retrieve data from memory (e.g., of the video control device), for rendering into one or more video streams.
With this surgical data, the second controller may create a layout of the GUI that includes the data, which the second controller transmits as a video stream (as one or more video frames, as described herein) to the first video controller 23. In some aspects, the second video controller may be configured to passthrough one or more video streams from one or more input devices to the first video controller 23. As described herein, the first video controller may receive the video stream from the second video controller and may produce a blended (or composite) video stream that includes the video stream from the second video controller superimposed above image data captured by one or more cameras (e.g., camera 18) for display one or more displays such that an operator of the surgical system may view surgical data contemporaneously with critical video content captured by one or more cameras 18. More about producing and displaying the composite video stream is described herein.
The first controller 23 includes several operational blocks, such as a system monitor 25, memory 27, a blending block 70, and a failover blending block 72. Examples of memory (e.g., non-transitory machine-readable storage medium) may include read-only memory, random-access memory, CD-ROMS, DVDs, magnetic tape, optical data storage devices, flash memory devices, and phase change memory. As described herein, the first video controller 23 may operate in one of one or more operational modes for blending and displaying video content from one or more video sources. For example, the first video controller may operate in a blended (video) mode in which video (e.g., frames) received from the second video controller are blended with video captured by the camera 18 to produce a blended (composite) video stream, which when displayed on the display 22 provides an operator with an overlay view of the surgical data on top of the video captured by the camera 18.
The first video controller 23 is configured to receive a (e.g., first) video stream (or one or more video streams) captured by (e.g., from) the camera 18 (of the surgical system), and is configured to receive another (e.g., second) video stream (or one or more video streams), which includes at least some surgical data from the input device 19, from the second video controller 24. In one aspect, the received video streams may be encoded using any video codec. For instance, the first video stream may be encoded by the camera 18 using H.264 codec, where the first video controller 23 may be configured to decode the video stream into (e.g., uncompressed) video data in any video format (e.g., in a RGBα signal format). In one aspect, the second video stream may be in a same video format as the first video stream. In some aspects, video stream(s) received from the camera(s) 18 and/or second video controller 24 may be high definition (HD) video that may include 10-bit 4K video such as, for example, of resolution 3840×2160 pixels (which is also referred to as 2160 p), 1920×1080 pixels (also referred to as 1080 p video), and 1280×720 pixels (also referred to as 720 p video) with a frame rate of 59.94 and/or 60 image frames per second (fps). In another aspect, the second video stream may be in an uncompressed video format (e.g., the video format that the first video controller 23 decodes the first video stream from the camera 18).
A description of the first video controller operating in the blended mode will now be described. In one aspect, the blending block 70 is configured to receive the (e.g., first) video stream from the camera 18 (which includes video content of a surgical site during a surgical procedure, for example) and the (e.g., second) video stream that includes surgical data, and produces a blended video stream by blending the first video stream with the second video stream. In one aspect, the second video stream produced by the second video controller may be graphics frames (e.g., which may be an uncompressed video stream) that allows the blending block 70 to produce a blended video stream of several video streams. For instance, the video stream of the second video controller 24 may be an RGBα stream, where “RGB” stands for the three-color channels (red, green, and blue) per pixel of the stream's graphics frames and “α” is the alpha (α)-channel representing the degree of transparency associated with each pixel of the frame. In one aspect, the a-channel may be separate or pre-multiplied. In one aspect, the a-channel allows the blending block to alpha blend (e.g., portions) of one video stream to another video stream. In particular, the blending block uses this channel to superimpose portions of the surgical data of the second video stream as an overlay stream above (or on top of) another video stream (e.g., the first video stream of the camera 18), by having values of the α-channel for individual pixels between 0.0, for fully transparent and 1.0, for fully opaque.
In particular, the blending block is configured to receive the first video stream from the camera 18 and the second video stream from the second video controller 24, and is configured to blend (e.g., at least a portion of) the second video stream with the first video stream. For example, the blending block may perform alpha blending, where the blending block produces a blended video stream, c, by blending (e.g., video frames of) the two streams together. In particular, the blended video stream may be as follows
c=αf+(1−α)b
where b represents (e.g., one or more video frames of) the first video stream being in the background, while f represents (e.g., one or more video frames of) the second video stream being in the foreground. In one aspect, the above-mentioned alpha blending equation assumes that gamma, γ, is 1 for the color space of each of the RBG color channels in the video streams video stream. Thus, with alpha blending, the blending block may produce a blended video stream by combining pixels (based on the alpha value) of the second video stream with corresponding pixels of the first video stream in order to either (e.g., on a per-pixel basis) 1) (at least partially) bring second video stream into the foreground or 2) display only the first video stream. For example, the blending block may alpha blend a first video frame of the first video stream with a (e.g., corresponding) second video frame of the second video stream, such that at least a portion the second video frame is superimposed above an area of the first video frame. For instance, a pixel of the second video frame that has an alpha value of 0.0 are transparent, meaning when the blended video is displayed the pixel values of the first video frame are shown. If, however, the alpha value is greater than 0.0, such as 1.0, the pixel value of the second video frame would be shown, due to that pixel being completely opaque. Thus, to superimpose a GUI within a video frame of the second video stream above a video frame of the first video stream, each pixel associated with the GUI may include an alpha value greater than 0.0, whereas portions of the video frame that do not have the GUI may have an alpha value of 0.0. In some aspects, the second video controller 24 may be configured to define the alpha values of the second video stream.
As described herein, the first video controller may be configured to “pass through” the blended video stream to the display 22. In particular, the (e.g., operational blocks of the) first video controller 23 may perform very little (to no) digital signal processing operations upon the received (e.g., first and/or second) video streams and/or the blended video stream(s), such that the first video controller acts like a cable (e.g., a circuit trace) that (e.g., electronically connects) the camera 18 to the display 22. In one aspect, to pass through video streams, the first video controller may have very little latency (e.g., compared to conventional video controller, such as a GPU), where latency can be defined as the added delay (or amount of time) between a first moment in which a video frame is captured by the camera 18 and a second (subsequent) moment that the frame is output by the display 22. In one aspect, to reduce latency, the first video controller may be configured to not store the blended video stream for an extended period of time (e.g., no longer than is necessary to output the video stream within a threshold amount of time from when the stream is received from the camera 18).
To reduce latency, the first video controller 23 buffers the blended video stream produced by the blending block 70 on a line-to-line basis in a line buffer 28, rather than a frame-to-frame basis in a frame buffer that would otherwise be used by a conventional GPU. For example, considering a framerate of the blended video stream (which may be the same for both the first and second video streams), e.g., 60 frames per second (FPS), the buffering of an entire video frame (e.g., into memory 27 of the first video controller 23) would add a minimum latency, LFB_min, from input (e.g., into the controller 23) to output (e.g., from the controller 23), of
In one aspect, the first video controller 23 may use the line buffer 28 instead of a frame buffer to write/read the video streams in order to significantly reduce latency caused by having to otherwise write and read an entire video frame from memory. In particular, the blending block 70 provides (e.g., at least a portion of a video frame of) a (e.g., blended) video stream into the line buffer 28. Specifically, the first video controller 23 writes one or more lines of the video stream into the line buffer at any given time (e.g., where each line includes one or more pixel values for one or more pixels of the display 22 that make up the line). For example, to display video on the display 22, the first video controller 23 may store, in the line buffer 28 selected pixels (e.g., pixel values, such as RGB values) for a number of lines of a total number of lines of pixels of the display based on the blended video stream. In one aspect, the number of lines written into the line buffer may be less than a number of lines that would otherwise make up an entire video frame of the blended video stream that may be rendered by the display 22. Once a number of lines are written, the first video controller 23 reads the stored lines and provides the lines to the display 22 for display. In one aspect, the blended video stream may be downscaled and placed into an appropriately sized line buffer (e.g., based on the blended video stream and/or the display). In particular, the first video controller may provide the stored selected pixels in the line buffer 28 to the display 22 for display (e.g., rendering). The first video controller may repeat these operations in order to fill up the pixel count of the display 22 with one frame, and then start over to add a new frame (e.g., line-by-line or groups-of-lines (e.g., two or more lines) by groups-of-lines, as described herein) to the display 22.
As described herein, the use of a line buffer 28 may drastically reduce latency in order to allow the first video controller to pass through the blended video stream from the video sources to the display 22. For example, considering a frame rate of 60 fps and a resolution of 1080 p (e.g., 1920 horizontal columns×1080 vertical lines), buffering, e.g., eight lines of a video frame would add a minimum latency, LLB_min, from input to output, of
In one aspect, the process of alpha blending the second video stream to the first video stream adds little (to no latency) for processing the video streams by the first video controller. In some aspects, once the blending block produces a blended video frame, it may provide (at least a portion of) the blended frame to the line buffer 28 for display. In some aspects, the first video controller 23 may perform one or more other video signal processing operations upon the blended video stream.
In one aspect, the second video stream may be superimposed above a predefined area (e.g., of pixels) that would otherwise display a portion of the first video stream. In one aspect, the area over which the second video stream is displayed may be reconfigurable (e.g., user configurable). For instance, the surgical system may be configured to receive user input indicating how big (or small) of an area that is to include the second video stream.
As described herein, the first video controller 23 may be configured to operate in a blended mode in which video streams from one or more cameras (e.g., a critical video captured by an endoscope 18) is blended with one or more video streams produced by the second video controller. In another aspect, the first video controller may be configured to operate in a “failover” mode, in which based on a determination that the surgical system includes a fault (e.g., based on a content analysis of the blended video stream, the second video stream produced by the second video controller, etc.) the first video controller may be configured to blend (e.g., one or more video frames of) the first video stream of the camera 18 with the (e.g., first) blended video stream produced by the blending block 70 to produce another (e.g., second) blended video stream for output to the display. As a result, the video control device 20 may continue to pass through the first video stream, even when a system fault may adversely affect the blending being performed by the blending block 70. More about these system faults adversely affecting the blended video stream is described herein.
In one aspect, the system monitor 25 is configured to determine which mode the first video controller is to operate. In particular, the system monitor determines whether the second video stream is no longer being superimposed above an area of the first video stream. Specifically, the system monitor may determine whether the blended stream does not include the appropriate video content of the second video stream (e.g., has a solid area of the same (or similar) pixel values) and/or may determine that the area that the second video stream is overlaid has grown above a threshold (thereby not allowing the operator to view video of the camera 18). In one aspect, the system monitor may make these determinations based on at least some received data. For example, the system monitor 25 is configured to receive data, which the monitor may use to determine whether the video control device is to continue to operate in one of the modes or is to switch between modes (e.g., while the video control device is displaying video on the display 22). For example, while operating in the blended mode, the monitor 25 may use (at least a portion) of the received data to determine whether to switch to the failover mode, and thus continue to display the first video stream (e.g., by alpha blending one or more video frames of the first video stream with one or more video frames of the blended video stream).
In one aspect, the system monitor 25 may be configured to receive user input (data) from the user input device 21, and based on the user input, determine which mode the surgical system 1 is to operate. For example, the user input device may be one or more physical buttons, each button indicating a particular mode in which the system is to operate. In another aspect, the input device may be one button with one or more stages, where each stage is user-configurable (e.g., by depressing the switch), and each stage indicates the mode the system is to operate. Once the user input device receives user input, the device may transmit a control signal to the system monitor 25 that may indicate a particular mode.
In another aspect, the system monitor 25 may be configured to receive system data, which indicates a status of the surgical system. For example, the system data may indicate whether an electronic device (e.g., the input device 19) of the surgical system 1 has a blocked process, or whether one or more operations that are being performed by the surgical system has failed to be performed, which may adversely affect the surgical data (e.g., freeze the data) that the video control device 20 is receiving. In another aspect, the system data may indicate an allocation of computational resources (e.g., hardware, such as memory and one or more processors) that are being used by the surgical system to perform computational operations. More about the system data is described herein. In some aspects, the system monitor 25 may be configured to determine whether to switch between modes based on an image analysis of the second video stream that is being received from the second video controller 24. In particular, the system monitor determines whether there is an issue with the second video stream (e.g., whether the images within the stream have frozen, indicating that the surgical system has one or more blocked processes), and based on that may determine whether to switch modes (e.g., switch from the blended mode to the failover mode). In another aspect, the content analysis may be performed upon the blended video stream produced by the blending block 70, as described herein. More about performing a content analysis is described herein.
Based on the received data, the system monitor 25 determines whether to switch modes, and in response, may produce a control signal that is sent to the failover blending block 72. The operations of switching from the blending mode to the failover mode will now be described. In response to receiving a control signal, the failover blending block 72 may be configured to cause the video control device to operate in the mode indicated by the control signal. For example, to operate in the blending mode, the failover blending block may receive a control signal such that it does not perform any blending operations, and therefore the first video controller draws (e.g., lines of) the blended video stream produced by the blending block 70 to display video of the camera 18 and/or input device 19. Upon determining that the first video controller 23 is to switch to the failover mode, the system monitor 25 transmits a control signal to the failover blending block 72, which produces a failover blended video stream for display by the display 22 (e.g., in lieu of the blended video stream produced by the blending block 70). For example, the frame buffer 71 may be configured to receive one or more video frames of the first video stream (e.g., capable of holding several video frames for a period of time). Upon receiving a control signal, the failover blending block 72 may be configured to produce a failover (e.g., second) blended video stream by retrieving one or more video frames (of the one or more video streams received from the one or more cameras 18) from the frame buffer 71, and blending at least one of the retrieved video frames with one or more video frames of the (e.g., first) blended video stream produced by the blending block 70. As a result, the produced failover blended video stream by the failover blending block 72 may include (at least) three layers of video content: a first (background) layer that includes the first video stream from the camera 18, a second (middle) layer that may include video content of the second video stream from the second video controller 24 (and/or may include faulty video content, as described herein), and 3) a third (top) layer that includes the first video stream from the camera 18. In one aspect, the first video stream from the frame buffer 71 may be superimposed above an area (less than a total area) of the first blended video stream from the blending block 70. The first video controller may be configured to provide the failover blended video stream to the display 22 in lieu of the first blended video stream (e.g., from the line buffer 28).
In one aspect, the frame buffer 71 may be configured to continuously receive video frames of the first video stream while the video control device operates in the blending mode. This may allow the video content device to seamlessly transition between the blending mode and the failover mode. For example, upon detecting a fault within the blended video stream, the failover blending block 72 may draw a video frame of the first video stream from the frame buffer at a moment in time at which the fault is detected. Thus, the video control device may automatically (e.g., without user intervention) and transparently transition between the two modes.
In one aspect, the blended mode may be a default mode of the video control device 20. For instance, once the surgical system is booted up (turned on), and the first video controller begins to receive the first video stream and the second video stream, the video control device 20 may (e.g., immediately) begin displaying the blended video stream.
As described thus far, the video control device 20 may be configured to switch from the blended mode to the failover mode based on the system monitor 25 monitoring data of the surgical system. In another aspect, the video control device may be configured to switch from the failover mode back to the blended mode based on one or more criteria, as described herein. For example, upon determining that the system no longer includes a blocking process (e.g., based on system data received by the system monitor), the video control device may begin displaying the blended video stream from the line buffer 28. In which case, the system monitor may transmit a control signal to the failover blending block 72 to cease blending and outputting blended video stream.
Regarding
The first video controller 20 receives a second video stream that includes surgical data (at block 33). For instance, the second video controller 24 may receive the surgical data (e.g., as text, image data, and/or video data), and may produce the second video stream that includes the data, and provide the stream to the first video controller. The first video controller displays the second video stream superimposed over an area of the first video stream (at block 34). For example, the blending block 70 may produce a blended video stream (e.g., c) that includes (e.g., an alpha blended) second video stream with the first video stream, as described herein. In which case, the displayed blended video shows (at least a portion of) the second video stream in an area within the first video stream (e.g., where the alpha values of pixels associated with the second video stream in the area are greater than 0.0). In one aspect, the blended video may show the second video stream in one or more areas within the first video stream. For example, when the second video stream includes a GUI with one or more UI items and a separate video stream (captured by a camera other than camera 18), the blended video stream may include one area that includes the GUI and another area that includes the separate video stream. In one aspect, the first video controller may display the blended video stream using the line buffer 28. For instance, the first video controller may store, in the line buffer (or one or more line buffers) 28 (in memory 27 of the first video controller) selected pixel values for one or more lines of pixels for display on the display 22 based on the blended video stream, and may provide the stored selected pixels from the line buffer to the display 22.
The first video controller 23 determines that the second video stream is no longer being superimposed above the area of the first video stream (at block 35). In particular, the system monitor 25 is configured to determine that the video control device is to switch from the blended mode to the failover mode based on one or more criteria. For example, the system monitor 25 may determine that the second video stream is to no longer being superimposed based on a content analysis of the second video stream (and/or the composite video stream). For example, the second video stream may no longer be superimposed by the blended video stream having other video content displayed in lieu of video content of the second video stream (e.g., displaying a black screen in the area as opposed to surgical data). More about determining whether the video control device is to switch between modes is described herein. Responsive to determining that the second video stream is no longer being superimposed, the video control device continues to display the first video stream (e.g., in failover mode) (at block 36). For example, the failover blending block 72 may be activated to produce a second blended video stream from the first (original) blended video stream produced by the blending block 70, which includes the first video stream (or a downscaled version of the stream) superimposed above the first blended stream.
As described thus far, the first video controller may receive a first video stream from the camera 18, and may be configured to blend the first video stream with another (second) video stream produced by the second video controller 24 to produce a blended video stream. In one aspect, the first video stream may comprise one or more 3D video streams. In which case, the camera 18 may be a 3D or stereoscopic camera associated with an endoscope, where the controller receives a left video channel and a right video channel, each 60 fps. In which case, the first video controller may perform at least some of the operations described herein to control and display the 3D video stream, such as alpha blending both the left and right video streams with the video stream received from the second video controller 24. In some aspects, the second video controller may produce one video stream to be blended with the 3D video stream captured by the camera 18. In another aspect, the second video controller may produce separate video streams (e.g., a left surgical data video stream that includes surgical data and a right surgical data video stream that includes surgical data, which may be the same or different than the surgical data within the left video stream). As a result, the video control device may blend left video streams with each other and right video streams with each other, producing a left blended video stream and a right blended video stream for displaying on the display (or displays).
In one aspect, the first video controller 23 may produce one or more failover blended video streams for one or more 3D video streams, as described herein. In the case of the video stream from the camera having one or more 3D video streams, a left channel and a right channel, the memory 27 may include one or more frame buffers for each channel. In which case, upon switching to failover mode, the failover blending block 72 may blend one or more video frames of the left channel, retrieved from a frame buffer, with the left blended video stream, as described herein.
Turning to
The system monitor determines whether the video control device should switch to the failover mode based on the content analysis and/or system data (at decision block 43). For example, the system monitor may determine that the video control device is to operate in the failover mode responsive to determining that the surgical system has a blocked process based on the system data. In one aspect, the system data may indicate that the system has a blocked process based on the (e.g., overall) status of the system indicated by the data. In another aspect, the system monitor may determine that the system has a blocked process (and/or is not operating normally and/or efficiently) based on a historical trend of the system data.
In another aspect, the system monitor may determine to switch modes based on whether the content analysis performed on the second (and/or blended) video stream. For example, the monitor determines whether the second video stream has frozen based on whether objects (e.g., video, images, UI items, etc.) within the video stream have not moved for a period of time. In particular, the system monitor may determine that the video stream is frozen based on a number of video frames received from the second video controller 24 being the same over a period of time. In another aspect, the system monitor may determine to switch modes in response to determining that the second video stream is not showing (does not include), at least a portion, of surgical data that should otherwise be shown. For example, in the case of a (e.g., partial) system failure, the second video stream produced by the second controller 24 may cease to include surgical data (or the surgical data that would otherwise be included during normal operations), and may (at least partially) be a black (or blank) screen (e.g., being a monochromatic screen, where all (or a majority) of pixel values are a same (e.g., RGB) value). Thus, the system monitor may perform the content analysis to determine whether at least some pixels values (e.g., a number of pixels above a threshold) will display one color (e.g., black), indicating that the second video stream includes a blank screen, which would be in contrast to a normal second video stream that would show surgical information (e.g., text, images, etc.) and/or surgical video.
If it is determined that the video control device is to switch to the failover mode (e.g., based on the second video stream not including (e.g., above a threshold amount of) surgical data and/or based on the second video stream obstructing (e.g., greater than a threshold area of) the first video stream, etc.), the first video controller presents a notification (e.g., on the display 22) that provides a recommendation for a user (e.g., an operator of the surgical system 1) to switch from the blended mode to the failover mode in which the first video stream (from one or more cameras 18) is superimposed above an area of the blended video stream (at block 44). In particular, the first video stream may be superimposed above a different area over the blended video stream than the area of the first video stream above which the second video stream is superimposed in the originally blended stream. In one aspect, the notification may be a pop-up notification that may include textual content, image data, and/or video data. In another aspect, the notification may be an audible notification. In which case, the video control device 20 may retrieve (e.g., from memory) an audio signal that contains an audible notification (e.g., speech), and may use the signal to drive one or more speakers (e.g., of the video control device of the) surgical system 1. In another aspect, the video control device may present the notification through other methods, such as electronic mail, SMS, etc.
The monitor 25 receives user input (e.g., via the user input device 21) to switch to the failover mode (at block 45). For instance, the user input device may be a physical switch, which when depressed by a user sends a control signal to the system monitor indicating that the user wishes for the mode of the video control device to switch (from a current mode to another mode). In this case, the system monitor may switch from the blended mode to the failover mode (at block 46). In particular, the system monitor may produce a control signal and provide the signal to the failover blending block 72, which may begin to produce a failover blended video stream using the (original) blended video stream produced by the blending block 70 and the first video stream for display, as described herein. Thus, by displaying the failover blended video stream the surgical system is able to continue to (e.g., seamlessly) display the video stream of the camera 18, as described herein.
Returning to decision block 43, the system monitor determines (e.g., responsive to determining that the video control device is not to switch based on system data and/or the content analysis) an area of the portion of the first video stream (e.g., with respect to a total area in which the first video stream is displayed) over which the second video stream is displayed (at block 47). In particular, the system monitor 25 may perform a content analysis upon the blended video stream to determine whether or not to switch between modes. For example, errors in the blended video stream that would result in non-display or obscuring of the first (e.g., live) video stream captured by the camera 18, and thus, the monitor may analyze the composite video for an indication that the blended video has errors or is obscuring the view of the first video stream. In one aspect, an error in the blended video stream may be due to the video stream from the second controller not (erroneously) including an alpha channel and/or alpha values erroneously resulting in more of the second video stream being displayed than necessary. In one aspect, to determine the area over which the second video stream is to be displayed based on one or more video frames of the second video stream. For instance, the system monitor may count a number of lines of a video frame of the second video stream are solid lines (e.g., having an alpha value that results in the second video being (partially or fully) opaque over one or more areas of the first video stream).
The system monitor 25 determines whether that area exceeds a threshold area (at decision block 48). For example, the system monitor may determine whether the number of solid lines is greater than some fraction “x” of lines of a corresponding frame of the first video stream. In particular, the system monitor is determining whether the second video stream is going to occlude a greater portion of the first video stream than it should (e.g., the monitor determines whether a percentage of the area of the first video stream that is occluded by the second video stream is greater than a threshold percentage). In one aspect, the area over which the second video stream indicates it should be displayed may increase due to computational issues within the surgical system. In some aspects, the system monitor compares the area of pixels that has an alpha value greater than a value (e.g., 1) of the second video stream to the threshold area. If the area is greater, the system monitor proceeds to block 44.
Otherwise, the system monitor 25 determines whether user input has been received to switch modes (at block 49). In particular, the system may determine that whether an operator wishes to switch to failover mode (e.g., based on receiving a control signal from the user input device 21), even though the second video stream is sufficiently being overlaid and displayed on top of the first video stream. In one aspect, the video control device may receive such user input when an operator only wants to focus on the image being displayed in the first video stream. If so, the system monitor switches to the failover mode (at block 46). Otherwise, the first and second video streams continue to be displayed in the blended mode (at block 50).
Some aspects may perform variations to the processes 30 and 40 described herein. For example, the specific operations of at least some of the processes may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations and different specific operations may be performed in different aspects. For example, the operations within dashed boxes may be optional operations that may not be performed while a respective process is performed. As an example, in process 40 blocks 44 and 45 are optional, such that the first video controller 23 may omit (either or both of) those operations. For instance, once it is determined that the are exceeds the threshold area, the first video controller may automatically (e.g., without user intervention) switch to the failover mode.
As described thus far, the process 40 describes operations for the video control device 20 to switch from blended mode to the failover mode. In another aspect, the video control device 20 may switch (e.g., back) from the failover mode to the blended mode. For example, the system monitor 25 may continuously monitor data to determine whether the second video stream from the second video controller is suitable for display on the display 22. As an example, upon determining that pixel values of the second (e.g., blended) video stream indicate that the stream is no longer displaying a blank screen (e.g., a monochromatic screen), and instead includes various pixel values (e.g., indicating that the stream includes objects, such as UI items of a GUI), the system monitor may instruct the (e.g., failover blending block 72 of the) first video controller 23 to switch back to blended mode. Thus, the system monitor may perform at least some of the operations described herein to determine whether to switch between the modes.
As described thus far, the first video stream may be aa video stream (or one or more combined video streams) that are displayed in the failover and blended modes, which is received from one or more cameras 18 (e.g., an endoscope, a laparoscope, a colonoscope, a bronchoscope, etc.). In one aspect, the first video stream may be a “critical” video stream which is a live video stream (e.g., being displayed within a threshold of time after it is captured by the camera 18) and/or is to always be displayed on a display during a surgical (e.g., an intraoperative) procedure. In another aspect, the first video stream may include image and/or video data from other image/video sources.
As described thus far, the video control device may receive user input to switch between the blended mode and the failover mode (e.g., the operator toggling a switch). In another aspect, the video control device may be configured to toggle between one or more video sources. For example, the first video controller may receive one or more video streams from one or more cameras 18. As an example, the system may include two endoscopes, where the video control device is coupled to both endoscopes. In one aspect, the user input device 21 may be configured to allow the operator to switch between the endoscopes. For instance, while active, the video control device may display a video stream from a first endoscope (e.g., as the first video stream, as described herein). Upon receiving user input, the video control device may switch to receiving video from a second endoscope. In another aspect, the first video stream may be a blended video stream of one or more (e.g., critical) video streams. In this case, the critical video stream may be a blended video stream, where one area of the blended stream includes video from one endoscope, while another area of the blended stream includes video from another endoscope.
The third stage 62 shows the video content of the failover blended video stream, displayed while the surgical system is operating in failover mode. In particular, this figure is showing that the first video stream is superimposed above an area of the blended video stream (e.g., above the monochromatic screen of the second video stream). In one aspect, the first video stream may be superimposed across an entire area of the screen (e.g., as large as the screen). In another aspect, the first video screen may be disposed within an area smaller than the screen, as shown herein. This may allow the surgical system to display one or more notifications in one or more areas that do not obstruct the first video stream (e.g., the surgical system may display a notification between the first video stream and the top of the display in the third stage 62.
While certain aspects have been described and shown in the accompanying drawings, it is to be understood that such aspects are merely illustrative of and not restrictive on the broad invention, and the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those of ordinary skill in the art. The description is thus to be regarded as illustrative instead of limiting.
To aid the Patent Office and any readers of any patent issued on this application in interpreting the claims appended hereto, applicants wish to note that they do not intend any of the appended claims or claim elements to invoke 35 U.S.C. 112(f) unless the words “means for” or “step for” are explicitly used in the particular claim.
As previously explained, an aspect of the disclosure may be a non-transitory machine-readable medium (such as microelectronic memory) having stored thereon instructions, which program one or more data processing components (generically referred to here as a “processor”) to automatically (e.g., without user intervention) perform video stream controlling and displaying operations, as described herein. In other aspects, some of these operations might be performed by specific hardware components that contain hardwired logic. Those operations might alternatively be performed by any combination of programmed data processing components and fixed hardwired circuit components.
While certain aspects have been described and shown in the accompanying drawings, it is to be understood that such aspects are merely illustrative of and not restrictive on the broad disclosure, and that the disclosure is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those of ordinary skill in the art. The description is thus to be regarded as illustrative instead of limiting.
In some aspects, this disclosure may include the language, for example, “at least one of [element A] and [element B].” This language may refer to one or more of the elements. For example, “at least one of A and B” may refer to “A,” “B,” or “A and B.” Specifically, “at least one of A and B” may refer to “at least one of A and at least one of B,” or “at least of either A or B.” In some aspects, this disclosure may include the language, for example, “[element A], [element B], and/or [element C].” This language may refer to either of the elements or any combination thereof. For instance, “A, B, and/or C” may refer to “A,” “B,” “C,” “A and B,” “A and C,” “B and C,” or “A, B, and C.”
Claims
1. A method performed by a video controller, the method comprising:
- receiving a first video stream captured by an endoscope of a surgical system;
- receiving a second video stream that comprises surgical data;
- displaying the second video stream superimposed above an area of the first video stream;
- determining that the second video stream is no longer being superimposed above the area of the first video stream;
- responsive to determining that the second video stream is no longer being superimposed above the area of the first video stream, continuing to display the first video stream.
2. The method of claim 1, wherein the surgical data comprises a graphical user interface (GUI) that includes at least one of a notification associated with a surgical procedure, image data captured by one or more cameras of the surgical system, and a user interface (UI) item for allowing a user to interact with the GUI through a display.
3. The method of claim 1, wherein a display comprises a plurality of lines of pixels, wherein displaying the second video stream superimposed above an area of the first video stream comprises:
- producing a blended video stream by blending the first video stream with the second video stream;
- storing, in a line buffer, selected pixels for a set of lines of the plurality of lines based on the blended video stream; and
- providing, from the line buffer, the stored selected pixels to the display.
4. The method of claim 1, wherein the area is a first area, wherein, while the second video stream is superimposed above the area of the first video stream, the surgical system is in a blended mode in which a blended video stream of the first and second video streams is displayed, wherein the method further comprises presenting a notification that provides a recommendation for a user of the surgical system to switch to a failover mode in which the first video stream is superimposed above a second area of the blended video stream, wherein the first video stream is continued to be displayed responsive to receiving user input via a user input device for switching from the blended mode to the failover mode.
5. The method of claim 1, wherein determining that the second video stream is no longer being superimposed comprises determining that the area of the first video stream over which the second video stream is displayed exceeds a threshold area.
6. The method of claim 1, wherein the video controller is a first video controller, wherein the second video stream is received from a second video controller of the surgical system.
7. The method of claim 6, wherein the first video controller is a field-programmable gate array (FPGA), and the second video controller is a graphics processing unit (GPU).
8. The method of claim 1, wherein displaying the second video stream superimposed above an area of the first video stream comprises providing a first blended video stream that includes the first and second video streams to a display, wherein continuing to display the first video stream comprises
- producing a second blended video stream by retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set video frames with a second set of one or more video frames of the first blended video stream; and
- providing the second blended video stream to a display.
9. A video control device for a surgical system that includes an endoscope and a display, the device comprising:
- a first video controller configured to receive surgical data associated with the surgical system; and
- a second video controller, wherein both video controllers are disposed within a housing of the video control device,
- wherein the second video controller configured to receive, from the endoscope, a first video stream; receive, from the first video controller, a second video stream that includes the surgical data; display, on the display, the second video stream superimposed above an area of the first video stream; determine that the second video stream is no longer being superimposed above the area of the first video stream; and responsive to determining that the second video stream is no longer being superimposed above the area of the first video stream, continue to display the first video stream.
10. The video control device of claim 9, wherein the first video controller draws power from a first power supply and the second video controller draws power from a second power supply.
11. The video control device of claim 9, wherein the first video controller is a graphics processing unit (GPU), and the second video controller is a field-programmable gate array (FPGA).
12. The video control device of claim 9, wherein the surgical data comprises a graphical user interface (GUI) that includes at least one of a notification associated with a surgical procedure, image data captured by one or more cameras of the surgical system, and a user interface (UI) item for allowing a user to interact with the GUI through a display.
13. The video control device of claim 9, wherein a display comprises a plurality of lines of pixels, wherein the second video controller displays the second video stream superimposed above an area of the first video stream by:
- producing a blended video stream by blending the first video stream with the second video stream;
- storing, in a line buffer, selected pixels for a set of lines of the plurality of lines based on the blended video stream; and
- providing, from the line buffer, the stored selected pixels to the display.
14. The video control device of claim 9, wherein the area is a first area, wherein, while the second video stream is superimposed above the area of the first video stream, surgical system is in a blended mode in which a blended video stream of the first and second video streams is displayed, wherein the second video controller is further configured to present a notification that provides a recommendation for a user of the surgical system to switch to a failover mode in which the first video stream is superimposed above a second area of the blended video stream, wherein the first video stream is continued to be displayed responsive to the second video controller receiving user input via a user input device for switching from the blended mode to the failover mode.
15. The video control device of claim 9, wherein the second video controller determines that the second video stream is no longer being superimposed by determining that the area of the first video stream over which the second video stream is displayed exceeds a threshold area.
16. The video control device of claim 9, wherein the second video controller displays the second video stream superimposed above an area of the first video stream by providing a first blended video stream that includes the first and second video streams to a display, wherein the second video controller displays continues to display the first video stream by:
- producing a second blended video stream by retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set of video frames with a second set of video frames of the first blended video stream; and
- providing the second blended video stream to a display.
17. A system comprising:
- an endoscope;
- a display; and
- a video controller; and
- memory having instructions which when executed by the video controller causes the system to: receive a first video stream captured by the endoscope; receive a second video stream that comprises surgical data; display, on the display, the second video stream superimposed above an area of the first video stream; determine that the second video stream is no longer being superimposed above the area of the first video stream; responsive to determining that the second video stream is no longer being superimposed above the area of the first video stream continue to display the first video stream.
18. The system of claim 17, wherein the surgical data comprises a graphical user interface (GUI) that includes at least one of a notification with a surgical procedure, image data captured by one or more cameras of the surgical system, and a user interface (UI) item for allowing a user to interact with the GUI.
19. The system of claim 17, wherein the display comprises a plurality of lines of pixels, wherein the instructions to display the second video stream superimposed above an area of the first video stream comprises instructions to:
- produce a blended video stream by blending the first video stream with the second video stream;
- store, in a line buffer, selected pixels for a set of lines of the plurality of lines based on the blended video stream; and
- provide, from the line buffer, the stored selected pixels to the display.
20. The system of claim 17, wherein the area is a first area, wherein, while the second video stream is superimposed above the area of the first video stream, the surgical system is in a blended mode in which a blended video stream of the first and second video streams is displayed,
- wherein the memory has further instructions to present a notification that provides a recommendation for a user of the surgical system to switch to a failover mode in which the first video stream is superimposed above a second area of the blended video stream, wherein the first video stream is continued to be displayed responsive to receiving user input via a user input device for switching from the blended mode to the failover mode.
21. The system of claim 17, wherein the instructions to determine that the second video stream is no longer being superimposed comprises instructions to determine that the area of the first video stream over which the second video stream is displayed exceeds a threshold area.
22. The system of claim 17, wherein displaying the second video stream superimposed above an area of the first video stream comprises providing a first blended video stream that includes the first and second video streams to a display, wherein continuing to display the first video stream comprises
- producing a second blended video stream by retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set video frames with a second set of one or more video frames of the first blended video stream; and
- providing the second blended video stream to a display.
23. The system of claim 17, wherein the video controller is a field-programmable gate array (FPGA), wherein the second video stream is received from graphics processing unit (GPU) of the surgical system.
Type: Application
Filed: Oct 14, 2022
Publication Date: Apr 20, 2023
Inventors: Yiming Xu (Sunnyvale, CA), Felix Jonas Bork (Schnuerpflingen), Axel Hanuschek (Munich, DE), Risto Kojcev (Cupertino, CA), Bernhard Adolf Fuerst (Mountain View, CA)
Application Number: 18/046,903
