MEDICAL IMAGAING SYSTEM
A medical imaging system includes multiple illuminating endoscopes with cameras having image sensors exposure control and a frame memory. The exposure control of each camera activates the frame memory and light source for collection of selected frame images by the frame memory. A data link among the endoscopes synchronizes exclusive initiation of the exposure controls in recurring serial order for activation of each respective frame memory and light source.
This application is a continuation application of U.S. patent application Ser. No. 16/782,300, filed on Feb. 5, 2020, entitled MEDICAL IMAGING SYSTEM, which is a divisional application that claims priority of U.S. patent application Ser. No. 16/041,605, filed on Jul. 20, 2018 and issued into U.S. Pat. No. 10,638,921 on May 5, 2020 entitled MEDICAL IMAGING SYSTEM, the entire contents of both of which are hereby incorporated herein by reference.
BACKGROUND OF THE INVENTIONThe field of the present invention is medical video imaging.
Video imaging in surgery and other medical procedures has become quite common as minimally invasive endoscopic techniques have replaced previously used open approaches to treat patients. Such imaging is very advantageous for endoscopic joint surgery, particularly in the knee and shoulder. Such surgery can additionally benefit from two or more simultaneous endoscope camera views. Where joint surgery requires that the camera move repeatedly between portals, multiple cameras can reduce surgery time. Other orthopedic applications where a rigid endoscope is now used may benefit from a second view reachable only by a small diameter needle endoscope. Using multiple endoscopes at the same time can be useful. However, most endoscopes are coupled to a camera to collect the image and also to a light source for illumination. Should one endoscope be directed toward another, the illumination from the one can spoil the other's image. Additionally, some endoscope cameras control the lighting as part of their automatic exposure system. If lighting comes from another source, a camera auto exposure system may not be able to maintain the correct image brightness.
Endoscopic video systems have included an endoscope, camera, light source, and display. The camera is comprised of a camera control unit (CCU) and an image sensor. These system components have been arranged in different ways depending on the application. For example, the image sensor may be placed at the distal tip of the endoscope and the light source located in the endoscope handle. A large high-performance image sensor may be placed in a camera head enclosure attached to the proximal endoscope eyepiece and the light delivered to the endoscope through glass fibers from a remote light source.
SUMMARY OF THE INVENTIONThe present invention is directed to video imaging systems that provide for the use of multiple video cameras with illumination in a medical procedure at the same time.
Method and apparatus are disclosed coordinating the timing of image capture and illumination of multiple endoscopes. Each can be made to video capture without others degrading the display. In doing this, the images being captured are not compromised by interfering illumination. At the same time, real time sequential images from multiple endoscopes may be separately displayed. The timing is coordinated such that only the light source associated with one endoscope is fully illuminated for image exposure during image capture from the associated endoscope. Background illumination may be provided in a second mode from another light source not from the light source then fully illuminated. By effecting image capture with each endoscope in serial order, multiple video displays from the endoscopes can run concurrently.
Accordingly, it is a principal object of the present invention to provide a system for advantageous employment of multiple illuminated video cameras in medical procedures. Other and further objects and advantages will appear hereinafter.
In the following description of the preferred embodiments, reference is made to the accompanying drawings which show by way of illustration specific embodiments in which the invention may be practiced. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTSIn
The video camera head 12 acquires image data and transmits it to the camera control unit 14 to process a usable image. Traditionally a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) is used as an image sensor 18. A CCD based video camera employs a global shutter image sensor; and a CMOS based video camera employs either a global shutter or a rolling shutter sensor. Further, CMOS rolling shutter sensors can incorporate global reset. The image sensor may be located at the distal end of an endoscope. From this group of shutter types, similar or dissimilar types may be used together in an individual system,
Referring specifically to
The light source 20 includes a lamp 30. The lamp 30 is traditionally a semiconductor light source such as laser or LED to require low power and provide rapid activation to illuminate the field of view of the endoscopic camera 10. A light guide 32 is shown in
Referring specifically to
Timing in video cameras must be very precise and consistent. Software code executing on a processor is typically nondeterministic and therefore unable to produce the needed consistent operation. Instead field programmable gate arrays (FPGA) are preferred to control and process the output from image sensors. All processing steps of the video image are preferably done with an FPGA to achieve the required real time precise timing needed to generate a standard video output signal. In an endoscopic video system user interface logic and possible external network connectivity might be performed by software running on a processor.
In a current embodiment, analog RGB data is transmitted from the image sensor 18 to the camera control unit 14. The Analog RGB data passes through an Analog/Digital converter 56 to an FPGA 58 where the video is processed. The processed video is then passed to a video output 60 that may include a formatter FPGA where the video is formatted into various display formats. The formatter FPGA may also overlay information, such as patient and/or doctor information, onto the video. The formatted video may be converted back to an analog signal for display. The formatted video is sent to the display 50 and/or the storage device 52. Alternatively, an Analog/Digital converter may be located in the camera head and digital RGB data transmitted from the camera head 12 to the camera control unit 14. Additionally, the imaging device 18 itself may include an Analog/Digital converter.
The camera control unit 14 issues commands to the camera head 12 to adjust its operating characteristics, and the camera head 12 may send confirmation to the camera control unit 14 that it received the commands. The processor FPGA 58 and/or the controller 48 may communicate with a shutter driver either in the camera control unit 14 or the camera head 12 to control the exposure period of the imaging device. The image sensor 18 employed with the camera 10 must be accommodated as to choice of frame rate, pixel rate, number of horizontal lines per frame, pixels per line, type of color representation, image aspect ratio. For example, the image sensor may be a single row and column array with color filters applied over individual pixels in a Bayer pattern or multiple sensors mounted on a wavelength separating prism. If there is more than a single type of sensor that can be used with a camera system then a choice from a predetermined set of these settings is made to match the sensor. When a camera operates in standalone mode, that is by itself before another camera responds on the data link, the frame rate is determined by the time it takes to scan all the pixels. Assume the sensor pixel rate is 148.5 Mpix/sec and the number of lines is 1125 and the pixels per line is 2200. 1125×2200=2.475 Mpixels per frame so that 148.5/2.475=60 frames per second. The actual number of active image pixels per frame is fewer because of dark reference lines and pixels as well as intentional blanking time. For example, in a standard HDTV frame there are 1920 pixels per line×1080 vertical lines in each image frame. Additionally, the processor FPGA 58 and/or the controller 48 communicates with a light source 20 either in the camera control unit 14 or the camera head 12 to control the drive current to the lamp 30 of the light source 20.
In addition to the processing elements 48, 58, functionally identified in
In the frame processing function 70, there also is the white balance process to correct for color variation of the illumination light to assure accurate color reproduction. As with traditional endoscopic cameras, each camera 10 is white balanced by the user while imaging a white target. Only the light source associated with the camera undergoing the white balance process is illuminated during white balance. The frame processing function 70 also provides image brightness measurement which is directed to exposure control 64. Image brightness is calculated from the pixel values produced by the image sensor for each frame. These are the same pixel values that make up the image. The result is used to choose exposure settings for the next frame. Exposure settings may be image sensor exposure duration, light source intensity or light source ON duration.
The frame memory 72 either discards or reserves the incoming normalized frame images according to the preselected pattern of exposure control 64. The reserved frame images are then communicated to the video output 60. The display 50 typically requires 60 frames/second. Because of the discarded frame images, a new frame image may not be available for every required frame at the display. The video output 60 uses frame rate conversion to present the last received selected frame image from the frame memory 72 at the required frame rate of the display 50. The display 50 may also be combined into one split screen monitor receiving multiple signals from the video output or outputs 60.
To begin, the lead camera 10A, with recognition of the formats of the other linked cameras 10, will define the order for the initiation of each camera exposure control 64 to activate each respective frame memory 72 and light source 20. Each activation is “exclusive” such that no two light sources 20 provide direct lighting at the same time. The term “direct lighting” is here used to denote lighting that is aligned with the associated camera 10 to illuminate the field of view of that camera. Having the direct lighting operating exclusively prevents the light source 20 from one linked endoscope overexposing a selected image frame of a field of view captured by another of the linked endoscopes. In turn, the respective light source 20 is timed to properly illuminate the field of view for the selected image frame or sequential image frames exclusively recorded. In steps once the first camera 10A has been established in the lead, the second camera 10B comes online and broadcasts its preferred sensor information on the data link 62. Referring to
The synchronized routines performed by the exposure control 64 of the lead camera 10 and the camera exposure controls 64 of each camera 10 tracking in repeated serial order is illustrated in
A first order process simply turns off all but the direct lighting of the light source 20 associated with the camera 10 capturing selected frame images, a first “mode”. However, need for supplemental lighting can be accommodated by modulating the light source or sources 20 associated with the camera or cameras 10 not concurrently capturing selected frame images in the system of
The background lighting will have varying effect based on the relative positioning of the distal ends of the endoscopes. If the viewing and lighting angle between endoscopes is less than 90°, the two endoscopes will possibly provide complementary light toward the field of view. At obtuse angles, the background lighting may not brighten the field of view but provide more light intensity to the image sensor. This would normally be a disfavored result. Therefore, it can be most advantageous to use a camera graphical user interface (GUI) to manually control the second mode of operation. The operator can then manipulate the orientation of the endoscopes in conjunction with control of the background mode. An automatic second mode using a slower exposure control feedback loop may also be used as the operator can better control the video by reorienting the relative positions of the endoscopes.
The light sources 20 are also synchronized as to timing and duration by the camera exposure controls 64. Each light source 20 is exclusively activated to coincide with the generation of video frame images by the associated image sensor 18 that are to be selectively captured in the associated frame memory 72. This exclusive activation occurs in recurring serial order as well to provide the light needed for proper frame exposure. To avoid chance overexposure, each light source 20 is deactivated when another of the data linked cameras 10 is generating video frame images by its own associated image sensor 18 that are to be selectively captured in its associated frame memory 72. The camera exposure controls 64 also may be employed to control light source activated duration and brightness to maintain proper exposure.
The timing synchronization is straight forward with two global shutter image sensors, as shown in
Using an image sensor 18 with a rolling shutter slows the frame rate for each camera to half because the sensor requires an extra frame time to generate all raster lines.
Thus, medical imaging systems are here disclosed which allow the use of multiple illuminating endoscopes with video cameras at a surgical site without light interference. While embodiments and applications of this invention have been shown and described, it would be apparent to those skilled in the art that many more modifications are possible without departing from the inventive concepts herein. The invention, therefore, is not to be restricted except in the spirit of the appended claims.
Claims
1. A medical imaging system comprising:
- a plurality of endoscopic video cameras, each endoscopic video camera further comprising:
- a light source; an image sensor capable of sensing video frame images; and a frame memory capable of collecting sensed video frame images;
- a controller coupled to the plurality of endoscopic video cameras, the controller synchronizing the endoscopic video cameras for alternate illumination by the light sources, and for alternate collection of the video frame images by the frame memories such that at least one endoscopic video camera frame memory is deactivated when the frame memory of another endoscopic video camera is collecting video frame images.
2. The medical imaging system of claim 1, wherein the controller controls the endoscopic video cameras so that at least endoscopic video camera light source is deactivated when the frame memory of another endoscopic video camera is collecting video frame images.
3. The medical imaging system of claim 1, wherein the controller activates the light source of one of the endoscopic video cameras for providing background lighting when the frame memory of another of the endoscopic video cameras is collecting video frame images.
4. The medical imaging system of claim 1, wherein the controller has a video capture profile of the image sensor frame rate, a sequence of the selected frame images and light source duration for the activation of each respective frame memory.
5. The medical imaging system of claim 4, each endoscopic video camera further including a video output receiving the respective selected frame images.
6. The medical imaging system of claim 5 operating at a preselected video frame rate, each video output presenting the last received selected frame image from the respective frame memory at the preselected video frame rate, respectively.
7. The medical imaging system of claim 6 further comprising
- at least one image display, each video output being coupled to the at least one image display, respectively.
8. The medical imaging system of claim 1, the endoscopic video cameras each further comprising an exposure control, the exposure control controlling the respective image sensor, frame memory, and light source.
9. The medical imaging system of claim 8, the exposure control modulating one or both of intensity and activation duration of the light source responsive to image brightness.
10. The medical imaging system of claim 1, each endoscopic video camera further including a shutter selected from the group consisting of a global shutter, a rolling shutter, and a rolling shutter having a global reset.
11. The medical imaging system of claim 1 wherein the system comprises two endoscopic video cameras and each of the endoscopic video cameras has a rolling shutter having a global reset.
12. The medical imaging system of claim 1, the controller including a camera control unit in each of the endoscopic video cameras and a data link therebetween.
13. A process for using two endoscopic cameras in a medical procedure, each endoscopic camera including an image sensor capable of sensing video frame images, a video output for selected sensed video frame images, and a light source having a first mode of direct lighting, comprising the steps of
- positioning the endoscopic cameras at a surgical site;
- synchronizing the endoscopic cameras for alternate first mode illumination by the light sources and for selecting video frame images from each endoscopic camera only during first mode illumination of the light source of that camera; and
- communicating the selected video frame images to the respective video outputs.
14. The process of claim 13, wherein the light sources of each endoscope further comprise a second mode of background lighting, and the process further comprises the step of
- activating the light sources so that the light source of at least one of the endoscopic cameras illuminates in the second mode of background lighting while the other endoscopic camera light source is illuminated in first mode illumination.
15. The process of claim 13, wherein the light sources of each endoscope further comprise a second mode of background lighting, and the process further comprises the step of
- activating the light sources so that the light source of each of the endoscopic cameras illuminates in the second mode of background lighting while the other endoscopic camera light source is illuminated in first mode illumination.
Type: Application
Filed: Apr 16, 2021
Publication Date: Jul 29, 2021
Inventors: BRUCE LAURENCE KENNEDY (Santa Barbara, CA), DAVID A. D'ALFONSO (Gaviota, CA)
Application Number: 17/233,131