DATABASE SYSTEMS AND INTERACTIVE USER INTERFACES FOR DYNAMIC INTERACTION WITH, AND COMPARISON OF, DIGITAL MEDICAL IMAGE DATA
Database systems and techniques are disclosed for accessing data stores of digital medical images, processing the digital images, and displaying the digital images to efficiently provide information in an interactive user interface. The disclosure may advantageously provide efficient and rapid dynamic interaction with digital images accessed from one or more databases to enable user detection of differences between related digital images. Interactive user interfaces may be dynamically updated to provide rapid comparison of digital images. Further, digital images from multiple data sources may be automatically sorted by the system according to attributes associated with the images and rules and/or preferences of the user. In an embodiment the user may select a digital image from a first data source, and the system automatically determines and displays one or more comparison images from other image data sources. Images may additionally be automatically registered and/or matched to enable more efficient comparison and evaluation.
Any and all applications for which a foreign or domestic priority claim is identified in the Application Data Sheet as filed with the present application are hereby incorporated by reference under 37 CFR 1.57.
This application claims benefit of U.S. Provisional Patent Application No. 62/155,362, filed Apr. 30, 2015, and titled “SYSTEMS AND USER INTERFACES FOR DYNAMIC INTERACTION WITH, AND COMPARISON OF, MEDICAL IMAGE DATA,” and also claims benefit of U.S. Provisional Patent Application No. 62/171,866, filed Jun. 5, 2015, and titled “SYSTEMS AND USER INTERFACES FOR DYNAMIC INTERACTION WITH, AND COMPARISON OF, MEDICAL IMAGE DATA.”
This application is also related to U.S. patent application Ser. No. ______, filed on Apr. 27, 2016, with attorney docket number MERG1.042A1, and titled “DATABASE SYSTEMS AND INTERACTIVE USER INTERFACES FOR DYNAMIC INTERACTION WITH, AND SORTING OF, DIGITAL MEDICAL IMAGE DATA,” U.S. patent application Ser. No. ______, filed on Apr. 27, 2016, with attorney docket number MERG1.042A3 and titled “DATABASE SYSTEMS AND INTERACTIVE USER INTERFACES FOR DYNAMIC INTERACTION WITH, AND REVIEW OF, DIGITAL MEDICAL IMAGE DATA,” U.S. patent application Ser. No. ______, filed on Apr. 27, 2016, with attorney docket number MERG1.042A4, and titled “DATABASE SYSTEMS AND INTERACTIVE USER INTERFACES FOR DYNAMIC INTERACTION WITH, AND INDICATIONS OF, DIGITAL MEDICAL IMAGE DATA.”
The entire disclosure of each of the above items is hereby made part of this specification as if set forth fully herein and incorporated by reference for all purposes, for all that it contains.
TECHNICAL FIELDEmbodiments of the present disclosure relate to systems and techniques for accessing one or more databases and providing user interfaces for dynamic interactions with digital medical image data.
BACKGROUNDThe approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
Medical images are typically viewed by radiologists and other physicians, patients, and/or others by interaction with desktop computer systems with stationary monitors. When comparison of related images is required, subtle differences between images may be difficult to detect. For example, if a lung radiograph from two months previous, and a current lung radiograph are to be compared in order to determine if any changes have occurred in the lungs over the previous two months, the viewer or reader typically views the two x-rays side by side. For example, the viewer or reader may have two monitors placed side by side, wherein each of the monitors displays a chest radiographic image. Alternatively, the viewer may view the two images side by side on a single monitor. However, as those of skill in the art will recognize, identifying differences in related images in this manner is often tedious and difficult. Some imaging modalities, such as CT and MRI, produce a large number of images, hundreds to even thousands of images per exam. In many cases, comparison of different series of images within the exam is required. For example, comparison of pre and post contrast images to detect areas of enhancement or comparison of PET and CT images for localization of activity is often necessary. Further, these often large exams may need to be compared to multiple prior exams to detect subtle, progressive changes over time, for example to detect a small, growing tumor.
SUMMARYThe systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of this disclosure, several non-limiting features will now be described briefly.
Embodiments of the present disclosure relate to database systems and techniques for accessing data stores of medical images and displaying the medical images to efficiently provide information in an interactive user interface. Previous systems for display of, and interaction with, image data were typically inefficient at enabling detection of differences between images and evaluation of images. Disclosed herein are systems that, according to various embodiments, advantageously provide highly efficient, intuitive, and rapid dynamic interaction with medical images (including two-dimensional images and images rendered from three-dimensional image data) to enable detection of differences between related medical images and evaluation of medical images. The systems may include interactive user interfaces that are dynamically updated to provide rapid comparison of images. Further, images from multiple series and/or exams, and/or images from other sources, may be automatically sorted (e.g., in one embodiment, interleaved) by the system according to attributes associated with the images and rules and/or preferences of the user.
In an embodiment, the user may select an image from a series of a first exam, and the system automatically determines and displays one or more comparison images from another image series and/or exams. Images selected for comparison, and/or images that are sorted, may additionally be automatically registered and/or matched to enable more efficient comparison and evaluation by the user. Accordingly, a user may use the systems described herein to more quickly, thoroughly, and efficiently interact with medical images, as compared to previous systems. The features and advantages noted above, as well as others, are discussed in further detail below.
In various embodiments, systems and methods are disclosed for matching related medical images and/or medical image series from multiple exams, automatically displaying medical images in particular arrangements, and automatically sorting medical images from related exams. In one example, a user selects a medical image, and the system automatically identifies related images and/or medical image series from 2, 3, or 4 (or more) other exams and displays the images next to one another in a grid arrangement, and/or sorts the images and displays them sequentially in an image pane.
Accordingly, in various embodiments, large amounts of data are automatically and dynamically calculated interactively in response to user inputs, and the calculated data is efficiently and compactly presented to a user by the system. Thus, in some embodiments, the user interfaces described herein are more efficient as compared to previous user interfaces in which data is not dynamically updated and compactly and efficiently presented to the user in response to interactive inputs.
Further, as described herein, the system may be configured and/or designed to generate user interface data useable for rendering the various interactive user interfaces described. The user interface data may be used by the system, and/or another computer system, device, and/or software program (for example, a browser program), to render the interactive user interfaces. The interactive user interfaces may be displayed on, for example, electronic displays (including, for example, touch-enabled displays).
It has been noted that design of computer user interfaces “that are useable and easily learned by humans is a non-trivial problem for software developers.” (Dillon, A. (2003) User Interface Design. MacMillan Encyclopedia of Cognitive Science, Vol. 4, London: MacMillan, 453-458.) The present disclosure describes various embodiments of interactive and dynamic user interfaces that are the result of significant development. This non-trivial development has resulted in the user interfaces described herein which may provide significant cognitive and ergonomic efficiencies and advantages over previous systems. The interactive and dynamic user interfaces include improved human-computer interactions that may provide reduced mental workloads, improved decision-making, reduced work stress, and/or the like, for a user. For example, user interaction with the interactive user interface via the inputs described herein may provide an optimized display of, and interaction with, image data (including medical images) and may enable a user to more quickly and accurately access, navigate, assess, and digest the image data than previous systems.
Further, the interactive and dynamic user interfaces described herein are enabled by innovations in efficient interactions between the user interfaces and underlying systems and components. For example, disclosed herein are improved methods of receiving user inputs (including methods of interacting with, and selecting, images), translation and delivery of those inputs to various system components, automatic and dynamic execution of complex processes in response to the input delivery, automatic interaction among various components and processes of the system, and automatic and dynamic updating of the user interfaces (to, for example, display the relevant medical images). The interactions and presentation of data via the interactive user interfaces described herein may accordingly provide cognitive and ergonomic efficiencies and advantages over previous systems.
Various embodiments of the present disclosure provide improvements to various technologies and technological fields. For example, as described above, existing medical image interaction technology (including, e.g., Picture Archiving and Communication Systems, Electronic Medical Record Systems, and/or the like) is limited in various ways (e.g., image review is slow and cumbersome, comparison of images is inefficient, etc.), and various embodiments of the disclosure provide significant improvements over such technology. Additionally, various embodiments of the present disclosure are inextricably tied to computer technology. In particular, various embodiments rely on detection of user inputs via graphical user interfaces, calculation of updates to displayed electronic data based on those user inputs, automatic processing of related electronic medical images, and presentation of the updates to displayed medical images via interactive graphical user interfaces. Such features and others (e.g., generation of 2D medical images from a 3D imaging volume and/or other 2D images, such as at automatically selected or user-selected planes) are intimately tied to, and enabled by, computer technology, and would not exist except for computer technology. For example, the interactions with displayed data described below in reference to various embodiments cannot reasonably be performed by humans alone, without the computer technology upon which they are implemented. Further, the implementation of the various embodiments of the present disclosure via computer technology enables many of the advantages described herein, including more efficient interaction with, and presentation of, various types of electronic image data.
The inventors have found that users using the system to flip through medical images (e.g., to analyze the images and perform diagnoses based on the images) are faster (e.g., 15% faster or more) and more accurate than users using traditional methods of comparing images.
In an embodiment, a method of viewing medical images from two or more image series on a display device coupled to a computing device comprises the actions of selecting a first image series comprising two or more medical images, selecting at least one comparison image series, each of the comparison image series comprising two or more medical images, interleaving images of the first image series and the comparison image series in order to form an interleaved image series, and sequentially displaying the images of the interleaved image series at a single location on the display device.
In another embodiment, a method of viewing a series of medical images on a display device coupled to a computing device comprises the actions of (a) selecting a first image series for viewing, the first image series comprising a plurality X of medical images, (b) selecting a second image series for viewing, the second image series comprising a plurality Y of medical images, (c) displaying at a predetermined location on the display device a Nth image of the first image series, (d) replacing the Nth image of the first image series with a Mth image of the second image series at the predetermined location, (e) incrementing N and M, and (f) repeating steps (c) to (f).
In yet another embodiment, a system for enhancing a viewer's ability to detect differences between medical images in two or more sets of medical images comprises a display device, a graphical user interface displayed on the display device and comprising an image pane configured to display a single medical image at a time, an image selection module to select two or more sets of medical images, each of the sets of medical images comprising two or more medical images, and a user interface to receive commands from a user, wherein in response to receiving a first command from the user, the image pane sequentially displays a first medical image from each of the image sets and, after displaying the first medical image from each image set, the image pane sequentially displays a second medical image from each image set. This process of displaying images from images series alternatively continues through subsequent images in the image series.
In another embodiment, a system of viewing medical images from two or more image series on a display device coupled to a computing device comprises means for selecting a first image series comprising two or more medical images, means for selecting at least one comparison image series, each of the comparison image series comprising two or more medical images, means for interleaving images of the first image series and the comparison image series in order to form an interleaved image series, and means for sequentially displaying the images of the interleaved image series at a single location on the display device.
In yet another embodiment, a method of forming an interleaved image series comprises selecting N groups of images, each of the groups of images comprising two or more images, determining a starting image of each of the groups of images, creating an interleaved image series comprising images from each of the selected N groups of images, wherein the images of the interleaved image series are ordered so that an image from each of the N groups of images is included in each sequential Nth group of images, and providing the interleaved image series to a user interface for sequential display in a predetermined location of a display device.
In another embodiment, a method of viewing medical images from two or more image series on a display device comprises: selecting a first image series comprising two or more medical images of a first portion of a human anatomy; selecting at least one comparison image series, each of the comparison image series comprising two or more medical images of the first portion of the human anatomy; interleaving images of the first image series and the comparison image series in order to form an interleaved image series such that the interleaved images series is ordered to include a single image of each of the comparison images series after each single image of the first image series; and sequentially displaying the images of the interleaved image series at a single location on the display device, wherein the method is performed by a computing system comprising one or more computing devices.
According to an aspect, the single location on the display device comprises a comparison pane.
According to another aspect, the method further comprises selecting a starting image in one or more of the first and comparison image series, wherein the starting images indicate respective first images from the one or more first and comparison image series that are to be included in the interleaved image series.
According to yet another aspect, selecting the at least one comparison image series is performed by the computing device based on predefined comparison rules.
According to another aspect, the predefined comparison rules indicate that image series with substantially the same name as a name of the first image series are selected for comparison with the first image series.
According to yet another aspect, the method further comprises: displaying a list of image series identifiers associated with one or more image series that satisfy the comparison rules; and selecting one or more of the image series identifiers, wherein the image series associated with the selected image series identifiers are selected as comparison image series.
According to another aspect, the sequentially displaying is controlled by an input device coupled to the computing device so that when a predetermined input is received by the computing device, the image displayed at the single location is replaced by an adjacent image in the interleaved image series.
According to yet another aspect, the predetermined input comprises moving a scroll bar on a mouse.
According to another aspect, the method further comprises displaying information regarding the medical image currently displayed at the single location.
According to yet another aspect, the method further comprises selecting a zoom level for each of the medical images in the first image series by adjusting a zoom level on one of the medical images of the first image series.
According to another aspect, the method further comprises selecting an image characteristic of the images in the first image series by adjusting the image characteristic on one of the medical images of the first image series.
According to yet another aspect, the method further comprises selecting a zoom level for each of the images in the comparison image series by adjusting a zoom level on one of the medical images of the respective comparison image series.
According to another aspect, the method further comprises automatically selecting an image characteristic of the images in the first image series according to differences in the image characteristics of the images detected by the computing device.
In yet another embodiment, a method of viewing a series of medical images on a display device comprises: (a) selecting a first image series for viewing, the first image series comprising a plurality X of medical images; (b) selecting a second image series for viewing, the second image series comprising a plurality Y of medical images; (c) displaying at a predetermined location on the display device a Nth image of the first image series; (d) replacing the Nth image of the first image series with a Mth image of the second image series at the predetermined location; (e) incrementing or decrementing N and M; and (f) repeating steps (c) to (f), wherein the method is performed by a computing system comprising one or more computing devices.
According to an aspect, the method further comprises displaying a user interface on the display device, wherein the predetermined location comprises a comparison pane within the user interface.
According to another aspect, the values of N and M are selected by a user of the computing device.
According to yet another aspect, the medical images comprise at least one of X-ray, mammogram, MRI, radiograph, computed tomography, magnetic resonance imaging, Ultrasound, positron emission tomography scan, angiogram, and nuclear scan images.
According to another aspect, the method further comprises: selecting a third image series for viewing, the third image series comprising a plurality Z of medical images; replacing the Mth image of the second image series with a Lth image of the third image series at the predetermined location; and incrementing or decrementing L.
According to yet another aspect, timing of the actions of displaying and replacing is controlled by an input device to the computing system, wherein the user may determine a speed at which images from the first and second image series are alternatively displayed at the predetermined location using the input device.
According to another aspect, the input device comprises at least one of: a mouse, a trackball, and a keyboard.
In another embodiment, a computerized method comprises: transmitting from a computing device to a display device data associated with two or more image panes, each depicting an image from a respective image series; receiving at the computing device selections of at least two image panes from an input device, in response to said receiving, interleaving by the computing device the images of the image series associated with the selected at least two selected image panes in order to form an interleaved image series; and initiating by the computing device display of an image of the interleaved image series in a comparison pane on the display device.
According to an aspect, a user of the computing device selects the image panes by clicking a mouse button after moving a pointer onto each of the image panes.
According to another aspect, a user of the computing device selects the image panes by clicking a mouse button after moving a pointer onto a border between the image panes.
According to yet another aspect, the comparison pane covers substantially all of a display area of the display device.
According to another aspect, the comparison pane is sized substantially equal to a size of the image panes.
According to yet another aspect, the displaying action comprises displaying from 2 to 40 image panes.
In yet another embodiment, a method of viewing medical images from two or more image series on a display device coupled to a computing device, wherein the method is performed by the computing device comprises: automatically selecting at least two comparison image series based on predefined user rules; interleaving images of the at least two comparison image series in order to form an interleaved image series; and sequentially displaying the images of the interleaved image series in a comparison pane on the display device.
According to an aspect, the user rules define criteria for selecting image series for comparison.
According to another aspect, the user rules include criteria related to DICOM header file information associated with medical images, wherein comparison image series are selected according to DICOM header file information.
In yet another embodiment, a method of viewing medical images from two or more image series on a display device, wherein the method is performed by a computing system comprising one or more computing devices comprises: selecting a first image series comprising two or more medical images; selecting at least one comparison image series, each of the comparison image series comprising two or more medical images; interleaving images of the first image series and the comparison image series in order to form two or more interleaved image series; and sequentially displaying the images of each of the interleaved image series in corresponding two or more comparison panes that are concurrently displayed on the display device.
According to an aspect, images in each respective interleaved image series are each of a particular projection so that a first interleaved image series comprises images of a first projection and a second interleaved image series comprises images of a second projection.
In another embodiment, a computing system comprises: an electronic display device; an input device; a non-transitory computer-readable storage medium configured to store software instructions; and one or more computing devices in communication with the electronic display device, the input device, and the non-transitory computer-readable medium, the one or more computing devices configured to execute the software instructions in order to: receive, at the computing system, a selection of a first image series comprising two or more medical images; receive, at the computing system, a selection of a second image series comprising two or more medical images obtained at a different time than the first image series; determine a first medical image of the first image series and a second medical image of the second image series that depict a same anatomical position; display the first medical image of the first image series at a first position on the electronic display device of the computing system; and in response to receiving a first command from a user of the computing system via the input device, replace at the same first position on the display the first medical image with the second medical image of the second image series; and in response to receiving a second command from the user of the computing device via the input device, replace the second medical image with the first medical image.
According to an aspect, the first medical image of the first image series is not an initial image of the first image series and/or the second medical image of the second image series is not an initial image of the second image series.
According to another aspect, the first command is triggered in response to input from one or more of a keyboard, a mouse, or a touchpad.
According to yet another aspect, the computing system further comprises: wherein the first command is triggered by the user pressing a button of a mouse or other input device and the second command is triggered by the user releasing the button.
In yet another embodiment, a method comprises: receiving, at a computing device, a selection of a first image series comprising two or more medical images; receiving, at the computing device, a selection of a second image series comprising two or more medical images obtained at a different time than the first image series; displaying a first medical image of the first image series at a first position on a display device of the computing device; in response to receiving a first command from a user of the computing device via an input device, replacing at the same first position on the display, by the computing device, the first medical image with a second medical image of the second image series; and in response to receiving a second command from the user of the computing device via the input device, replacing the second medical image with the first medical image, wherein the first command is triggered by the user moving a scroll wheel of a mouse or other input device in a first direction and the second command is triggered by the user moving the scroll wheel in a second direction that is opposite the first direction.
According to an aspect, the first image series is acquired at a first time associated with a first exam of a patient and the second image series is acquired at a later time associated with a second exam of the patient.
According to another aspect, the first images series and the second image series are related so that when images of the image series are compared, meaningful distinctions may be detected.
According to yet another aspect, the first image series and second image series are acquired using a common type of imaging device.
According to another aspect, the first image series and second image series are acquired using different types of imaging devices.
In another embodiment, a method comprises: displaying, by a computing device comprising hardware, a medical image of a first image series at a first position on a display device of the computing device; in response to receiving a first command from a user of the computing device via an input device, replacing, by the computing device, the medical image of the first image series acquired during a first time period with a comparison medical image of a second image series acquired during a second time period different than the first time period; and in response to receiving a second command from the user via the input device, replacing, by the computing device, the comparison medical image of the second image series with the medical image of the first image series, wherein the first command is triggered by the user moving a scroll wheel of the input device in a first direction and the second command is triggered by the user moving the scroll wheel in a second direction that is opposite the first direction.
According to an aspect, at least one of the first image series or the second image series is automatically selected by the computing device.
According to another aspect, at least one of the first image series or the second image series is automatically selected based on DICOM header information associated with the image series.
According to yet another aspect, at least one of the first image series or the second image series is automatically selected based on filenames associated with the image series.
According to another aspect, the comparison medical image is selected from a plurality of images in the second image series such that the comparison image and the medical image of the first image series depict the same anatomical position.
According to yet another aspect, the computing device automatically selects the comparison medical image.
According to another aspect, the computing device receives an indication of one or more of the medical image and/or the comparison medical image from the user of the computing device.
According to yet another aspect, a first value associated with a first image characteristic of the medical images of the first image series is different than a second value associated with the first image characteristic of the second image series.
According to another aspect, the first image characteristic comprises one or more of size, rotation, location, zoom level, cropping, or color characteristics.
According to yet another aspect, the first value associated with the first image characteristic of the first image series or the second value associated with the first image characteristic of the second image series is automatically adjusted such that the first value and the second value are equivalent.
According to another aspect, the method further comprises receiving input from the user indicating adjustments to at least a centering and a window level of one or more of the medical image of the first image series or the comparison medical image of the second image series.
According to yet another aspect, one or more values associated with image characteristics of the comparison medical image are adjusted in response to user input via the input device.
According to another aspect, the one or more values associated with the image characteristics that are adjusted for the comparison medical image are applied to at least some other images of the second image series.
According to yet another aspect, one or more values associated with image characteristics of images of the first and/or second image series are automatically adjusted so that similar anatomical features depicted in images of the first and second image series are aligned.
According to another aspect, the first command comprises positioning a cursor near a border between the first medical image of the first image series and the second medical image of the second image series.
In yet another embodiment, a non-transitory computer-readable medium storing software code is disclosed that is configured for execution by a computing system in order to cause the computing system to perform operations including: displaying a medical image of a first image series at a first position on a display device of the computing device; displaying a medical image of a second image series at a second position on the display device, wherein the first image series and the second image series were obtained at different times; in response to receiving a first command from a user of the computing device via an input device: replacing the medical image of the first image series with a comparison medical image of a third image series; and replacing the medical image of the second image series with a comparison medical image of a fourth image series; and in response to receiving a second command from the user via the input device: replacing the comparison medical image of the third image series with the medical image of the first image series; and replacing the comparison medical image of the fourth image series with the medical image of the second image series, wherein the first command is triggered by the user moving a scroll wheel of the input device in a first direction and the second command triggered by the user moving the scroll wheel in a second direction that is opposite the first direction.
According to an aspect, the software code is further configured for execution in order to cause the computing system to perform operations including: selecting the first and third image series for comparison based on one or more similarities in DICOM information of files associated with the first and third image series.
According to another aspect, the first medical image and the second medical image are each of a same modality such that differences between the first medical image and the second medical image are detectable by the user viewing the replacement of the first medical image by the second medical image.
According to yet another aspect, the first image series is acquired at a first time associated with a first exam of a patient and the second image series is acquired at a later time associated with a second exam of the patient.
According to another aspect, the one or more computing devices are further configured to execute the software instructions in order to: automatically adjust one or more characteristics of the first medical image and/or the second medical image such that common anatomical features shown in each of the first and second medical images are aligned.
According to yet another aspect, the first medical image of the first image series is not an initial image of the first image series and/or the second medical image of the second image series is not an initial image of the second image series.
According to another aspect, the method further comprises: automatically adjusting, by the computing device, a value associated with a particular characteristic of each of the first medical image and the second medical image such that the values are equivalent.
In another embodiment, a computing system comprises: a display device configured to render images for viewing by one or more users; an input device configured to receive input from a user of the computing system; one or more hardware processors configured to execute software code in order to direct display of images on the display device; and a module comprising software code stored on one or more non-transitory computer readable mediums, the software code configured for execution by the one or more processors in order to: determine an image of a first image series and a comparison image of a second image series that depict a same anatomical position; cause the display device to display the image of the first image series at a first position; and in response to detecting a predetermined input from the user of the computing device via the input device, cause the display device to replace the image of the first image series with the comparison image of the second image series, wherein images of the first image series and images of the second image series were generated by imaging equipment at different times, on different dates, and/or during different time periods.
In yet another embodiment, a non-transitory computer readable medium storing software code configured for execution by a computing system in order to cause the computing system to perform operations is disclosed, the software code comprising: a first module configured to receive input from a user of the computing system via one or more input devices and to detect a predetermined input from the user; a second module configured to: determine an image of a first image series and a comparison image of a second image series that depict a same anatomical position; cause a display device of the computing system to display the image of the first image series at a first position on the display device; and in response to the first module detecting the predetermined input, cause the display device of the computing system to replace the image of the first image series with the comparison image of the second image series, wherein the first image series and the second image series were originally generated at different times, on different dates, and/or during different time periods.
In another embodiment, a method comprises: receiving, at a computing system, at least a first plurality of images comprising a first image series and a second plurality of images comprising a second image series, wherein the first plurality of images and the second plurality of images together comprise a set of images; determining, by the computing system, at least a first characteristic associated with each of the images of the set of images; determining, by the computing system, at least a second characteristic associated with each of the images of the set of images; determining, by the computing system, an ordering of the images of the set of images based on: first: values of the first characteristic associated with respective images, and second: values of the second characteristic associated with respective images; displaying at a location of a display a first image of the set of images that is first in the determined ordering of the images; and in response to input from a user of the computing system indicating a direction of movement within the set of images, replacing the first image or other image displayed at the same location of the display with another image that is adjacent to the first image or other image displayed at the same location of the display based on the determined ordering of the images.
According to an aspect, the first characteristic comprises a DICOM header information item.
According to another aspect, the first characteristic comprises at least one of a type of image, an area imaged, a clinical indication, a source of image, a display device, or a user.
According to yet another aspect, the first characteristic comprises at least one of brightness, contrast, size, opacity map, rotation, location, zoom level, cropping, morphing, or color.
According to another aspect, the first characteristic comprises an anatomical position depicted by the image.
According to yet another aspect, the second characteristic comprises a time at which the image was obtained.
According to another aspect, the first characteristic comprises at least one of a modality or an application of contrast agent.
According to yet another aspect, the second characteristic comprises a time at which the image was obtained.
According to another aspect, the method further comprises: automatically adjusting, by the computing system, at least one characteristic associated with the first image of the set of images such that the first image is better aligned with the second image of the set of images.
According to yet another aspect, the first and second images are adjacent to one another in the ordering of the images.
According to another aspect, an image characteristic of the second image is adjusted such that the information displayed in the first image and the second image more closely matches, and differences between the first image and the second medical image are more easily detectable by the user.
According to yet another aspect, the image characteristic comprises at least one of brightness, contrast, size, opacity map, rotation, location, zoom level, cropping, morphing, or color.
According to another aspect, the computing system is configured to provide the input in response to the user moving a scroll wheel of a mouse or other input device in a first direction.
According to yet another aspect, the method further comprises: in response to a second input from the user of the computing system indicating an opposite direction of movement within the set of images, replacing the second image displayed at the same location of the display with the first image or other image that is adjacent to the second image at the same location of the display based on the determined ordering of the images.
According to another aspect, the computing system is configured to provide the input in response to the user moving a scroll wheel of a mouse or other input device in a first direction.
According to yet another aspect, the computing system is configured to provide the second input in response to the user moving a scroll wheel of a mouse or other input device in a second direction.
According to another aspect, the method further comprises: determining, by the computing system, a first characteristic associated with the first image series; identifying, by the computing system, the second image series for comparison with the first image series, wherein the second image series is identified based on the second image series being associated with a second characteristic have a same value as the first characteristic of the first image series.
According to yet another aspect, the first and second characteristics comprise at least one of DICOM header information items, filenames, or modalities.
In yet another embodiment, a computing system comprises: an electronic display; an input device; a non-transitory computer-readable storage medium configured to store software instructions; and one or more computer processors in communication with the electronic display, the input device, and the non-transitory computer-readable medium, the one or more computer processors configured to execute the software instructions in order to cause the computing system to: receive at least a first plurality of images comprising a first image series and a second plurality of images comprising a second image series, wherein the first plurality of images and the second plurality of images together comprise a set of images; determine at least a first characteristic associated with each of the images of the set of images; determine at least a second characteristic associated with each of the images of the set of images; determine an ordering of the images of the set of images based on: first: values of the first characteristic associated with respective images, and second: values of the second characteristic associated with respective images; display at a location of a display a first image of the set of images that is first in the determined ordering of the images; and in response to input from a user of the computing system indicating a direction of movement within the set of images, replace the first image or other image displayed at the same location of the display with another image that is adjacent to the first image or other image displayed at the same location of the display based on the determined ordering of the images.
Another embodiment comprises a method of displaying medical data. The method comprises receiving a plurality of medical images of a first medical examination and receiving at least one user-defined matching rules, at least one of user-defined matching rules identifying selection criteria for the medical images. The method also comprises selecting medical images that satisfy the selection criteria of the user-defined rules, thereby matching medical images according to user-specific rules and receiving at least one user-defined display rule, at least one of user-defined display rules identifying a display preference with respect to selected medical images. The method also comprises displaying the selected medical images according to the identified display preferences, thereby allowing matched medical images to be visually compared and displayed in a manner that is suitable to the user's preferences.
Yet another embodiment comprises a method displaying medical data. The method comprises receiving a plurality of medical images of a first medical examination and receiving a plurality of medical images of a second medical examination. The method also comprises receiving at least one user-defined matching rule, at least one of user-defined matching rules identifying selection criteria for matching the medical images of the first and second medical examinations. The method also comprises selecting medical images that satisfy the selection criteria of the user-defined rules, thereby matching medical images of the first medical examination with medical images of the second examination according to user-specific rules. The method also comprises receiving a plurality of user-defined display rules, at least one of user-defined display rules identifying a display preference with respect to selected medical images. The method also comprises displaying the selected medical images according to the identified display preferences, thereby allowing matched medical images to be visually compared and displayed in a manner that is suitable to the user's preferences.
Another embodiment comprises a system for displaying medical data. The system comprises an electronic device being configured to receive a plurality of medical images of a first medical examination. The electronic device is configured to receive a plurality of user-defined matching rules. At least one of user-defined matching rules identify selection criteria for the medical images. The electronic device is further configured to select medical images that satisfy the selection criteria of the user-defined rules, thereby matching medical images according to user-specific rules. The electronic device is further configured to receive at least one user-defined display rules. At least one of user-defined display rules identify a display preference with respect to selected medical images. The electronic device is further being configured to display the selected medical images according to the identified display preferences, thereby allowing matched medical images to be visually compared and displayed in a manner that is suitable to the user's preferences.
In yet another embodiment, a method of displaying medical data comprises: receiving a plurality of medical images of a first medical examination; receiving a plurality of medical images of a second medical examination; receiving at least one user-defined matching rule, at least one of the user-defined matching rules identifying selection criteria for matching the medical images of the first and second medical examinations; selecting medical images that satisfy the selection criteria of the user-defined rules, thereby matching medical images of the first medical examination with medical images of the second medical examination according to the user-defined matching rules; receiving a plurality of user-defined display rules, at least one of user-defined display rules identifying one or more display preference with respect to selected medical images; and displaying the selected medical images according to the one or more identified display preference, thereby allowing matched medical images to be visually displayed in a manner that is suitable to the user's preferences, wherein at least some of the method is performed by a computing system comprising one or more computing device.
According to an aspect, the medical images of the first medical examination and of the second medical examination are each grouped in one or more image series.
According to another aspect, the method additionally comprises naming the matched medical images according to at least one user-defined naming rule.
According to yet another aspect, the method additionally comprises naming the medical images according to at least one user-defined naming rule prior to the selecting action.
In another embodiment, a system comprises: an electronic device configured to receive a plurality of medical images of a first medical examination, the electronic device being further configured to receive a plurality of medical images of a second medical examination, the electronic device being further configured to receive a plurality of user-defined matching rules, at least one of user-defined matching rules identifying selection criteria for matching the medical images of the first and second medical examinations, the electronic device being further configured to select medical images that satisfy the selection criteria of the user-defined rules, thereby matching medical images of the first medical examination with medical images of the second medical examination according to the user-defined matching rules, the electronic device being further configured to receive a plurality of user-defined display rules, at least one of the user-defined display rules identifying a display preference with respect to selected medical images, and the electronic device being further configured to display the selected medical images according to the identified display preference[[s]], thereby allowing matched medical images to be visually displayed in a manner that is suitable to the user's preferences.
According to an aspect, the medical images of the first medical examination and of the second medical examination are each grouped in one or more image series.
In yet another embodiment, a method of displaying medical data comprises: receiving a plurality of medical images of a first medical examination; receiving at least one user-defined matching rule including at least one user-defined matching rule identifying selection criteria for selecting medical images; selecting medical images that satisfy the selection criteria of the user-defined matching rules, thereby matching medical images according to user-specific rules; receiving at least one user-defined display rule including at least one user-defined display rule identifying a display preference with respect to the selected medical images; and displaying the selected medical images according to the identified display preference, thereby allowing the matched medical images to be visually compared and displayed in a manner that is suitable to user preferences, wherein at least some of the actions are performed by a computer system having one or more processors.
According to an aspect, the method additionally comprises naming the matched medical images according to at least one user-defined naming rule.
According to another aspect, the method additionally comprises naming the medical images according to at least one user-defined naming rule prior to said selecting.
According to yet another aspect, the medical images are grouped in one or more image series.
According to another aspect, the method additionally comprises: receiving a second plurality of medical images of a second medical examination; and selecting medical images of the first medical examination and the second medical examination that satisfy the selection criteria of the user-defined rules, thereby allowing a user to match medical images of the first medical examination with medical images of the second medical examination according to user-specific rules.
According to yet another aspect, the method additionally comprises: receiving at least one second user-defined display rule, the second user-defined display rule identifying a display preference with respect to matched medical images of the first and second examinations; and displaying the matched medical images in accordance with the second user-defined display rules.
According to another aspect, the method additionally comprises displaying the matched medical images in accordance with the user-defined display rules.
According to yet another aspect, the method additionally comprises: receiving a plurality of second user-defined matching rules, at least one of the user-defined matching rules identifying second selection criteria for matching the selected medical images of the first examination with medical images of a second medical examination; and selecting medical images of the first medical examination and the second medical examination that satisfy the second selection criteria of the user-defined rules, thereby allowing a user to match medical images of the first medical examination with medical images of the second medical examination according to user-specific rules.
According to another aspect, the method additionally comprises: receiving a plurality of second user-defined display rules, at least one of the second user-defined display rules identifying a display preference with respect to matched medical images of the first and second examinations; and displaying the matched medical images in accordance with the second user-defined display rules.
According to yet another aspect, the method additionally comprises displaying the matched medical images in accordance with the user-defined display rules.
In yet another embodiment, a system for displaying medical data comprises: an electronic device configured to receive a plurality of medical images of a first medical examination, the electronic device being further configured to receive a plurality of user-defined matching rules, at least one of the user-defined matching rules identifying selection criteria, the electronic device being further configured to select medical images that satisfy the selection criteria of the user-defined matching rules, thereby matching medical images according to user-specific rules, the electronic device being further configured to receive a plurality of user-defined display rules, at least one of the user-defined display rules identifying a display preference with respect to selected medical images; and the electronic device being further configured to display the selected medical images according to the identified display preferences, thereby allowing matched medical images to be visually compared and displayed in a manner that is suitable to user preferences.
According to an aspect, the medical images are grouped in one or more image series.
According to another aspect, the electronic device is configured to receive a second plurality of medical images of a second medical examination and select medical images of the first medical examination and the second medical examination that satisfy the selection criteria of the user-defined rules, thereby allowing a user to match medical images of the first medical examination with medical images of the second medical examination according to user-specific rules.
According to yet another aspect, the electronic device is further configured to receive a plurality of second user-defined display rules, at least one of the second user-defined display rules identifying a display preference with respect to matched medical images of the first and second examinations; and wherein the electronic device is further configured to display the matched medical images in accordance with the second user-defined display rules.
According to another aspect, the electronic device is further configured to display the matched medical images in accordance with the user-defined display rules.
According to yet another aspect, the electronic device is configured to receive a plurality of second user-defined matching rules, at least one of the user-defined matching rules identifying second selection criteria for matching the selected medical images of the first examination with medical images of a second medical examination; and wherein the electronic device is configured to select medical images of the first medical examination and of second medical examination that satisfy the second selection criteria of the user-defined rules, thereby allowing a user to match medical images of the first medical examination with medical images of the second medical examination according to user-specific rules.
According to another aspect, the electronic device is configured to receive a plurality of second user-defined display rules, at least one of the second user-defined display rules identifying a display preference with respect to matched medical images of the first and second examinations, and wherein the electronic device is configured to display matched medical records in accordance with the second user-defined display rules.
According to yet another aspect, the electronic device is configured to display matched medical records in accordance with the user-defined display rules.
In another embodiment, a computing system comprises: one or more hardware computer processors; and one or more software modules configured for execution by the one or more computer processors to cause the computing system to: in response to user input, automatically: select a first image series and a second image series each associated with a particular patient and including images of a particular modality; determine an order for display of images of the first image series and the second image series, wherein the order is based on anatomical positions of the patient depicted in respective images; and display images of the first image series and the second image series, one at a time in a single image pane, in the determined order.
According to an aspect, the order is further based on one or more of date or time of acquisition of respective displayed images, laterality of images, or view of images.
According to another aspect, the user input is received from an input device in communication with the computing system.
According to yet another aspect, the one or more software modules are further configured to cause the computing system to: select images of the first image series and the second image series each associated with a different particular modality; determine a second order for display of the selected images of the first image series and the second image series, wherein the second order is based on anatomical positions of the patient depicted in respective selected images; and display the selected images of the first image series and the second image series, one at a time in a single second image pane, in the determined second order.
According to another aspect, the computing system further comprises: registering, by the computing system, images of different image series that are displayed in the single image pane.
According to yet another aspect, said registering is based on one or more of aligning centers of images in the different image series or linking scrolling between images displayed in different image panes.
In yet another embodiment, a computing system comprises: one or more hardware computer processors; and one or more software modules configured for execution by the one or more computer processors to cause the computing system to: in response to user input, automatically: select a first image series and a second image series each associated with a particular patient and including images of a particular modality, wherein the first image series and the second image series together comprise a matched set; determine an order for display of images of the matched set, wherein the images of the first image series and a second image series are ordered for progressive display of the matched set; and display images of the matched set, one at a time in a single image pane, in the determined order.
According to an aspect, the order is based on one or more of date or time of acquisition of respective displayed images, laterality of images, or view of images.
According to another aspect, the user input is received from an input device in communication with the computing system.
According to yet another aspect, the one or more software modules are further configured to cause the computing system to: select images of the first image series and the second image series each associated with a different particular modality, wherein the selected images of the first image series and the second image series together comprise a second matched set; determine a second order for display of the selected images of the second matched set, wherein the selected images of the first image series and the second image series are ordered for progressive display of the second matched set; and display the selected images of the second matched set, one at a time in a single second image pane, in the determined second order.
In another embodiment, a computing system comprises: one or more hardware computer processors; and one or more software modules including computer executable instructions and configured for execution by the one or more computer processors to cause the computing system to: access images of one or more image series; identify images of the one or more image series that are similar based on similarities in one or more of name, modality, plane, contrast, and/or width of respective images; determine a display order of a first set of images that are determined to be similar, wherein the display order is determined based on a first DICOM attribute of respective images of the first set of images; and generate a user interface for display of the first set of images in the determined display order.
According to an aspect, the first DICOM attribute includes at least one of orientation, contrast use, thickness of slices, field of view, or MRI tissue contrast weighting.
According to another aspect, the first DICOM attribute of respective images is contained in a header file of the respective images.
According to yet another aspect, the first set of images includes only images from a first image series.
According to another aspect, the first set of images includes images from at least two image series.
According to yet another aspect, the first set of images is displayed in the determined display order in a single image pane.
According to another aspect, the first set of images are displayed sequentially in the determined display order in response to a user input.
In yet another embodiment, a method comprises: accessing, by a computing system having one or more hardware processors executing computer executable instructions, images of one or more image series; identifying, by the computing system, images of the one or more image series that are similar based on similarities in one or more of name, modality, plane, contrast, and/or width of respective images; determining, by the computing system, a display order of a first set of images that are determined to be similar, wherein the display order is determined based on a first DICOM attribute of respective images of the first set of images; and generating, by the computing system, one or more user interfaces to display the first set of images in the determined display order.
According to an aspect, the method further comprises: determining, by the computing system, a second display order of the first set of images, wherein the second display order is determined based on a second DICOM attribute of respective images of the first set of images; and displaying, on an electronic display of the computing system or another computing device, the first set of images in the determined second display order.
According to another aspect, the method further comprises: identifying, by the computing system, a second set of images of the one or more image series that are similar based on second similarities in one or more of name, modality, plane, contrast, and/or width of respective images; determining, by the computing system, a second display order of the second set of images, wherein the second display order is determined based on a second DICOM attribute of respective images of the second set of images; and updating, by the computing system, the one or more user interfaces to display the second set of images in the determined second display order.
According to yet another aspect, the first set of images includes only images from a first image series.
According to another aspect, the first set of images includes images from at least two image series.
According to yet another aspect, the first set of images is displayed in the determined display order in a single image pane.
In another embodiment, a non-transitory computer-readable storage medium storing software code is disclosed that, when executed by a computer system, configures the computer system to perform operations comprising: accessing images of one or more image series; identifying images of the one or more image series that are similar based on similarities in one or more of name, modality, plane, contrast, and/or width of respective images; determining a display order of a first set of images that are determined to be similar, wherein the display order is determined based on a first DICOM attribute of respective images of the first set of images; and providing the first set of images for display in the determined display order.
According to an aspect, the first set of images includes only images from a first image series.
According to another aspect, the first set of images includes images from at least two image series.
According to yet another aspect, the first set of images is displayed in the determined display order in a single image pane.
According to another aspect, the first set of images are displayed sequentially in the determined display order in response to a user input.
According to another embodiment, a database computing system for processing digital images comprises: an electronic display; an input device; and one or more computer processors in communication with the electronic display, the input device, and a non-transitory computer-readable medium storing software modules including software instructions that are executable by the one or more computer processors, the software modules including at least: a user input module configured to: receive an indication of a selection of a first image series of a first medical exam associated with a patient, wherein the first image series includes a first plurality of medical images; and a rules engine module configured to access one or more matching rules and a first one or more sorting rules, the first one or more sorting rules indicating a first attribute and a second attribute, the rules engine module further configured to: determine, based on the one or more matching rules, a second image series of a second medical exam for comparison with the first image series, wherein the second image series includes a second plurality of medical images, and wherein the second medical exam is also associated with the patient and was acquired at a time different from a time that the first medical exam was acquired; sort a set of images including the images of both the first image series and the second image series to determine a sorted set of images by at least: sorting the set of images based on the first attribute; and for each sorted group of images having a common first attribute, further sorting the group of images based on the second attribute such that the sorted set of images are sorted based on both the first and second attributes; wherein the user input module is further configured to: receive a first user input; wherein the rules engine module is further configured to: in response to the first user input, access a second one or more sorting rules indicating a third attribute and a fourth attribute; and re-sort the set of images to determine a re-sorted set of images by at least: sorting the set of images based on the third attribute; and for each sorted group of images having a common third attribute, further sorting the group of images based on the fourth attribute such that the re-sorted set of images are sorted based on both the third and fourth attributes.
According to an aspect, the first and fourth attributes are the same, and the second and third attributes are the same.
According to another aspect, the software modules further include at least: a user interface module configured to: generate user interface data useable for displaying a user interface on the electronic display, the user interface including a first image of the sorted set of images in a first location; and in response to a second user input indicating a direction of movement within the sorted set of images, update the user interface data such that, in the user interface, the first image is replaced with a second image of the sorted set of images that is adjacent to the first image in the sorted set of images, wherein the second image is included in the first location.
According to yet another aspect, the set of images is re-sorted in response to the first user input.
According to another aspect, the user interface module is further configured to: further in response to the first user input, update the user interface data such that the user interface includes a third image of the re-sorted set of images in the first location; and in response to a third user input indicating a direction of movement within the re-sorted set of images, update the user interface data such that, in the user interface, the third image is replaced with a fourth image of the re-sorted set of images that is adjacent to the third image in the re-sorted set of images, wherein the fourth image is included in the first location.
According to yet another aspect, the rules engine module is further configured to: adjust one or more image characteristics of images of the sorted set of images to reduce artifactual differences between images for comparison.
According to another aspect, the rules engine module is further configured to: adjust the one or more image characteristics of images of the sorted set of images based on a landmark identified in at least two of the images of the sorted set of images.
According to yet another aspect, the rules engine module is further configured to: adjust one or more image characteristics of images of the re-sorted set of images to reduce artifactual differences between images for comparison
According to yet another embodiment, a database computing system for processing digital images comprises: an electronic display; an input device; a non-transitory computer-readable storage medium configured to store software instructions; and one or more computer processors in communication with the electronic display, the input device, and the non-transitory computer-readable medium, the one or more computer processors configured to execute the software instructions in order to cause the computing system to: receive an indication of a selection of a first image series of a first medical exam associated with a patient, wherein the first image series includes a first plurality of medical images; select from a plurality of decision support image series, based on one or more rules and at least one characteristic of the patient, a clinical decision support image series including a second plurality of medical images; access a first one or more sorting rules indicating a first attribute and a second attribute; and determine a set of images to be sorted, the set of imagines including the first plurality of images of the first image series and the second plurality of images of the clinical decision support image series; sorting the set of images based on the first attribute; and further sorting the set of images based on the second attribute such that the sorted set of images are sorted based on both the first and second attributes.
According to an aspect, the clinical decision support image series includes at least one of: reference medical images, or medical images of another patient that has at least one characteristic matching the at least one characteristic of the patient.
According to another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: in response to a first user input, access a second one or more sorting rules indicating a third attribute and a fourth attribute; and re-sort the set of images to determine a re-sorted set of images by at least: sorting the set of images based on the third attribute; and further sorting the set of images based on the fourth attribute such that the sorted set of images are sorted based on both the third and fourth attributes.
According to yet another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: in response to receiving the indication of the selection, automatically analyze data of the first medical exam; determine the first medical exam is a candidate for clinical decision support; and prompt a user to select to compare at least the first image series with clinical decision support data, wherein the clinical decision support image series is determined in response to a second input from the user selecting to compare at least the first image series with clinical decision support data.
According to another embodiment, a database computing system for processing digital images comprises: an electronic display; an input device; a non-transitory computer-readable storage medium configured to store software instructions; and one or more computer processors in communication with the electronic display, the input device, and the non-transitory computer-readable medium, the one or more computer processors configured to execute the software instructions in order to cause the computing system to: access a first dual-image radiography medical exam of a patient obtained at a first time, wherein the first dual-image radiography medical exam includes a first plurality of image series of a chest of the patient; receive an indication of a selection of a first image series of the first plurality of image series of the first dual-image radiography medical exam; in response to receiving the indication of the selection, determine, based on one or more matching rules, a second image series of a second plurality of image series of a second dual-image radiography medical exam, wherein the second dual-image radiography medical exam is also of the patient and was obtained at a second time different than the first time; access a first one or more sorting rules indicating a first attribute and a second attribute, wherein one of the first attribute and the second attribute is a view type and another of the first attribute and the second attribute is a time of acquisition; and sort a set of images including the images of both the first and second image series to determine a sorted set of images by at least: sorting the set of images based on the first attribute; and further sorting the set of images based on the second attribute such that the sorted set of images are sorted based on both the first and second attributes.
According to an aspect, each of the first plurality of image series and the second plurality of image series include at least one image series of at least one of: a traditional view type, or a bones out view type.
According to another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: in response to a first user input, re-sort the set of images to determine a re-sorted set of images by at least: sorting the set of images based on the second attribute; and further sorting the set of images based on the first attribute such that the sorted set of images are sorted based on both the second and first attributes.
According to yet another embodiment, a database computing system for processing digital images comprises: an electronic display; an input device; a non-transitory computer-readable storage medium configured to store software instructions; and one or more computer processors in communication with the electronic display, the input device, and the non-transitory computer-readable medium, the one or more computer processors configured to execute the software instructions in order to cause the computing system to: receive an indication of a selection of a first medical image of a first image series of a first medical exam associated with a patient, wherein the first image series includes a first plurality of medical images; determine, based on one or more matching rules, a comparison image series of a comparison medical exam for comparison with the first image series, wherein the comparison image series includes a second plurality of comparison images, and wherein the comparison medical exam is also associated with the patient and was acquired at a time different than the first medical exam; identify, based on one or more attributes of the first medical image, a first comparison image of the comparison image series for comparison with the first medical image; generate user interface data useable for displaying a user interface on the electronic display, the user interface including at least: the first medical image of the first image series displayed in a first image pane in a first location; and the first comparison image of the second image series displayed in a second image pane in a second location; in response to a first user input indicating a direction of movement within the first image series: update the user interface data such that, in the user interface, the first medical image is replaced with a second medical image of the first image series that is adjacent to the first medical image in the first image series, wherein the second medical image replaces the first medical image in the first image pane; identify, based on one or more attributes of the second medical image, a second comparison image of the comparison image series for comparison with the second medical image; and further update the user interface data such that, in the user interface, the first comparison image is replaced with the second comparison image, wherein the second comparison image replaces the first comparison image in the second image pane; and in response to a second user input indicating selection of a second image series of the first medical exam: determine, based on one or more attributes of the second medical image, a first medical image of the second image series for display; update the user interface data such that, in the user interface, the second medical image is replaced with the first medical image of the second image series, wherein the first medical image of the second image series replaces the second medical image of the first image series in the first image pane; identify, based on one or more attributes of the second image series, a second comparison image series of the comparison medical exam for comparison with the second image series; identify, based on one or more attributes of the first medical image of the second image series, a first comparison image of the second comparison image series for comparison with the first medical image of the second image series; and further update the user interface data such that, in the user interface, the second comparison image of the comparison image series is replaced with the first comparison image of the second comparison image series, wherein the first comparison image of the second comparison image series replaces the second comparison image of the comparison image series in the second image pane.
According to an aspect, the first comparison image of the comparison image series is further identified based on one or more matching rules.
According to another aspect, the second comparison image of the comparison image series is further identified based on one or more matching rules.
According to yet another aspect, the first comparison image of the second comparison image series is further identified based on one or more matching rules.
According to another aspect, the first location of the first image pane and the second location of the second image pane are adjacent to one another.
According to another embodiment, a database computing system for processing digital images comprises: an electronic display; an input device; a non-transitory computer-readable storage medium configured to store software instructions; and one or more computer processors in communication with the electronic display, the input device, and the non-transitory computer-readable medium, the one or more computer processors configured to execute the software instructions in order to cause the computing system to: display, on the electronic display, a first medical image of a first image series of a first medical exam in a first image pane, wherein the first medical exam is associated with a patient; display, on the electronic display, a first medical image of a comparison image series of a second medical exam in a second image pane, wherein the comparison medical exam is also associated with the patient, wherein the second medical exam was acquired at a time different from the first medical exam, and wherein the first medical image of the comparison image series matches the first medical image of the first image series in at least one attribute such that the first medical image of the first image series and first medical image of the comparison image series may be usefully compared for diagnosis; in response to a first user input indicating a change from the first medical image of the first image series to another medical image of the first image series: replace, in the first image pane, the first medical image of the first image series with the another medical image of the first image series; and replace, in the second image pane, the first medical image of the comparison image series with another medical image of the comparison image series, wherein the another medical image of the comparison image series matches the another medical image of the first image series in at least one attribute such that the another medical image of the first image series and the another medical image of the comparison image series may be usefully compared for diagnosis; and in response to a second user input indicating a change from the first image series of the first medical exam to another image series of the first medical exam: replace, in the first image pane, the first medical image of the first image series with the another medical image of the another image series of the first medical exam; and replace, in the second image pane, the first medical image of the comparison image series with another medical image of another image series of the second medical exam, wherein the another medical image of the another image series of the second medical exam matches the another medical image of the another image series of the first medical exam in at least one attribute such that the another medical image of the another image series of the first medical exam and the another medical image of the another image series of the second medical exam may be usefully compared for diagnosis.
According to an aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: in response to a third user input, freeze at least the second image pane such that a medical image displayed in the second image pane is not replaced in response to the first and/or second user inputs.
According to another aspect, the first and second image panes are adjacent to one another.
According to yet another aspect, the second medical exam was acquired before the first medical exam.
According to yet another embodiment, a method comprises: by one or more computer processors configured to execute software instructions: displaying, on an electronic display, a first medical image of a first image series of a first medical exam in a first image pane, wherein the first medical exam is associated with a patient; displaying, on the electronic display, a first medical image of a comparison image series of a second medical exam in a second image pane, wherein the comparison medical exam is also associated with the patient, wherein the second medical exam was acquired at a time different from the first medical exam, and wherein the first medical image of the comparison image series matches the first medical image of the first image series in at least one attribute such that the first medical image of the first image series and first medical image of the comparison image series may be usefully compared for diagnosis; in response to a first user input indicating a change from the first medical image of the first image series to another medical image of the first image series: replacing, in the first image pane, the first medical image of the first image series with the another medical image of the first image series; and replacing, in the second image pane, the first medical image of the comparison image series with another medical image of the comparison image series, wherein the another medical image of the comparison image series matches the another medical image of the first image series in at least one attribute such that the another medical image of the first image series and the another medical image of the comparison image series may be usefully compared for diagnosis; and in response to a second user input indicating a change from the first image series of the first medical exam to another image series of the first medical exam: replacing, in the first image pane, the first medical image of the first image series with the another medical image of the another image series of the first medical exam; and replacing, in the second image pane, the first medical image of the comparison image series with another medical image of another image series of the second medical exam, wherein the another medical image of the another image series of the second medical exam matches the another medical image of the another image series of the first medical exam in at least one attribute such that the another medical image of the another image series of the first medical exam and the another medical image of the another image series of the second medical exam may be usefully compared for diagnosis.
According to an aspect, the method further comprises: by the one or more computer processors configured to execute software instructions: in response to a third user input, freezing at least the second image pane such that a medical image displayed in the second image pane is not replaced in response to the first and/or second user inputs.
According to another aspect, the first and second image panes are adjacent to one another.
According to yet another aspect, the second medical exam was acquired before the first medical exam.
According to another embodiment, a database computing system for processing digital images comprises: an electronic display; an input device; a non-transitory computer-readable storage medium configured to store software instructions; and one or more computer processors in communication with the electronic display, the input device, and the non-transitory computer-readable medium, the one or more computer processors configured to execute the software instructions in order to cause the computing system to: access a plurality of medical images associated with a plurality of medical exams of a patient; analyze the plurality of medical images to determine, for each medical image, respective attributes of the medical images; sort the plurality of medical images into a plurality of image series based on the attributes of the medical images, wherein the plurality of image series includes at least a first image series and a second image series; analyzing the first and second image series to determine pairs of medical images of the respective image series that are related to one another; generate user interface data useable for displaying a user interface on the electronic display, the user interface including at least: a first medical image of the first image series displayed in a first image pane in a first location; and a first medical image of the second image series displayed in a second image pane in a second location; and in response to a first user input indicating a direction of movement within the first image series: update the user interface data such that, in the user interface, the first medical image of the first image series is replaced with a second medical image of the first image series that is adjacent to the first medical image of the first image series in the first image series, wherein the second medical image of the first image series replaces the first medical image in the first image pane; identify, based on one or more attributes of the second medical image of the first image series, a second medical image of the second image series for comparison with the second medical image of the first image series; and further update the user interface data such that, in the user interface, the first medical image of the second image series is replaced with the second medical image of the second image series, wherein the second medical image of the second image series replaces the first medical image of the second image series in the second image pane.
According to an aspect, the one or more computer processors are further configured to execute the software instructions in order to cause the computing system to: in response to a second user input indicating a direction of movement within the first image series: update the user interface data such that, in the user interface, the first medical image of the first image series is replaced with a second medical image of the first image series that is adjacent to the first medical image of the first image series in the first image series, wherein the second medical image of the first image series replaces the first medical image in the first image pane; determine, based on one or more attributes of the second medical image of the first image series, that no medical image of the second image series corresponding to the second medical image of the first image series exists; and further update the user interface data such that, in the user interface, the first medical image of the second image series is replaced with a placeholder in the second image pane.
According to an aspect, the placeholder comprises at least one of a blank image or an absence of an image.
According to an aspect, analyzing the plurality of medical images to determine, for each medical image, respective attributes of the medical images comprises: analyzing the plurality of medical images to determine, for each medical image, respective view types of the medical images; The computing system of claim 14, wherein each image series of the plurality of image series comprises a series of medical images having a common view type
According to an aspect, the plurality of medical images comprise chest radiograph images.
According to an aspect, the analyzing the first and second image series to determine pairs of medical images of the respective image series that are related to one another comprises: analyzing the first and second image series to determine pairs of medical images of the respective image series that are from a same exam and/or acquired at a similar time.
According to an aspect, the one or more computer processors are further configured to execute the software instructions in order to cause the computing system to: determine the first and second locations of the respective first and second image panes based on one or more user preferences.
According to an aspect, the one or more computer processors are further configured to execute the software instructions in order to cause the computing system to: determine an association between the first image pane and the first image series based on one or more user preferences.
According to an aspect, the one or more computer processors are further configured to execute the software instructions in order to cause the computing system to: registering the medical images of the first image series so as to enable efficient comparison of the medical images of the first image series.
According to another embodiment, a database computing system for processing digital images comprises: an electronic display; an input device; a non-transitory computer-readable storage medium configured to store software instructions; and one or more computer processors in communication with the electronic display, the input device, and the non-transitory computer-readable medium, the one or more computer processors configured to execute the software instructions in order to cause the computing system to: generate user interface data useable for displaying a user interface on the electronic display, the user interface including at least a plurality of selectable image thumbnails; and in response to a user input indicating selection of a first image thumbnail of the plurality of selectable image thumbnails: determine a first medical image of a first image series of a first medical exam associated with the first image thumbnail, wherein the first image series includes a first plurality of medical images, and wherein the first medical exam is associated with a patient; determine, based on one or more matching rules, a comparison series of a comparison exam for comparison with the first image series, wherein the comparison series includes a second plurality of medical images, and wherein the comparison medical exam is also associated with the patient and was acquired at a time different than the first medical exam; identify, based on one or more attributes of the first medical image, a comparison medical image of the comparison image series for comparison with the first medical image; and update the user interface data such that the user interface includes at least: the first medical image of the first image series displayed in a first image pane in a first location; and the comparison medical image of the comparison image series displayed in a second image pane in a second location.
According to an aspect, the user input indicating selection of the first image thumbnail of the plurality of selectable image thumbnails comprises a dragging of the first image thumbnail from a first portion of the user interface to a second portion of the user interface.
According to another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: in response to a second user input indicating a request to sort medical images: access a first one or more sorting rules indicating a first attribute and a second attribute; sort a set of images including the first and second plurality of medical images to determine a sorted set of images by at least: sorting the set of images based on the first attribute; and further sorting the set of images based on the second attribute such that the sorted set of images are sorted based on both the first and second attributes; and update the user interface data such that the user interface includes at least: a third image pane overlaying the first and second image panes, wherein a first image of the sorted set of images is displayed in the third image pane.
According to yet another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: in response to a third user input indicating a direction of movement within the sorted set of images, update the user interface data such that, in the user interface, the first image is replaced with a second image of the sorted set of images that is adjacent to the first image in the sorted set of images, wherein the second image is displayed in the third image pane.
According to another aspect, the first and second locations are adjacent to one another.
According to yet another embodiment, a method comprises: by one or more computer processors configured to execute software instructions: generating user interface data useable for displaying a user interface on an electronic display, the user interface including at least a plurality of selectable image thumbnails; and in response to a user input indicating selection of a first image thumbnail of the plurality of selectable image thumbnails: determining a first medical image of a first image series of a first medical exam associated with the first image thumbnail, wherein the first image series includes a first plurality of medical images, and wherein the first medical exam is associated with a patient; determining, based on one or more matching rules, a comparison series of a comparison exam for comparison with the first image series, wherein the comparison series includes a second plurality of medical images, and wherein the comparison medical exam is also associated with the patient and was acquired at a time different than the first medical exam; identifying, based on one or more attributes of the first medical image, a comparison medical image of the comparison image series for comparison with the first medical image; and updating the user interface data such that the user interface includes at least: the first medical image of the first image series displayed in a first image pane in a first location; and the comparison medical image of the comparison image series displayed in a second image pane in a second location.
According to an aspect, the user input indicating selection of the first image thumbnail of the plurality of selectable image thumbnails comprises a dragging of the first image thumbnail from a first portion of the user interface to a second portion of the user interface.
According to another aspect, the method further comprises: by the one or more computer processors configured to execute software instructions: in response to a second user input indicating a request to sort medical images: accessing a first one or more sorting rules indicating a first attribute and a second attribute; sorting a set of images including the first and second plurality of medical images to determine a sorted set of images by at least: sorting the set of images based on the first attribute; and further sorting the set of images based on the second attribute such that the sorted set of images are sorted based on both the first and second attributes; and updating the user interface data such that the user interface includes at least: a third image pane overlaying the first and second image panes, wherein a first image of the sorted set of images is displayed in the third image pane.
According to yet another aspect, the method further comprises: by the one or more computer processors configured to execute software instructions: in response to a third user input indicating a direction of movement within the sorted set of images, updating the user interface data such that, in the user interface, the first image is replaced with a second image of the sorted set of images that is adjacent to the first image in the sorted set of images, wherein the second image is displayed in the third image pane.
According to another aspect, the first and second locations are adjacent to one another.
According to another embodiment, a computer program product comprises a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer processor to cause the computer processor to: generate user interface data useable for displaying a user interface on an electronic display, the user interface including at least a plurality of selectable image thumbnails; and in response to a user input indicating selection of a first image thumbnail of the plurality of selectable image thumbnails: determine a first medical image of a first image series of a first medical exam associated with the first image thumbnail, wherein the first image series includes a first plurality of medical images, and wherein the first medical exam is associated with a patient; determine, based on one or more matching rules, a comparison series of a comparison exam for comparison with the first image series, wherein the comparison series includes a second plurality of medical images, and wherein the comparison medical exam is also associated with the patient and was acquired at a time different than the first medical exam; identify, based on one or more attributes of the first medical image, a comparison medical image of the comparison image series for comparison with the first medical image; and update the user interface data such that the user interface includes at least: the first medical image of the first image series displayed in a first image pane in a first location; and the comparison medical image of the comparison image series displayed in a second image pane in a second location.
According to an aspect, the user input indicating selection of the first image thumbnail of the plurality of selectable image thumbnails comprises a dragging of the first image thumbnail from a first portion of the user interface to a second portion of the user interface.
According to another aspect, the program instructions are executable by a computer processor to further cause the computer processor to: in response to a second user input indicating a request to sort medical images: access a first one or more sorting rules indicating a first attribute and a second attribute; sort a set of images including the first and second plurality of medical images to determine a sorted set of images by at least: sorting the set of images based on the first attribute; and further sorting the set of images based on the second attribute such that the sorted set of images are sorted based on both the first and second attributes; and update the user interface data such that the user interface includes at least: a third image pane overlaying the first and second image panes, wherein a first image of the sorted set of images is displayed in the third image pane.
According to yet another aspect, the program instructions are executable by a computer processor to further cause the computer processor to: in response to a third user input indicating a direction of movement within the sorted set of images, update the user interface data such that, in the user interface, the first image is replaced with a second image of the sorted set of images that is adjacent to the first image in the sorted set of images, wherein the second image is displayed in the third image pane.
According to another aspect, the first and second locations are adjacent to one another.
According to yet another embodiment, a database computing system for processing digital images comprises: an electronic display; an input device; and one or more computer processors in communication with the electronic display, the input device, and a non-transitory computer-readable medium storing software modules including software instructions that are executable by the one or more computer processors in order to cause the one or more computer processors to: access an image series comprising a plurality of images, wherein at least some of the plurality of images comprise multi-frame images; display a first image of the plurality of images at a location on the electronic display; in response to a first input, received via the input device, display a frame of a second image of the plurality of images at the location on the electronic display, wherein the frame of the second image replaces the first image on the electronic display, wherein the second image comprises a multi-frame image; in response to a second input, received via the input device: create a copy of the frame of the second image; and add the copy of the frame of the second image to the image series as a new image; and sort the image series such that the new image is located adjacent to the second image in the image series.
According to an aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: further in response to the second input, add the copy of the frame of the second image to a montage; and add the montage to the image series.
According to another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: further in response to the second input, generate data necessary to create a montage including the frame of the second image and any changes to the frame of the second image indicated by the second input; and add an indication of the montage to the image series.
According to yet another aspect, the second input indicates adding an annotation to the frame of the second image.
According to another aspect, the second input indicates the frame of the second image is a key image.
According to yet another aspect, the second input indicates a change of an image characteristic of the frame of the second image.
According to another embodiment, a database computing system for processing digital images comprises: an electronic display; an input device; and one or more computer processors in communication with the electronic display, the input device, and a non-transitory computer-readable medium storing software modules including software instructions that are executable by the one or more computer processors in order to cause the one or more computer processors to: access an image series comprising a plurality of images, wherein at least some of the plurality of images comprise multi-frame images; display a first image of the plurality of images at a location on the electronic display; in response to a first input, received via the input device, display a frame of a second image of the plurality of images at the location on the electronic display, wherein the frame of the second image replaces the first image on the electronic display, wherein the second image comprises a multi-frame image; in response to a second input, received via the input device: create data including an indication of the frame of the second image and information associated with the second input; and add the data to the image series as an indication of an additional image; and sort the image series such that the additional image is located adjacent to the second image in the image series.
According to an aspect, the information associated with the second input includes at least one of: an indication that the frame of the second image is a key image, an addition of an annotation to the frame of the second image, or an adjustment to an image characteristic of the frame of the second image.
According to another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: in response to a third input, received via the input device, indicating that the additional image is to be displayed: generate the additional image based on the data including the indication of the frame of the second image and the information associated with the second input; and display the additional image in the location on the electronic display.
According to yet another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: further in response to the second input, add a copy of the frame of the second image to a montage; and add the montage to the image series.
According to another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: further in response to the second input, generate data necessary to create a montage including the frame of the second image and any changes to the frame of the second image indicated by the second input; and add an indication of the montage to the image series.
According to yet another embodiment, a database computing system for processing digital images comprises: an electronic display; an input device; and one or more computer processors in communication with the electronic display, the input device, and a non-transitory computer-readable medium storing software modules including software instructions that are executable by the one or more computer processors in order to cause the one or more computer processors to: access an image series comprising a plurality of images, wherein at least some of the plurality of images comprise multi-frame images; display a first image of the plurality of images at a location on the electronic display; in response to a first input, received via the input device: create a copy of the first image; and add the copy of the first image to the image series as a new image; and sort the image series, including the new image, based one or more user preferences or sorting rules.
According to an aspect, sorting the image series comprises: accessing a first one or more sorting rules indicating a first attribute and a second attribute; and sorting the image series by at least: sorting images of the image series based on the first attribute; and further images of the image series based on the second attribute such that the images of the image series are sorted based on both the first and second attributes.
According to another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: further in response to the second input, add the copy of the first image to a montage; and add the montage to the image series.
According to yet another aspect, the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to: further in response to the second input, generate data necessary to create a montage including the first image and any changes to the first image indicated by the second input; and add an indication of the montage to the image series.
According to another aspect, the second input indicates adding an annotation to the first image.
According to yet another aspect, the second input indicates the first image is a key image.
According to another aspect, the second input indicates a change of an image characteristic of the first image.
Additional embodiments of the disclosure are described below in reference to the appended claims, which may serve as an additional summary of the disclosure.
In various embodiments, computer systems are disclosed that comprise one or more hardware computer processors in communication with one or more non-transitory computer readable storage devices, wherein the one or more hardware computer processors are configured to execute the plurality of computer executable instructions in order to cause the computer system to operations comprising one or more aspects of the above-described embodiments (including one or more aspects of the appended claims).
In various embodiments, computer-implemented methods are disclosed in which, under control of one or more hardware computing devices configured with specific computer executable instructions, one or more aspects of the above-described embodiments (including one or more aspects of the appended claims) are implemented and/or performed.
In various embodiments, computer readable storage mediums storing software instructions are disclosed, wherein, in response to execution by a computing system having one or more hardware processors, the software instructions configure the computing system to perform operations comprising one or more aspects of the above-described embodiments (including one or more aspects of the appended claims).
The following drawings and the associated descriptions are provided to illustrate embodiments of the present disclosure and do not limit the scope of the claims. Aspects and many of the attendant advantages of this disclosure will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:
Although certain preferred embodiments and examples are disclosed below, inventive subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and/or uses and to modifications and equivalents thereof. Thus, the scope of the claims appended hereto is not limited by any of the particular embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations may be described as multiple discrete operations in turn, in a manner that may be helpful in understanding certain embodiments; however, the order of description should not be construed to imply that these operations are order dependent. Additionally, the structures, systems, and/or devices described herein may be embodied as integrated components or as separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are achieved by any particular embodiment. Thus, for example, various embodiments may be carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other aspects or advantages as may also be taught or suggested herein.
I. OVERVIEWAs mentioned above, according to various embodiments systems are disclosed that enable a user to more quickly, thoroughly, and efficiently, as compared to previous systems, interact with image data, including medical images, to determine differences between related medical images and evaluate medical images.
As also mentioned above, according to various embodiments a data navigation system is disclosed in which a user may interact with medical images (including two-dimensional images and images rendered from three-dimensional image data) to enable detection of differences between related medical images and evaluation of medical images. In various embodiments described herein, images from one or more series and/or exams are automatically analyzed and sorted. In an embodiment, reference image data is alternatively, or also, sorted with images. The system enables fast and efficient evaluation and comparison of related images as a result of the sorting. For example, sorted images may be rapidly paged through to enable detailed comparison of images and detection of changes (including, e.g., changes in a medical condition of a patient). User interfaces of the system are thus dynamically updated to provide rapid comparison of images. Further, images from multiple series and/or exams, and/or images from other sources, may be automatically sorted by the system according to attributes associated with the images and rules and/or preferences of the user.
In an embodiment, the user may select an image from a series of a first exam, and the system automatically determines and displays one or more comparison images from other image series and/or exams. Images selected for comparison, and/or images that are sorted, may additionally be automatically registered and/or matched to enable more efficient comparison and evaluation by the user. Accordingly, a user may use the systems described herein to more quickly, thoroughly, and efficiently interact with medical images, as compared to previous systems.
In various embodiments, systems and methods are disclosed for matching related medical images and/or medical image series from multiple exams, automatically displaying medical images in particular arrangements, and automatically sorting medical images from related exams. In one example, a user selects a medical image, and the system automatically identifies related images and/or medical image exams from 2, 3, or 4 (or more) other exams and displays the images next to one another in a grid arrangement, and/or sorts the images and displays them sequentially in an image pane.
In an embodiment, a user interface of the system includes one or more image panes in which medical images may be displayed. As described in further detail below, such a user interface may provide one or more comparison panes on a display device in which images from multiple image series are sequentially displayed.
In various embodiments, medical images may be reconstructed and/or rendered from 3D or volumetric image data using methods including multiplanar reformation/reconstruction (MPR), maximum intensity projection (MIP), and/or the like. Such rendered images may be rendered to match comparison images from other image series and/or exams so as to enable more efficient comparison and evaluation by the user.
Rendering and/or reconstruction of images from 3D of volumetric image data can be a computationally intensive task that requires significant processing power. Three-dimensional medical image data sets can be on the order of multiple gigabytes in size, therefore requiring efficient computer algorithms to generate human-useable images and other information. Typically, such 3D data sets are acquired by CT, MRI, and/or other similar modality. Volume rendering (e.g., rendering 2D projections of 3D data sets) may be accomplished by, e.g., direct volume rendering, maximum intensity projection, and/or the like, and may require one or more computational optimization techniques to be fast enough to be useful. Other reconstruction processes (e.g., rendering 2D slices of 3D data sets) include, e.g., multiplanar reconstruction. These processes also require processing of large 3D data sets via optimized rendering and/or reconstruction algorithms on computer processors.
While the use of medical imaging data (e.g., medical images) is described in the example embodiments herein, in various embodiments the systems and methods described may be used for display of, and interaction with, non-medical information, for example, seismic information, weather data, and/or financial data, among others.
Additionally, while the examples herein describe the use of information acquired from a physical object such as a patient, the systems and methods described may be used to display information obtained or generated in other ways, for example, information calculated in a simulation (for example, a financial simulation, and/or a physics simulation, among others). The systems and methods described herein may be used for display of any type of information that can be represented on a digital display.
As described above, various embodiments of the present disclosure provide improvements to various technologies and technological fields, including medical image interaction technology (e.g., Picture Archiving and Communication Systems, Electronic Medical Record systems, and/or the like). Additionally, various embodiments of the present disclosure are inextricably tied to computer technology. In particular, various embodiments rely on detection of user inputs via graphical user interfaces, calculation of updates to displayed electronic data based on those user inputs, automatic processing of related electronic medical images, and/or presentation of the updates to displayed medical images via interactive graphical user interfaces. Such features and others (e.g., generation of 2D medical images from a 3D imaging volume and/or other 2D images, such as at automatically selected or user-selected planes) are intimately tied to, and enabled by, computer technology, and would not exist except for computer technology. For example, the interactions with displayed data described below in reference to various embodiments cannot reasonably be performed by humans alone, without the computer technology upon which they are implemented. Further, the implementation of the various embodiments of the present disclosure via computer technology enables many of the advantages described herein, including more efficient interaction with, and presentation of, various types of electronic image data.
Embodiments of the disclosure will now be described with reference to the accompanying figures, wherein like numerals refer to like elements throughout. The terminology used in the description presented herein is not intended to be interpreted in any limited or restrictive manner, simply because it is being utilized in conjunction with a detailed description of certain specific embodiments of the disclosure. Furthermore, embodiments of the disclosure may include several novel features, no single one of which is solely responsible for its desirable attributes or which is essential to practicing the embodiments of the disclosure herein described.
II. DEFINITIONSIn order to facilitate an understanding of the systems and methods discussed herein, a number of terms are defined below. The terms defined below, as well as other terms used herein, should be construed broadly to include the provided definitions, the ordinary and customary meaning of the terms, and/or any other implied meaning for the respective terms. Thus, the definitions below do not limit the meaning of these terms, but only provide exemplary definitions.
User: Also referred to herein as “reviewer” and/or “viewer.” An individual (or group of individuals) that interfaces with a computing device to, for example, view medical images. Users may include, for example, physicians (including, for example, doctors, radiologists, etc.) hospital staff, and/or any other individuals (including persons not medically trained) involved in analysis, annotation, comparison, acquisition, storage, management, or other tasks related to medical images (or any other types of images) as described herein. Any discussion herein of user preferences and/or rules associated with users should be construed to also, or alternatively, include user group preferences (or rules associated with groups of users), site preferences/rules, system preference/rules, and/or default software preferences/rules.
Medical Image (also referred to herein as an “Image”): Any type of image of an organism (e.g., a human patient). It may include but is not limited to a radiograph (e.g., an x-ray image), computed tomography (CT), magnetic resonance imaging (MRI), Ultrasound (US), mammogram, positron emission tomography scan (PET), nuclear scan (NM), pathology, endoscopy, ophthalmology, or many other types of medical images. As mentioned above, medical images may be reconstructed and/or rendered from 3D or volumetric image data using methods including multiplanar reformation/reconstruction (MPR), maximum intensity projection (MIP), and/or the like (including, e.g., any Computerized Advanced Processing (CAP), as described below). Images of the present disclosure also include “multi-frame” images, which are images comprising multiple frames (also referred to herein as sub-images). For example, a multi-frame image may be played as a movie (e.g., showing a beating heart, where each frame shows the beating heart at a different point in time).
Modality: A medical imaging method (e.g., a patient who undergoes an MRI is said to have been scanned with the MRI modality).
Image Series (also referred to herein as a “Series”): Any two or more images that are related. Images in a series typically share one or more common attributes, for example, a type of anatomic plane and/or an image orientation. For example, an image series may comprise two or more images of a particular patient that are acquired on a particular date, e.g., different x-ray projections of the chest. A series of contiguous 3 mm axial CT scans of the chest is another example of an image series. A brain MRI scan might include the following series: sagittal T1 weighted images, axial T1 weighted images, axial FLAIR images, axial T2 weighted images, as well as post contrast axial, sagittal and coronal T1 weighted series. An image series of an exam may be identified by its “type” (also referred to herein as a “series type” and/or a “view type”). For example, series may be acquired using different pulse sequences, acquired in different anatomic planes (also referred to herein as “imaging planes”), and/or acquired before or after administration of intravenous contrast material. An image series may be limited to images of a certain modality or may comprise images of multiple modalities.
Montage: An arrangement of images. In some implementations, a montage may itself be an image which comprises two or more images stitched together into a single image in a particular arrangement. In some implementations, a montage may be a file comprising sufficient information regarding each image of the montage so that the entire montage can be recreated upon display of the montage. A montage may include images from one or more series, exams, and/or the like. Examples of montages are described in more detail below in reference to
Patient: An individual who undergoes a medical imaging examination.
Medical Imaging Exam (also referred to herein as a “Medical Exam” and/or an “Exam”): A collection of data related to an examination of a patient. May be specific to a particular time or time period. Generally includes one or more medical images and/or image series, montages, reports, notes, graphs, measurements, annotations, videos, sounds or voice data, diagnoses, and/or other related information. May include multiple image series of multiple modalities, volumetric imaging data, reconstructed images and/or rendered images. For example, an exam of a patient may be the brain MRI scan mentioned above, and may include each of the image series obtained on a particular date including: sagittal T1 weighted images, axial T1 weighted images, axial FLAIR images, axial T2 weighted images, as well as post contrast axial, sagittal and coronal T1 weighted series. Another example of an exam may be a dual-energy radiography exam, which may include image data including traditional x-ray image images, bone subtracted (or “bone out”) x-ray images, and/or tissue subtracted (or “tissue out”) x-ray images.
Image Characteristic: Any characteristic related to display of an image. Includes without limitation, image angle (e.g., an angle of an image with reference to a standard one or more planes of human anatomy; also referred to herein as “scan plane”), anatomical position (and/or location) (e.g., a location, with reference to a standard one or more planes of human anatomy, of the patient represented in a particular image), image orientation (e.g., an orientation of the image with reference to a standard one or more planes of human anatomy), image rotation (e.g., a rotation of the image with reference to a standard one or more planes of human anatomy), image field of view, slice thickness, image window and/or level (e.g., a contrast of the image, a brightness of the image, and/or the like), image color map (e.g., that includes information for rendering different pixel intensities as different colors), other color characteristics, image opacity (and/or opacity map), image zoom level, image cropping information, and/or the like. In some instances, one or more image characteristics may be user defined and/or based on user preferences. The term “image characteristics” is used herein in reference to 2D medical images to refer to the various characteristics of the images with reference to the physical anatomy of a patient from which they were obtained. Such 2D medical images may be obtained to show a particular slice of a patient at a particular location such that a diagnosis of the patient may be made. As described below, later obtained 2D medical images of a same location may vary with respect to one or more image characteristics such that a valid comparison between the later obtained 2D image and the earlier 2D image is difficult. Such difficulty may arise due to variance in any image characteristics of the images, including those mentioned above. Image characteristics are also referred to herein as image “attributes.” Further examples of attributes are described below.
Artifactual Difference: A difference between two images that is not due to changes in patient structure, physiology, anatomy, pathology, and/or the like, but rather is a result of different image characteristics between the two images. Thus, artifactual differences do not indicate changes in physical or bodily structure of a patient (such as position, size, density, etc., of particular organs, tendons, muscles, bones, tumors, or other anatomical features). For example, scan planes of two images may differ, causing features of the two images to appear different from one of the other, even when there may be no actual differences in patient physiology. Such scan plane differences may be a result of, e.g., differences in patient position within the imaging scanner, differences in imaging scanner used and/or settings of the imaging scanner, and/or differences in how the 2D images are generated (e.g., how the plane of reconstruction and/or rendering of 2D images from a 3D volume is selected). Artifactual differences contrast with “physical,” or “actual,” differences which are a result of changes in patient physiology, anatomy, pathology, and/or the like. Accordingly, artifactual differences between medical images can be a source of faulty image interpretation, diagnosis, and even patient treatment. In various embodiments, the systems and methods described herein efficiently and automatically eliminate, or substantially eliminate, artifactual differences (due to, e.g., differing image characteristics) in images for comparison such that actual differences (e.g., changes in a physical characteristic of a patient) may be more accurately, efficiently, and easily discerned by a user.
Attribute: Any characteristic associated with a data item (e.g., a data item such as a medical exam, an image series, a medical image, and/or the like). Attributes may be inherited in a hierarchical manner. For example, a medical image may inherit attributes of an image series of which it is a part, and an image series may inherit attributes of a medical exam of which it is a part. Attributes may be stored as part of an associated data item (e.g., as metadata, DICOM header data, etc.) and/or separately from an associated data item.
Sorting: A process of arranging images from multiple image series (and/or medical exams and/or other sources (e.g., Clinical Decision Support (“CDS”) data, as described below)). In some embodiments, the process of sorting images from multiple image series may include generating a resultant “sorted” image series. While in some embodiments a sorted image series (including images from multiple image series) is generated, generation of a sorted image series is not necessary. Rather, in various embodiments, the process of sorting images may include determining an order of the images, which order may then be referenced when, for example, the images are displayed and/or viewed. For example, the system may simply reference pointers to images from multiple image series; the system may generate a “sorting metadata” file associated with the sorted series that indicates how images from multiple image series are sorted; and/or pointers to images from multiple image series may be determined in real-time as images are viewed in a sorted order. “Interleaving” is an example of a type of sorting of images in which images of multiple images series are regularly alternated. For example, in some embodiments, an interleaved image series comprises images from multiple image series ordered so that the interleaved image series alternates between the images of the original series. For example, when image series A comprising images A1, A2, . . . , An, image series B comprising images B1, B2, . . . , Bn, and image series C comprising images C1, C2, . . . , Cn are interleaved, the resultant interleaved image series may be ordered: A1, B1, C1, A2, B2, C2, . . . , An, Bn, Cn. Some embodiments and examples described below refer to “interleaving” of images, however other types of sorting may be used in those embodiments and examples. Images from multiple image series may be sorted in various patterns and multiple sortings may be determined from two or more image series. Images may be sorted based on one or more attributes associated with images, series, and/or exams. When images are sorted based on multiple attributes, a sorting of the images may be performed based on the attributes based on a priority of the attributes (as described below). Examples of sorting based on multiple attributes are described below in reference to, e.g.,
Image Pane: Also referred to herein as “image frame,” “viewing pane,” “viewing frame,” “comparison pane,” “comparison frame,” and/or simply “pane.” A region of a computer display that may display an image.
Annotation: Any notes, measurements, links, assessments, graphics, and/or the like, associated with a data item, either automatically (e.g., by one or more CAP, described below) or manually (e.g., by a user). For example, when used in reference to a medical image, annotations include, without limitation, any added information that may be associated with the image, whether incorporated into an image file directly, comprising metadata associated with the image file, and/or stored in a separate location but linked to the image file in some way. Examples of annotations include measurements by using linear dimensions, area, density in Hounsfield units, optical density, standard uptake value (e.g., for positron emission tomography), volume, curved lines (such as the length of a curved vessel), stenosis (e.g., percent narrowing of a vessel at a certain location relative to a reference location), or other parameters. Additional examples of annotations include arrows to indicate specific locations or anatomy, circles, polygons, irregularly shaped areas, notes, and/or the like. Additional examples of annotations include arrows to indicate specific locations or anatomy, circles, polygons, irregularly shaped areas, notes, and/or the like. Further examples of annotations include graphics that, for example, outline lesions, lumbar discs, and/or other anatomical features.
Computerized Advanced Processing (CAP): Any computerized image analysis, image analysis technique, and/or image processing technique discussed herein, and/or any similar computerized processing technique that is currently or later available. CAP is described herein with regard to radiology images, but CAP and the systems and methods described herein may be applied in other areas including, but not limited to, other types of medical images (for example, cardiology, dermatology, pathology and/or endoscopy, among others), computer generated images (for example, 3D images from virtual colonoscopy, 3D images of vessels from CTA, and the like), images from other fields (for example, surveillance imaging, satellite imaging, and the like), as well as non-imaging data including audio, text, and numeric data. In some embodiments, CAP may include, but is not limited to, volume rendering (including, for example, multiplanar reformation/reconstruction (MPR), maximum intensity projection (MIP), 3D volume rendering, and/or 3D surface rendering), graphical processing/reporting (e.g., automated identification and outlining of lesions, lumbar discs etc.), automated measurement of lesions or other anatomical features, other image processing techniques, and/or the like.
User Input (also referred to herein as “Input”): As used herein in reference to user interactions with data displayed by a computing system, “user input” is a broad term that refers to any type of input provided by a user that is intended to be received and/or stored by the system, to cause an update to data that is displayed by the system, and/or to cause an update to the way that data is displayed by the system. Non-limiting examples of such user input include keyboard inputs, mouse inputs, digital pen inputs, voice inputs, finger touch inputs (e.g., via touch sensitive display), gesture inputs (e.g., hand movements, finger movements, arm movements, movements of any other appendage, and/or body movements), and/or the like. Additionally, user inputs to the system may include inputs via tools and/or other objects manipulated by the user. For example, the user may move an object, such as a surgical instrument, tool, stylus, or wand, to provide inputs. Further, user inputs may include motion, position, rotation, angle, alignment, orientation, configuration (e.g., fist, hand flat, one finger extended, etc.), and/or the like. For example, user inputs may comprise a position, orientation, and/or motion of a hand and/or a 3D mouse.
Data Store: Any computer readable storage medium and/or device (or collection of data storage mediums and/or devices). Examples of data stores include, but are not limited to, optical disks (e.g., CD-ROM, DVD-ROM, etc.), magnetic disks (e.g., hard disks, floppy disks, etc.), memory circuits (e.g., solid state drives, random-access memory (RAM), etc.), and/or the like. Another example of a data store is a hosted storage environment that includes a collection of physical data storage devices that may be remotely accessible and may be rapidly provisioned as needed (commonly referred to as “cloud” storage).
Database: Any data structure (and/or combinations of multiple data structures) for storing and/or organizing data, including, but not limited to, relational databases (e.g., Oracle databases, mySQL databases, etc.), non-relational databases (e.g., NoSQL databases, etc.), in-memory databases, spreadsheets, as comma separated values (CSV) files, eXtendible markup language (XML) files, TeXT (TXT) files, flat files, spreadsheet files, and/or any other widely used or proprietary format for data storage. Databases are typically stored in one or more data stores. Accordingly, each database referred to herein (e.g., in the description herein and/or the figures of the present application) is to be understood as being stored in one or more data stores.
III. EXAMPLE COMPUTING DEVICES AND SYSTEMSAs shown, the network environment 100 may include the computing system 150, a computer network 190, an image store 122, a server 120, a user preferences and rules database 124, one or more optional external sensors 125, one or more imaging devices 170, and/or a clinical decision support (CDS) server 171. As described below, in various embodiments the computing system 150, the image storage 122, the server 120, the user preferences and rules database 124, the external sensor(s) 125, the imaging devices 170, and/or the CDS server 171 may be in communication with one another via the network 190. In some embodiments, various of the image storage 122, the server 120, the user preferences and rules database 124, the external sensor(s) 125, the imaging devices 170, and/or the CDS server 171 may or may not be considered a part of the computing system 150. For example, in some embodiments one or more of these components may be implemented as part of the computing system 150, may be in direct communication with the computing system 150, and/or may be in indirect communication (e.g., over network 190) with the computing system 150.
The computing system 150 may include various components as shown and described in detail below. As described below, the computing system 150 may display images (including, e.g., medical images) and/or other data to a user via a display 155. The computing system 150 may include one or more sensors 161 (and/or one or more external sensors 125) that detect input from a user (e.g., gesture inputs via hand motion). Inputs may also be received via the input devices 156 described below. Sensors 161 may take various forms and may utilize, for example, cameras that utilize visible light, cameras that utilize infrared light, ultrasonic sensors, etc. Sensors 161 may be placed in a variety of positions that allow visualization of a body part to be monitored. As described below, in response to user input received by the computing system 150 (including, e.g., detected motion of a user's hand and/or other body part and/or inputs via the input devices 156), information displayed (e.g., medical images) may be updated.
Additional components of the computing system 150 may include, for example, one or more processors 152 and memory and/or data storage 153 (including one or more software modules 151 and/or a rules engine 163 (which may itself comprise a software module)). In particular, as described below, the rules engine 163 may execute various rules (for example, one or more rules stored in the user preferences and rules database 124) that may be used to translate various user inputs into corresponding changes of displayed images and/or other data. In another example, various rules of the user preferences and rules database 124 may indicate user preferences for sorting of images having certain attributes, automatic selection of images for comparison with a user selected image, preferences for registration of sorted and/or comparison images, selection of CDS data, and/or the like.
“Sorting rules” are referred to in reference to
“Matching rules” are referred to in reference to
Further, matching rules may also be used by the system to accomplish registration and/or matching of data (e.g., images, image series, exams, etc.) by adjustment of one or more image characteristics so as to remove artifactual differences, as described herein. For example, in some embodiments the system may include a registration and matching processor (and/or may include the described functionality within the computing system 150) that may automatically register and/or match images for more efficient comparison. Example systems and methods for registration and matching of images are described in the '167 application.
In various embodiments, any of the rules of the user preferences and rules database 124 (including, e.g., sorting rules and/or matching rules) may be selected based on, for example, one or more attributes of data displayed and/or an identifier or characteristic associated with a user. In various embodiments, any rules and/or particular sets of rules of the user preferences and rules database 124 may be associated with specific users, groups of users (e.g., a type of doctor, etc.), sites (e.g., a hospital, etc.), other characteristics of users, computing devices used the users, and/or the like. Thus, rules may be automatically selected by the system based on one or more characteristics associated with a user. In some embodiments, a default set of rules may apply to all user interactions, and/or when there are no rules specifically associated with the user. The various rules may be provided by the users themselves, by a system administrator, and/or they may be preprogrammed in to the system.
Additional components of the computing system 150 may include sensors 161, which may include, for example, motion sensors 159, orientation sensors 160, and/or location sensors 162. The various sensors 161 may include, for example, a camera (for example, video cameras, infrared cameras, and/or the like) or array of cameras capable of detecting (and/or providing data necessary for detecting) a position (including a location and/or orientation) and/or motion of the user's hand (or other gesture from the user). For example, sensors 161 may comprise a commercially available position sensor such as Microsoft's Kinect or Leap Motion's Leap Motion Controller. In various embodiments, one or more, or in some cases all, of the sensors 161 may be located as external sensors 125. In some embodiments, combinations of external sensors 125 and sensors 161 of the computing system 150 may be used for detecting user inputs.
In some embodiments, one or more sensors (for example, external sensors 125) may be attached and/or removably attached to the user, and/or may be located in tools or other objects operated by the user (e.g., a surgical instrument, tool, stylus, or wand, as described above) and such sensors may transmit information to the computing system 150 (through either a wired or wireless connection) such that the inputs of the user may be determined. For example, such sensors on the user may include gyroscopes (that may be used, for example, to detect and measure rotational motion), accelerometers (that may be used, for example, to detect and measure linear motion), compasses, Global Positioning System (GPS) transceivers and devices, near field communication (NFC) technology devices, Radio Frequency Identification (RFID) devices, systems and devices utilizing WiFi, systems and devices utilizing Bluetooth such as iBeacons, and/or the like. In another example, a position transmitter may be used to track the location, position, movement, and/or orientation of the user and/or the user's body parts (e.g., the user's hand). Such a position transmitter may transmit a position and/or motion of, e.g., the user's hand and/or an input device, to one of the external sensors 125. In some embodiments position information from the external sensors 125 may supplement position information from the sensors 161. In other embodiments, a user inputs may be determined based on only information from sensors 161 or based on only information from external sensors 125.
The various sensors and transmitters (e.g., any combination of the sensors described above) may provide input data to the computing system 150 such that inputs from the user may be detected and/or determined. Such input data may be processed by, for example, one or more software modules 151 of the computing system 150 and/or other components of the system, as described below, such that displayed image data and/or other displayed information may be updated.
As further described below, network environment 100 may include a server 120 that provides information that is displayed by computing system 150. Network environment 100 may include image storage 122 (for example, a data store, database, and/or storage system) that may be configured to store information, such as image data (also referred to herein as image and/or imaging information) (for example, images (e.g., two-dimensional (2D) images), image series, three-dimensional (3D) imaging data, medical imaging exams, and/or the like), that is processed by server 120 and/or computing system 150. In various embodiments, image data is stored in Digital Imaging and Communications in Medicine (“DICOM”) format and/or any other appropriate format. Network environment 100 may also include user preferences and rules database 124 which may store user, group, or site preferences and rules that determine the operation of various embodiments as described herein. Additionally, user preferences and rules database 124 may include various rules that are used by the system to translate user inputs into corresponding movement of displayed images and/or other data, as described below.
In some embodiments, user preferences and rules database 124 may be used to store particular rules and preferences to apply to medical imaging data (or other data types in other embodiments) for particular users. For example, certain rules may be preferred by certain users and/or groups of users. Accordingly, preferences associated with those certain users and/or groups of users may indicate that the certain rules are to be applied such that, for example, certain types of images are to be sorted (and/or selected for comparison) based on certain attributes of those images. As discussed in further detail below, the rules and preferences of the user preferences and rules database 124 may vary depending of the type of image, area imaged, clinical indication, source of image, display device, user, or other factors. Accordingly, any type of user preferences and/or rules are expressly contemplated for use in conjunction with the systems and methods described herein.
In various embodiments, the functionality provided by image storage 122, server 120, CDS server 171, and/or user preferences and rules database 124 may reside within computing system 150.
The imaging devices 170 may include any devices capable of obtaining images for use by the system. For example, imaging devices 170 may include one or more medical scanners, such as X-Ray scanners, MRI scanners and/or CT scanners (as further described below).
The clinical decision support (CDS) server 171 may provide any type of imaging and/or other data (referred to herein a clinical decision support (CDS) data) to the computing system 150 that may be useful for analysis and evaluation of images by a user (e.g., for more efficient diagnosis of medical conditions). Determination of CDS data may be by execution of one or more rules of the user preferences and rules database 124. For example, the CDS server may, in response to a request by a user for CDS data in relation to a series of medical images, analyze one or more characteristics associated with a patient from which the series was obtained. The CDS server 171 may then determine, based on the analysis, a comparison series of images comprising images of a “normal” patient having similar characteristics. The determined comparison series may then be provided and sorted with (or, alternatively, displayed alongside) the series of medical images. Accordingly, the comparison series and the series of medical images may be compared and evaluated. As described above, certain rules may be executed based on a user, group, site, and/or the like. Thus, a doctor with a particular specialty may receive difference CDS data from the CDS server 171 than a doctor with a different specialty. Accordingly, the CDS server 171 may enable more efficient diagnosis of patients. As mentioned, the CDS server 171, and/or the functionality of the CDS server 171, may, in some embodiments, be included in the computing system 150.
IV. EXAMPLE USER INTERFACES FOR COMPARISON OF IMAGESIn the embodiment of
In some embodiments, such as where the series A and B images are related, e.g., series A comprises mammogram images of a patient taken on a first date and series B comprises the mammogram images of the same patient taking on a later date, it may be advantageous to identify differences between the images of series A and series B. For example, if a lung radiograph from two months previous, and a current lung radiograph are to be compared in order to determine if any changes have occurred in the lungs over the previous two months, the viewer or reader may view the two x-rays side by side, such as in image panes 212 and 214 as illustrated in
In embodiments with more than two series, the interleaved images may be ordered according to various schemes and based on various attributes, as described below. For example, images from four image series may be ordered as follows: first image of first image series, first image of second image series, first image of third image series, first image of fourth image series, second image of first image series, and so forth. In other embodiments, however, the interleaved images may be ordered differently. For example, images from four image series may also be ordered as follows: first image of first image series; first image of second image series; first image of first image series; first image of third image series; first image of first image series; and first image of fourth image series. While some of the description above and below refers to “interleaving,” any other sorting of images from multiple image series falls within the scope of the present disclosure. For example, images may be sorted based on attributes other than image number, and may further be sorted based on multiple attributes, as described below in reference to
In one embodiment, the computing system 150 that is coupled to the display 210 may store settings for displaying images of particular image series, such as, for example, time for displaying each image, resolution of each image, cropping to be applied to each image, and any other setting that maybe appropriate. In one embodiment, the time for displaying an image may be determined real time by the user. For example, the user may press a designated key on a keyboard or mouse, and/or provide other user input (as described above) in order to indicate that the current image should be replaced with an adjacent image in the interleaved image series. In another embodiment, the user selects settings for display of the images. For example, the user may select an appropriate zoom level of an image series that should be applied to each image in the image series.
Thus, the images of series A may be magnified more or less than the images of series B. In addition, the user may adjust any other visualization settings for individual images, an entire image series, or two or more image series. In an embodiment, as described above and below, the system may automatically register and/or match images for more efficient comparison.
With the first image of series B displayed in the comparison pane 410 (state 404), the user may initiate viewing of an adjacent image in the interleaved image series by pressing a certain key on a keyboard or mouse, or by other user input, for example. In an embodiment, a first input from a mouse indicates that a next image, e.g. image 2 of series A (state 406), should be displayed in the comparison pane 410, and a second input from the mouse indicates that a previous image, e.g. image 1 of series A (state 402), should again be displayed in the comparison pane 410. In one embodiment, the first input is entered by the user moving a scroll button on the mouse in a first direction and the second input is entered by the user moving the scroll button on the mouse in an opposite direction. In another example, the first and second user inputs may be provided by gestures (e.g., the user moved their hand right or left in front of the display 210). Thus, the user may change the content of the comparison pane 410 to either a next or a previous image in the interleaved image series. For example, at state 404, if the user wishes to again view the first image of series A, e.g., in order to locate difference in the first images of series A and B, the user may provide an input to the computing system 150 indicating movement to a previous image. Alternatively, at state 404, if the user wishes to view a next image in the interleaved image series, the user may provide an input to the computing system 150 indicating movement to a next image.
At state 406, the second image of series A is displayed in the comparison pane 410, replacing the first image of series B (state 404). At state 406, the user may provide inputs to the computing system 150 indicating that the comparison pane 410 should be updated with a previous image, e.g. state 404, or a subsequent image, e.g., state 408.
At state 408, the second image of series B is displayed in the comparison pane 410, replacing the second image of series B (state 406). At state 406, the user may provide inputs to the computing system 150 indicating that the comparison pane 410 should be updated with a previous image, e.g. state 404, or a subsequent image. In one embodiment, each of the image series A and B include more than two images, such as 3 or more images, and the images of series A and B are displayed in the manner described above with respect to
Accordingly, in some embodiments, the images of each of the image series are automatically modified (e.g., registered and/or matched) so that image characteristics of the images are similar. For example, images may be adjusted by changing their size, rotation, and location. If the images are of substantially the same anatomical structure, when the images of the interleaved image series are displayed in the comparison pane, differences between adjacent images may be more easily detected. In one embodiment, selected images are morphed in order to achieve a common size of the anatomical structure of interest in each of the images. In one embodiment, photographic filters may be applied to all images of one or more image series, or to selected images of one or more image series, to further enhance the viewer's ability to distinguish differences in the images.
In one embodiment, information regarding the image currently displayed in the comparison pane 410 is displayed on the display device 210 and updated as the images in the comparison pane 410 are changed. For example, information regarding the images series and image number within the series may be displayed for each image. In addition, the exam date and time may also be displayed and updated as the images of the interleaved image series are displayed in the comparison pane 410. In one embodiment, an indicator of whether the current display is of an interleaved image series or a single image series is displayed on the display device. For example, “interleaved” or “sorted” may be displayed at the top of the display device when an interleaved image series is displayed in a comparison pane. In some embodiment, the user may choose what information related to the images of the interleaved image series should be displayed. The user may also be provided the ability to turn the display of information on and off, such as by pressing a particular key or key combination on the keyboard.
In
As illustrated in
In an embodiment, the interleaving described above in reference to
In block 610, a first image series is selected. As noted above, an image series is a group of two or more images that are in some way related. For example, a first image series may comprise multiple chest x-rays of a patient that are taken on a given date.
In block 620, one or more comparison image series are selected. These image series also each comprise two or more images that are in some way related, such as having been taken at a common exam. The comparison image series may be related to the first image series so that when the first image series and the comparison image series are compared, meaningful distinctions between the image series may be detected. As described below in reference to
In one embodiment, the first image series and the comparison image series are selected by the user clicking on a button indicating that image sorting is desired. In one embodiment, a user right-clicks with a mouse on an image of a first image series in order to initiate display of an “interleave menu” or “sort menu” listing options for selecting image series for sorting and viewing the sorted image series. In one embodiment, the interleave menu includes an option, such as “interleave adjacent,” indicating that the user may select one of the other image series displayed on the display for interleaving. In certain embodiments, any number of image panes may be simultaneously displayed on the display device 210. For example, in
When the “interleave adjacent” option is selected, the user may select one or more comparison series by moving the pointer to a border between the adjacent series and clicking the mouse button. In one embodiment, the cursor icon changes when it is positioned in a border indicating that the adjacent image series may be selected for comparison by clicking the mouse button. With reference to
In one embodiment, the interleave menu also includes an option that allows the user to select an adjacent image series for interleaving and, after selection of the first and comparison image series, displays images of the interleaved images series in a comparison pane that covers the entire display area, or substantially all of the display area, of the display device. In this way, the images of the selected image series may be viewed at a higher magnification level and, accordingly, differences in the images may be more easily detectable. Thus, in an embodiment that displays four image panes on the display device (e.g.,
In one embodiment, the interleave menu includes an option that initiates automatic selection of one or more comparison image series based upon attributes of the selected first image series. For example, image series with the same or similar names may be selected as comparison image series. In addition, image series may be selected automatically based upon any other criteria, such as one or more information items contained in the DICOM headers of images. In one embodiment, when this option is chosen from the interleave menu, a list of image series that have the same series name, or other criteria that may be user defined, may be displayed. The user may then select one or more of the displayed series as comparison image series.
The interleave menu advantageously allows the user to select image series for interleaving and automatically display the generated interleaved image series with minimal input from the user. For example, after selecting “interleave adjacent” on the interleave menu, a single click of a mouse, for example, on a border between the images to be interleaved causes the computing system 150 to generate an interleaved image series and display a first image of the interleaved image series in a comparison pane on the display device.
In an embodiment, the user may initiate sorting of image series by selection of button as described below in reference to the user interface of the 18C. In the embodiment of
Returning to the diagram of
In an advantageous embodiment, the starting image in each of the series should be related so that meaningful differences between the images of the image series are detectable. In one embodiment, the user may adjust image characteristics of each image in an image series by adjusting the image characteristics of the image currently displayed from the desired image series. For example, if the first image series is at a higher zoom level than the comparison image series, the zoom level of each image in the first image series may be adjusted by adjusting the zoom level of the currently display image of the first image series.
After selecting the first image series and the comparison image series, and selecting the image characteristics of one or more of the images which are then applied to the other images in the series to which it belongs, the image series are interleaved so that an interleaved image series is created, as illustrated in
In a block 640, images from the interleaved image series are displayed in a comparison pane displayed on the display device. In one embodiment, the comparison pane fills substantially the entire display area of a display device. In another embodiment, the comparison pane is smaller than a total display area of the display device. In one embodiment, a user input determines when a current image displayed in the comparison pane is updated with an adjacent image, by moving a scroll wheel on a mouse while pressing a mouse button, for example. In one embodiment, when the user has completed viewing the interleaved image series, the previous layout on the display device may be restored by the user performing a specific action, such as releasing the mouse button that is depressed while viewing interleaved images.
In various embodiments, as mentioned above, images to be interleaved may be reconstructed and/or rendered from volumetric imaging data via one or more of the CAP method described above. In some instances, two exams comprising volumetric image data may be selected for comparison, and matching images may then be reconstructed and/or rendered for interleaving. In other instances, a set of volumetric imaging data may be selected for comparison with an existing image series, and a comparison image series may then be reconstructed/rendered and matched from the set of volumetric imaging data.
In another embodiment, interleaving of image series produces two or more interleaved image series. In one embodiment, images of a first interleaved image series may be alternatively displayed in a first comparison pane of the display device, while images of a second interleaved image series may be alternatively displayed in a second comparison pane of the display device, and so on. For example, if the image series to be interleaved are series A comprising images A1, A2, and series B comprising images B1, B2, the system may generate a first interleaved series ordered A1, B1, and a second interleaved image series ordered A2, B2. In one embodiment, images from each from each of the first and second interleaved image series are concurrently displayed on a display device in separate comparison panes. In one embodiment, an option on the interleave menu, discussed above, may be selected in order to initiate generation of multiple interleaved image series and concurrent display of multiple comparison panes on the display device. In one embodiment, more than two images of image series may be interleaved in the above-described manner and more than two comparison panes may be displayed concurrently on a display device for viewing the generated interleaved image series.
As mentioned above, the inventors have found that users using the system to flip through medical images (e.g., to analyze the images and perform diagnoses based on the images) are faster (e.g., 15% faster or more) and more accurate than users using traditional methods of comparing images.
Referring to
At block 1402, if images to be sorted are from a 3D exam (e.g., volumetric image data) then, if not already reconstructed and/or rendered, or if reconstruction and/or rendering is otherwise needed, the system may optionally automatically reconstruct and/or render the images. Reconstruction and/or rendering of images may be based on registration and/or matching techniques as described herein. For example, particular slices and slice thicknesses of the volumetric image data may be selected to match other images for comparison.
At block 1404, images to be sorted may optionally be registered/matched so as to reduce artifactual differences, as described herein. Such registration/matching may be based on one or more matching rules, as described above. Additional examples of registration and/or matching of images are described in the '167 application.
At block 1406, sorting rules for sorting of the images to be sorted are identified by the system. As described above, sorting rules may be selected based on an identity of, and/or characteristics associated with, the user. Alternatively, the user may select particular sorting rules. For example, in an embodiment the system may present one or more attributes to the user that may be used as a basis for sorting images (as described below). The user may then select one or more of the attributes, and a prioritization of the one or more attributes, such that the system may then apply sorting rules based on the selected one or more attributes. In some embodiments, sorting rules (e.g., attributes for sorting images) may be automatically selected based on a type of exam and/or series, and/or based on another characteristic of the exam and/or series. Examples of sorting rules are further described below.
At block 1408, the system determines a sorting of the images based on the determined sorting rules. For example, the system may process the images to be sorted, analyze attributes associated with the images, and, based on the attributes identified by the sorting rules, automatically sort the images. Examples of sorting of images are further described below.
At block 1410, the sorted images may optionally be registered/matched so as to reduce artifactual differences, such as by using one or more processes described herein. In some embodiments, registration/matching after sorting may be more effective than registration/matching before sorting (e.g., in optional block 1404) in that adjacent images may be better matched to each other. Such registration/matching after sorting may be performed in addition to registration/matching before sorting, or in place of the registration/matching before sorting. Such registration/matching may be based on one or more matching rules, as described above. Further, as mentioned above, in some embodiments, reconstructions and/or renderings of images from volumetric medical data may be performed based on, or simultaneously with, registration/matching and/or sorting. Additional examples of registration and/or matching of images are described in the '167 application.
At block 1412, images may be displayed by the system in the sorted order in response to user inputs, as described above. For example, the images may be displayed sequentially, in the sorted order, in a single image pane such that the user may quickly and efficiently determine differences between adjacent images.
At block 1414, the system may receive a user input to change the sorting rules. For example, the user may decide that a different sorting of the images may be more advantageous for identifying differences between two exams/series. Accordingly, the user may, on-the-fly, update the sorting rules. In response, the system automatically, as indicated by arrow 1418, determines a new sorting of the images based on the updated sorting rules, optionally registers/matches the sorted images (e.g., in the updated sorting order so adjacent images are registered with one another), and displays the images in the updated sorted order. In an embodiment, the sorting of the images may be updated in real-time or substantially real-time such that that user's review of the images is not interrupted. In an embodiment, an image displayed to the user prior to updating the sorting rules remains a same image displayed to the user after the images are re-sorted. Accordingly, the user may continue their review from the same image, but upon switching to adjacent images the images will be shown in the re-sorted order.
If no updates to the sorting order are received by the system, then the method continues with block 1412 wherein the sorted image series can be navigated by the user, as indicated by arrow 1416.
Referring to
In order to evaluate the dual-energy radiography exam, and thereby determine a health and a diagnosis of the patient, the user may desire to compare images of the various types, and obtained at various times, to determine differences among the images. Accordingly, the images of one or more exams may be sorted and displayed by the system, based on various rules (including, sorting rules).
An example of such a sorting of two dual-energy radiography exams is now described.
Referring now to
In the example of
At block 1504 (and as described above in reference to blocks 1406 and 1408 of
At block 1506, the image series as sorted, based on the rules, are shown such that the traditional view series C and A are ordered before the bones out view series D and B. Further, the series are secondly sorted based on time of acquisition such that the most recent series art sorted first, resulting in a final sorting order of Series C, A, D, B. The series may then be displayed to the user in response to user inputs, as described above. Further, as described above, the series may optionally be registered and/or matched by the system. As also described above, in the example in which each series includes a single image, the user may then quickly flip through the images of the various types in the sorted order to quickly and efficiently identify differences among the different types of images, and the images obtained at different times.
Although not provided herein for ease of illustration, the sorting rules may further include an indication of sorting order associated with each attribute. Thus, for example, the sorting rules may indicate sorting based on time of acquisition, with the most recent appearing first (or, alternatively, the oldest appearing first). In another example, the sorting rules may indicate sorting based on a type of view, with the view ordered alphabetically and/or based on a particular order (e.g., traditional first, then bones out, then tissue out, etc.).
At block 1507, the user can navigate through the sorted image series, such as to view a single image of the sorted image series in a particular image pane on a display and update the single image displayed in that image pane to an adjacent image in the sorted image series based on commands from a user input device.
In some implementations, the user may want the images sorted based on different sorting rules (e.g., a different hierarchy of attributes). If such an alternative sorting order is indicated (e.g., by the user as images sorted based on first sorting rules are being view or based on user rules before viewing of the first sorted image series begins), at block 1508 (and as described above in reference to block 1414 of
Accordingly, at block 1510, the image series as re-sorted, based on the updated rules, are shown such that the series from most recent exam, series C and D are ordered before the series from the older exam, series A and B. Further, the series are secondarily sorted based on a type of view such that the traditional view appears before the bones out view, resulting in a final sorting order of Series C, D, A, B. The series may then be displayed to the user in response to user inputs, as described above. Further, as described above, the series may optionally be registered and/or matched by the system after the resorting.
At block 1511, the user can navigate through the re-sorted image series. As noted above, in some embodiments the user can switch between multiple sorting orders of images series via a particular input command.
As shown, in
As shown, in
In the present examples the images are sorted based on an image number for illustrative and convenience purposes. However, rather than image number, more commonly images may be sorted based on an anatomical location or position, such that adjacent images in the sorting order show a same anatomical location or position. For convenience, image number is used in the present examples.
Referring now to
As described above, sorting of images, and display of the sorted images, allows a user to more accurately and efficiently detect differences between images. In many of the examples described above, the user uses the system to compare images from the same patient to efficiently identify differences. For example, the system may be used to compare a series from one exam of the patient to another series from a second exam on the patient, which could be the same or a different modality, e.g., compare an MRI to an older MRI, compare an MRI to a PET obtained at substantially the same time, or compare a mammotomography series to a breast MRI series. In addition, a series from an exam might be compared to another series in the same exam, e.g., post-contrast images vs pre-contrast images. These examples may be considered a form of perceptual enhancement.
Using CDS data as described below, in some embodiments the user is able to view (e.g., as part of a sorted image series) images from the patient with images from another person, such as an age-matched normal, to determine if the patient's images are abnormal. These examples may be considered a form of clinical decision support, including a form of cognitive enhancement.
Further, using CDS data as described below, in some embodiments the user is able to view (e.g., as part of a sorted image series) images from the patient and reference images, such as an anatomical drawings, to aid him in determining features of an abnormality, such as anatomical location. These examples may be considered another form of clinical decision support.
As described below, CDS data may include any data that may be sorted with patient data to enable one or more of the advantages described above. For example, CDS data may include data from other patients (e.g., patients considered “normal”), wherein patients may be filtered to include only those other patients having characteristics similar to those of the patient; reference data (e.g., references images); and/or the like. For simplicity, the description below refers to a CDS series (e.g., a series of references image, or a series of images from another patient), however other types of CDS data may be utilized by the system in similar ways.
In an embodiment, using CDS data, a patient's images may be compared to another person's images (or a composite normal exam) for the purpose of determining if the patient's images are normal or abnormal. For example, a patient might undergo a PET scan of the brain in a work-up of dementia. The user's job may be to determine if the distribution of activity in the patient's brain is normal or abnormal. So, an axial PET series from the patient's brain PET exam might be compared (via sorting as described herein) with an axial series from a normal aged-matched control to allow the user to more easily and accurately detect regions of abnormal activity, where abnormal activity might be indicated by lower (or higher) standard uptake values (SUVs) at particular anatomical areas represented in the scan from the normal aged-matched control.
In another example, a brain MRI in, e.g., a 7 month old child, might be compared to a normal aged-matched control to allow the user to more easily detect abnormalities in brain maturation, such as evolving MRI signal intensity changes related to myelination.
Such CDS data (e.g., normal aged-matched control studies) may be stored in the CDS server 171. One or more rules of the user preferences and rules database 124 may automatically determine CDS series for sorting and display based on, for example, user preferences, exam type, and/or other clinical information associated with the exam. For example, when a PET scan is performed for the work-up of dementia, a relevant normal scan may be automatically retrieved and sorted.
In an embodiment, retrieval and sorting of comparison scans may be performed manually, on-demand (e.g., in response to a user input).
In another embodiment, a CDS series may include, for example, anatomical drawings labeling the anatomy and anatomically matching the patient's series. For example, when reading a CT of the liver or MRI of the brain, cross sectional images containing anatomical drawings may be automatically chosen and sorted to allow the user to more easily and accurately determine the anatomical locations of lesions in the patient's series, for example, the genu of the corpus callosum in the case of a brain MRI.
In one embodiment, a CDS series may be registered and/or matched, as described above. Thus, a CDS series may be selected to match an orientation of the patient's series and/or the CDS series may be reformatted, registered, and/or morphed to match the location and geometry of images in the patient's exam.
At block 1450, a patient exam and/or a series associated with a patient may be received. For example, the patient exam may be select by a user for review. Patient information may be received and/or accessed by the system such that one or more characteristics associated with the patient may be determined.
In an embodiment, patient information may be retrieved from data stored in, and/or attributes of, an exam, series, and/or image (as described above). In some embodiments, patient information may be obtained from other sources.
For example, in some embodiments the system may include a software module/process (e.g., a webservice) that is connected to an exam reading system of a user and that passively gathers information regarding exams, reports, images, etc., read by the user. For example, when a radiologist reads an exam, the exam may include structured data, or structured data may be derived from the report using natural language processing. Similarly, images of an exam may be processed using machine learning, and may read annotations on the images. In some instances, image annotations may adhere to a standard such as the automated image mark-up (AIM) standard via which information may be gathered. Such gathered information may comprise patient information useable by the system, such as to determine a more relevant CDS series (or series) for comparison with the patient exam.
At block 1452, one or more rules are accessed to determine, e.g., whether a CDS series is to be compared, and/or what type of CDS series is to be compared (as described above in reference to the various examples).
In an embodiment, as a user reads an exam, the user receives, from the system (at block 1453), a prompt to view CDS data related to the exam. For example, the prompt may say, “would you like to compare to the closest age and gender matched patient with the same imaging findings (or normal findings) for this type of exam?” Accordingly, the user may indicate that CDS data is to be obtained and compared to an exam. Thus, a sorted image series may include multiple image series of patient images (such as two image series of a same modality taken some time, e.g., weeks or months, apart in order to determine differences), as well as one or more CDS image series. Thus, in various embodiments (whether or not including CDS data)_more than two series/exams may be selected for comparison and sorted. For example, 3, 4, 5, or more, series/exams may be selected. As described above, in various embodiments, the multiple sorted series of images may be displayed in one or more image panes of the user interface.
In some embodiments, a ratio of images included in a sorted image series from the various image series and/or CDS series sorted varies. For example, in a sorted image series including two image series of a patient (e.g., two MRI series separated by two months) and one CDS series of a baseline MRI from a patient having certain characteristics similar to the patient (e.g., gender, age, etc.), the user may not want or need images from the CDS image series interleaved between each set of matched images in the two MRI image series. Thus, the CDS image series may be interspersed into the sorted series at a different ratio, perhaps including an image from the CDS image series in only every second, third, fourth, etc., set of matched images from the other image series. In another embodiment, the CDS image series may be included in the sorted image series, but only displayed in response to a predefined input from the user. For example, as the user scrolls through a sorted image series of the two patient MRI series noted above, the user could provide a particular input (e.g., pressing Control) while scrolling to indicate that a CDS image should be shown as the next image in the image series (e.g., so that unless the particular input is pressed by the user, the CDS images may not be displayed in the comparison pane). In other embodiments, CDS images may be interleaved and/or accessed with reference to images of the patient under review in various other matters.
At block 1454, the determined CDS series is retrieved from the CDS server 171. The method then proceeds with block 1402 of
Using the computing system 150, in an embodiment a user may manually sort matched medical images, e.g., order the selected/comparison medical images for progressive display. The system may also provide an interface to re-order images or image series, and may further enable reorienting images (flip, rotate) and/or changing other image characteristics in order to best match the display. In various embodiments, more than two series/exams may be selected for comparison and sorted. For example, 3, 4, 5, or more, series/exams may be selected. Further, multiple sorted series of images may be displayed in one or more image panes of the user interface.
VI. EXAMPLE USER INTERFACES AND INTERACTIONS FOR AUTOMATIC SELECTION AND DISPLAY OF COMPARISON IMAGESAt block 1602, an image and/or series (and/or exam, in some embodiments) is selected by the user for viewing. Example user interfaces for such selection are described below in reference to
At block 1604, the system may, as mentioned above, automatically determine, based on one or more matching rules, one or more comparison images and/or image series (and/or exams). Such comparison images/series may be selected from a same exam as the selected image/series and/or another exam. For example, as described above, corresponding series from a previously obtained exam of the patient may be selected for comparison to a current series.
The matching rules may establish criteria for matching related images and/or image series. For example, matching rules may be established to select medical data based upon any of the following non-limiting criteria: modality (e.g., MRI, CT, X-ray, etc.); exam type (e.g., left knee X-ray, CT Chest, MRI Brain, etc.); archive status (e.g., has the exam been archived, archived and restored, not yet archived); assigned physician (e.g., has the exam been assigned to a particular physician for interpretation); exam age (e.g., how long ago was the exam done); patient age; any item in a DICOM header file (e.g., such as orientation, contrast use, thickness of slices, field of view, MRI tissue contrast weighting); time/day of acquisition; and/or any other attribute and/or image characteristic. In further examples, criteria may include information extracted through analysis of images (e.g., via one or more CAP), such as a view type of an image determined through automatic analysis of the image. With regard to some criteria, such as MRI tissue contrast weighting, the rules may analyze the MRI pulse sequence and the imaging parameters in order to determine the tissue contrast weighting and subcategorize images into weighting categories or weighting names.
The matching rules can be used to match medical images in the same and/or different medical series. For example, assume the medical images relate to three series each having 6 x-rays. The matching rules can be established such that like views among each of the different series are grouped together for subsequent viewing. The matching rules be defined using simple or complex search expressions such as “AND” or “OR.”
In another example, the matching rules may establish criteria for matching related images based on time and/or user preferences. Thus, as described below in reference to
At block 1606, the system may optionally register and/or match the selected images/series and/or the comparison images/series, as described above in reference to block 1404 and/or 1408 of
At block 1608, user selected and comparison images are displayed to the user simultaneously in multiple image panes of a user interface.
In an embodiment, the user may select an image series and/or an image of the image series to view. In response the system may determine one or more comparison series. The one or more comparison series may be selected from a same or different exam of the selected image series. Further, when the user views an image from the selected series, the system may automatically determine matching images from the one or more comparison image series. For example, the system may determine images that match in regards to an anatomical area displayed. Thus, the user may be able to easily compare the displayed images. Further, as mentioned above, the images may be registered and/or matched with regards to one or more image characteristics.
In an embodiment, more or fewer image panes than are shown in
In various embodiments, the user preferences and rules database 124 may include “display rules” that may indicate ways images and/or image series are to be displayed in user interfaces of the system. For example, an image or exam matching particular criteria in the display rules may be displayed with a certain pixel window level or width (similar to brightness and contrast), in color, based on a certain color map, opacity map, or other image characteristics. Display rules may be user-defined allowing the user to determine the timing, positioning, and size of displayed selected/comparison images. For example, a user can define that selected/comparison medical images are all displayed concurrently on a display. Also, for example, a user can define that the most recent of the selected/comparison medical images are displayed on the left hand portion of the display and the other selected/comparison medical images are displayed in sequence on the right hand side of the display, the sequence advancing in response to user prompting. In one embodiment, the display rules include directives (e.g., timing, positioning, and size). As an example, directives can include the following for identifying location information: TOP_DISPLAY, BOTTOM_DISPLAY, RIGHT_DISPLAY, LEFT_DISPLAY, CENTER_DISPLAY. Furthermore, if the number of comparison medical images is variable, the display rules can include instructions for identifying selected medical images based upon further rules, such as using the matching rules listed above. In addition, the display rules may or may not define how many images or image series are displayed per monitor, a display grid (e.g., 2×3, 5×4, etc.), or whether like images are displayed neighboring each other side by side horizontally or vertically. Furthermore, the display rules may also specify how different selected/comparison medical images from different series may be sorted together for successive display. Using the display rules, the user can provide display rules such that related medical images are readily comparable.
In an embodiment, display rules may be set to display pre- and post-contrast axial T1 weighted images from a brain MRI from the same exam in adjacent panes on a monitor. Display rules may also be set to display axial T2 weighted MRI images from a prior spine MRI adjacent to axial T2 weighted images from the current exam. Display rules may also be set to display a PA projection from a chest radiograph from a prior exam adjacent to the same projection from the current exam. As described below in reference to
It is noted that the display rules may define user display preferences that are specific for particular types of medical images, e.g., the imaging modality, the body part, whether there is one exam, two, or more medical images being displayed. In one embodiment, one set of display rules can apply to one modality and another set of display rules can apply to another modality. In addition, the display rules may include specific triggers or warnings that occur if the user-defined display rules are not satisfied.
In various embodiments, display rules may be based on any image characteristic, as described above. Further, in some embodiments, display rules may be stored as image characteristics.
At block 1610, the system determines whether or not the user has provided a user input to change a displayed image series. For example, in reference to
At block 1612, the system determines whether or not the user has provided a user input to change a displayed image to another displayed image of a displayed image series. For example, in reference to
In an embodiment, the user may change the image of any of the displayed panes (e.g., including the comparison images), and the images of the other panes are automatically updated by the system.
In an embodiment, as indicated at block 1614, in some instances some image series may not include a corresponding image to be displayed. For example, one of the displayed image series may not include a particular image slice, a particular image that corresponds to an anatomical position, and/or a particular image of a certain view type. In these instances, as described below in reference to
As indicated by arrow 1620, if no input is received from the user the method continues at block 1608, displaying images and waiting for input from the user.
In an embodiment, a first user input (e.g., pressing an arrow or pageup/pagedown key, a side-to-side hand gesture movement, etc.) may change displayed image series, while another user input (e.g., a scroll of a mouse wheel, a forward-and-back hand gesture movement, an hand rotation gesture, etc.) may change displayed images. In an embodiment, user inputs may apply to a one of the panes that is currently selected by, e.g., a mouse cursor hovering over the pane, the user's hand being located near the pane, the pane being highlighted in the user interface, and/or the like. In an embodiment, the user interface may enable a user to “lock” certain panes such that the user's inputs only affect “unlocked” panes. Thus, the user may compare other images/image series to a “frozen” image in one or more of the panes. In another example, the user may similarly select a single pane to be “unlocked,” causing all the other panes to be “locked.”
To view one of the images in more detail, the user may provide a user input to select a thumbnail image and/or drag a thumbnail image into a review portion 1805 of the user interface. Dragging of thumbnail image 1804 from the row to location 1806 is illustrated in
In an embodiment, selecting and/or dragging the thumbnail image to the review portion 1805 causes the system to automatically determine one or more comparison images/series, as described above in reference to
In an embodiment, the user interface may include one or more “sort” buttons, such as sort button 1842. Selection of the sort button causes the system to sort, based on various rules and as described above in reference to
In an embodiment, a selected image, and/or an image on which the “sort” button is pressed, may be the first image displayed in the image pane displaying the sorted images.
In an embodiment, the selected thumbnail image will be enlarged in a preferred image pane, and/or an image pane to which it is dragged.
In an embodiment, different user inputs when selecting and/or dragging the thumbnail image may cause the selected image to either be displayed alone in greater detail, or compared to other image series (as in
With this landmark structure identified in each of the two chest radiographs 1902 and 1904, an automated registration of the image may be performed, based on this common structure in each image. In
In the example of
However, in other embodiments, the computer system may make other adjustments to alter the scales in order to best match the actual anatomy understanding that different radiographic images can show various levels of geometric magnification based on the projection (AP vs PA) or distance of the patient from the detector and x-ray tube.
In the example of
With images 1902R and 1904R registered in this manner (e.g., via an automated process of identifying the aortic knob in each of the images 1902 and 1904 and adjusting the images to be registered in view of those identified positions), the images can advantageously be presented in a sorted series for comparison, such as by alternative display of the images in a particular comparison pane on a display in order to allow the user to more easily identify actual differences in the images.
In the example of
These comparison panes (which may include two, three, four, or more panes) may be organized and/or positioned in various arrangements. For example, the comparison panes may be positioned to the left and right of one another, above and below one another, in a grid, and/or according to any other arrangement. The number and positioning of the comparison panes may be determined based on one or more user preferences, and/or may be predefined by the system. Such comparison pane determinations may be based on one or more display rules as described above in reference to, for example,
Further, the system may be configured to display particular images, image series, and/or types of images, etc. in each comparison pane. Thus, for example, any image characteristic/attribute may be associated with the comparison panes. These associations between panes and image characteristics/attributes may be determined based on one or more user preferences, and/or may be predefined by the system. Thus, the system may advantageously be configured to display images, which may be flipped through by a user, in particular arrangements (e.g., hanging protocols). For example, one user may prefer to see left-facing lateral view images in one pane, while another user may prefer to see right-facing lateral view images in that pane.
For example, as indicated by image set 1922, a user may view two chest radiograph images 1932 and 1934 side by side (e.g., in comparison panes 1010A and 1010B). Images of image set 1922 may be related, e.g., they may be part of a same exam (e.g., obtained on the same day), and/or otherwise related. The images may also each be of particular types (and/or have a particular one or more image characteristics) and may be associated with respective series of images. For example, image 1932 may be a PA view of a chest (and part of a sorted image series of PA views including images 1936 and 1940) while image 1934 may be a Lateral view of a chest (and part of a sorted image series of Lateral views including image 1942). The image series associated with each image may be comprised of sorted image series that may be sorted according to any method described herein. For example, each image series may comprise multiple images of a same type that were obtained at different times (e.g., as part of different exams), but associated with a same patient. Further, the images of each image series may be registered/matched (e.g., scale, skew, contrast, etc. may be automatically adjusted in one or both images so they match and are more comparable) to enable efficient comparison as the user moves/flips from one image to the next.
As mentioned above, as the user provides input, the system may display (e.g., flip through) images of the multiple image series in a sorted order. Thus, for example, the system may replace images 1932 and 1934 (in their respective comparison panes) with images 1936 and 1938 (in their respective comparison panes, as indicated by 1924). In response to a further input, the system may replace images 1936 and 1938 (in their respective comparison panes) with images 1940 and 1942 (in their respective comparison panes, as indicated by 1926).
As mentioned above, each set of images (1922, 1924, and 1926) may be related (e.g., part of a same exam), however in some instances a particular exam may not include an image or image series of the type associated with a comparison pane. For example, sometimes a patient may have PA and Lateral chest exam one day, a single view AP chest exam the next day, and an AP and Lateral chest exam the day after that. To compare the images associated with these exams by flipping through the images in the comparison panes, the system may (as described above in reference to
In various implementations, as described above, images/image series may be determined to be related (e.g., matched) based on various image characteristics and/or other information. For example, DICOM header information may be analyzed to determine related series of images. In an embodiment, the system may perform one or more CAP on images/images series to determine relationships. For example, in the context of chest radiographs, images may be analyzed by the system to determine dates and/or other information contained within the images that may be used for matching images/image series.
Further, in some implementations the system may perform one or more CAP to identify image characteristics (e.g., as may be useful when no DICOM header information is present). For example, in the context of chest radiography, the system may analyze an image to determine that the image is a PA chest view, AP chest view, Lateral chest view, and/or the like.
Thus, the system may automatically determine an image series with which an image should be associated, an ordering of images within the image series, and a comparison pane where images of the images series are to be displayed.
In some implementations, the system may enable the user to manually register images by, for example, overlaying the images and/or placing the images side-by-side and providing tools for efficiently adjusting various image characteristics of one or both images.
As mentioned above, the inventors have found that users using the system to flip through medical images (e.g., to analyze the images and perform diagnoses based on the images) are faster (e.g., 15% faster or more) and more accurate than users using traditional methods of comparing images.
As with other examples herein, shuffling of two images (or two image series or exams) are discussed for purposes of illustration of the various automated processing techniques, but the same processes could be used for more images, so that a user could easily page or scroll through a sorted image series including chest radiographs (or any other image) taken over time to more efficiently and accurately detected changes. For example, registration of chest radiographs of a child taken every 3 months from birth to age 6 may require significant adjustments in various image characteristics in order to register the images such that a “timeline” of changes over those years (e.g., including 20-40 images—none of which have precise alignment prior to auto-registration) can be viewed in a slideshow (or “cine” mode, whether automatically advancing through images at a predefined interval, such as ¼, ½ or 1 second, or manually in response to user input) and can provide to a user visual comparisons that may be useful for diagnosis or treatment of the patient. Without such auto-registration techniques (e.g., based on alignment of the aortic knob that may be automatically detected in each of a plurality of images), such a timeline of multiple images would be difficult to derive useful data from, as positions of anatomical structures (e.g., the or aortic knob) would move from one position to another (as well as having other image characteristics different from one image to another) as each subsequent image is displayed.
In an embodiment, the system generates a chest radiograph sorted series in substantially real-time so that differences in chest radiographs can be identified with relative ease and time constraints on the viewing user. Without such an automated registration and sorting algorithm, a user could not practically make a comparison of chest radiographs in a similarly beneficial manner, especially where the number of chest radiographs is larger than two.
In one example implementation that makes use of certain features discussed herein, the system provides a specific hanging protocol for a chest radiograph combined with shuffling, combined with image analytics to select the proper views, combined with image analytics to result in display with matching orientation, registration, and scale.
VIII. MONTAGE IMAGE SORTINGIn some implementations, montages are saved as separate files, such as separate image files that are essentially a screenshot of a montage (e.g., a snapshot of the montages of
In some implementations, a montage having one or more images can be stored in one or multiple ways, including (1) storage of the complete composite montage image and/or (2) storage of sufficient information regarding each image so that the entire montage can be recreated upon future display or the individually stored images can be displayed, depending on the user's preferences, depending on the display environment (such as aspect ratio of the display window, monitor resolution, a combination of user preferences and display environment, or other factors.) For example, information regarding the arrangement of images in the montage, as well as information regarding display settings of respective images (e.g., magnification, brightness, centering, cropping, filters, annotations, insets, etc.) may be stored. These montage characteristics may then be recalled at a future time and used to re-build the montage. In this implementation, storage of an image of the entire montage may not be necessary, while in other embodiments the montage image (e.g., a snapshot of the montage) may be stored and used in certain circumstances, such as when the montage is to be displayed on a display having essentially the same resolution and aspect ratio as the display on which the montage was originally created. As used herein, the arrangement information included in montage characteristics may include a number of rows and columns of medical images in the montage, indications of specific locations of each medical image in the montage, indications of an order that medical images of the montage are displayed, and/or any other information that may be usable to read construct a montage on the same or another computing device based on the layout of the montage images.
Additional examples of montages and uses of montages are described in U.S. patent application Ser. No. 13/572,397, filed Aug. 10, 2012, and titled “DYNAMIC MONTAGE RECONSTRUCTION,” (the “397 application”), the disclosure of which is hereby incorporated by reference in its entirety and for all purposes, as if set forth fully herein.
As mentioned above, and according to any of the image viewing techniques described above, while viewing images (e.g., of one or more exams, series, etc., which may or may not be sorted), the user may designate certain images as key images. For example, the user may click a mouse button, keyboard shortcut, or by other input means indicate that an image is a key image. In some implementations, key images may be placed in a montage. For example, the user may indicate that certain images are to be placed in a montage by designation of those images as key images.
As described above and in the '397 application, images of a montage may be stored as separate files and information necessary to reconstruct the montage may be stored, and/or automatically combined into a single montage image. At the time a user requests to view a montage, the system may generate the montage, if necessary. Further, the system and/or user may determine if the montage is to be presented/assembled as originally stored, broken up into individual image components, and/or otherwise rearranged.
In some implementations, when an image is designated as a key image, an annotation is added to an image, and/or a user otherwise indicates that they want to add the image to a montage, the system may automatically perform additional actions with respect to that image. The actions performed, and the user input that causes the actions to be performed, may be based on one or more user preferences.
In one implementation, in response to such a user input (e.g., marking an image as a key image), the system automatically adds the indicated image to a series/exam in an appropriate location in the series/exam (e.g., in a sorted location). For example, suppose a series/exam consists of 8 images, some of which are multi-frame images (MF), and some of which are single frame images (SF). In this example, the series/exam may include the following images:
-
- Image 1—MF
- Image 2—MF
- Image 3—SF
- Image 4—MF
- Image 5—SF
- Image 6—SF
In this example, the user may indicate certain sub-images of one or more of the multi-frame images, and/or one or more single frame images, for further actions (e.g., may indicate as key images, may indicate as images to add to a montage, may add annotations, and/or the like). For example, the user may indicate a few frames (e.g., sub-images) of Image 1 (e.g., frames 32 and 38), and also Image 3. Thus, the system may, based on the user preferences add these indicated images to a montage and/or save the indicated images to the series/exam. Accordingly, in the example, the series/exam becomes:
-
- Image 1—MF
- Image 1.32—SF
- Image 1.34—SF
- Image 2—MF
- Image 3—SF
- Image 3.1—SF
- Image 4—MF
- Image 5—SF
- Image 6—SF
- Montage
As shown, the images are automatically added to the series/exam and are further automatically sorted such that they are located in appropriate locations in the series/exam. Images of a series/exam may be sorted based on one or more sorting rules and/or user preferences, as described herein. Advantageously, adding the images to the series/exam in the sorted locations may enable the user to easily display the series/exam's initial images and the added images in sequence (e.g., via any of the methods described above), and also be able to see the montage in a separate sequence.
In various implementations, the images that are added to the series/exam may be sorted based on various sorting rules, image characteristics, user preferences, and/or any other criteria, as described above in reference to the various embodiments of the disclosure. For example, the images may be sorted based on image numbers such that added images are added near (e.g., immediately after) an existing image from which the added image is derived.
In an implementation, the user may indicate where in a series/exam an image is to be added. Additionally, as mentioned above, the indicated images may be added based on one or more user preferences and/or image characteristics.
At block 2204, the system receives a user input with respect to the displayed image/frame. For example, the user may indicate an image/frame is a key image/frame, may annotate the image/frame, and/or may otherwise indicate the image/frame is to be processed according to the method described below. The user inputs that cause the system to perform the method described below may be based on one or more user preferences, one or more image characteristics, and/or the like.
At block 2206, the system optionally adds the displayed image/frame that has been indicated by the user to a montage, as described above. Further, the montage may be added to the series/exam of the image/frame, or to another series/exam. In some implementations, the user may indicate image characteristics to be applied to the image/frame added to the montage, may indicate a location of the image/frame in the montage, may indicate annotations to the image/frame, and/or the like.
At block 2208, the system may create copy of the displayed image/frame that has been indicated by the user. In various implementations, the user may indicate image characteristics to be applied to copy of the image/frame, may indicate annotations to the image/frame, and/or the like. In some implementations, an actual copy of the image/frame is not created, but rather data necessary to identify the indicated image/frame and update the indicated image/frame (e.g., with annotations, changed image characteristics, etc.) is created and stored by the system in the series/exam. This implementation may advantageously save data stores requirements and/or bandwidth requirements as a copy of a high resolution image may not need to be stored/transferred. Rather, system may generate the “copy” of the image/frame based on the data and the original copy of the image/frame when display is requested by the user. In some implementations the “copy” of the image/frame may be generated preemptively, prior to the user requesting to display the image/frame.
At block 2210, the system may insert the copy of the displayed image/frame into the series/exam of the image/frame, or another series/exam. As mentioned above, in some implementations only data necessary to create the image/frame is generated and inserted into the series/exam.
At block 2212, the images/frames, including the inserted copies, are sorted according to any other methods described here. For example, the inserted copy of the image/frame may be added to the series/exam at a location following an original copy of the image/exam, as described above. In other implementations, the images/frames may be sorted in other ways.
The user may view the images/frames of the series/exam accordingly to any of the methods described herein. In some implementations, the indicated image/frame may be automatically processed in any other way by the system, including by one or more CAP, as described herein.
Advantageously, according to certain embodiments, the original copies of images/frames may be preserved while updated copies of the images/frames (e.g., with annotations, with changed image characteristics, etc.) may be added to series/exams automatically. Advantageously, according to certain embodiments, frames may be extracted from multi-images and automatically added to a series/exam for further review by a user. Advantageously, according to certain embodiments, images/frames of a series/exam may be sorted for efficient review by a user. Advantageously, according to certain embodiments, the system may accordingly enable a user to efficiently review and compare images/frames of a series/exam, including copies (which may be updated/changed) of images/frames indicated by the user.
IX. EXAMPLE COMPUTING SYSTEMSReferring again to
As described above, the computing system may take various forms. In one embodiment, the computing system 150 may be an information display computing device and/or system, a server, a computer workstation, a desktop computer, a Picture Archiving and Communication System (PACS) workstation, a laptop computer, a mobile computer, a smartphone, a tablet computer, a wearable computer (for example, a head-mounted computer and/or a computer in communication with a head-mounted display), a smartwatch, a mobile computer, a cell phone, a personal digital assistant, a gaming system, a kiosk, an audio player, and/or any other device that utilizes a graphical user interface, such as office equipment, automobiles, airplane cockpits, household appliances, automated teller machines, self-service checkouts at stores, information and other kiosks, ticketing kiosks, vending machines, industrial equipment, and/or a television, for example. In an embodiment the computing system 150 comprises one or more computing devices in communication with one another.
The computing system 150 may include various components including, for example, one or more processors 152, memory and/or data storage 153 (including one or more software modules 151 and/or a rules engine 163 (which may itself comprise a software module)), an operating system 154, a display 155, one or more input devices 156, one or more interfaces 157, an audio input/output 158, and/or one or more sensors 161 (including, for example, zero or more motion sensors 159, zero or more orientation sensors 160, and/or zero or more location sensors 162). Each of the components of the computing system 150 may be connected and/or in communication with each other using, for example, a standard based bus system. In different embodiments, the standard based bus system could be Peripheral Component Interconnect (“PCI”), PCI Express, Accelerated Graphics Port (“AGP”), Micro channel, Small Computer System Interface (“SCSI”), Industrial Standard Architecture (“ISA”) and Extended ISA (“EISA”) architectures, for example. In addition, the functionality provided for in the components and modules of computing system 150 (as described above and below) may be combined into fewer components and modules or further separated into additional components and modules.
In various embodiments the software modules 151 may provide functionality as described above with reference to the various figures. For example, modules 151 of the computing system 150 may include user input modules, image display modules, motion detection/determination modules, orientation detection/determination modules, position detection/determination modules, location detection/determination modules, patient information display modules, rules engine modules (for example, rules engine 163), user interface modules, and/or the like. For example, each of the motion, orientation, position, and/or location detection/determination modules may determine user inputs and/or gestures such that the user may interact with medical images, as described above. Further, the image display modules and/or the user interface modules may display user interfaces, images, and/or other data on the display 155 in response to user inputs (as described in reference to various embodiments of the present disclosure). Further, the image display modules and/or the user interface modules may be configured and/or designed to generate user interface data useable for rendering the interactive user interfaces described herein, such as a web application and/or a dynamic web page displayed by a computing device. In various embodiments the user interface data may be used by the computing system 150, and/or communicated to any other computing device, such that the example user interfaces are displayed to a user. For example, the user interface data may be executed by a browser (and/or other software program) accessing a web service and configured to render the user interfaces based on the user interface data.
The rules engine 163 may operate in conjunction with the other modules to perform various functionality of the data navigation systems described above. For example, the rules engine 163 may determine, based on one or more rules of the user preferences and rules database 124, that a user input (e.g., a particular type of movement or input) is to be translated into a particular adjustment of a displayed image and/or other data. Further, the rules engine 163 may determine, based on one or more rules of the user preferences and rules database 124, certain types of images are to be sorted (and/or selected for comparison) based on certain attributes of those images, automatic selection of images for comparison with a user selected image, preferences for registration of sorted and/or comparison images, selection of CDS data, and/or the like. Such determinations may be based on, for example, a type of data displayed and/or an identify or characteristic associated with a user. As described above, rules of the user preferences and rules database 124 that may be executed by the rules engine 163 may include sorting rules, matching rules, and/or any other types of rules.
As described below, the software modules 151 may include various software instructions, code, logic instructions, and/or the like that may be executed by the one or more processors 152 to accomplish the functionality described above. In other embodiments, software modules 151 may reside on another computing device and/or system, such as a web server or other server (for example, server 120) or other server, and a user may directly interact with a second computing device and/or system that is connected to the other computing device and/or system via a computer network.
The computing system 150 may run an off-the-shelf operating system 154 such as a Windows, Linux, MacOS, Android, or iOS, or mobile versions of such operating systems. The computing system 150 may also run a more specialized operating system which may be designed for the specific tasks performed by the computing system 150, or any other available operating system.
The computing system 150 may include one or more computer processors 152, for example, hardware computer processors. The computer processors 152 may include central processing units (CPUs), and may further include dedicated processors such as graphics processor chips, or other specialized processors. The processors may be used to execute computer instructions based on the software modules 151 to cause the computing system 150 to perform operations as specified by the modules 151. The software modules 151 may include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. For example, modules may include software code written in a programming language, such as, for example, Java, Objective-C, Swift, JavaScript, ActionScript, Visual Basic, HTML, Lua, C, C++, or C#. While “modules” are generally discussed herein with reference to software, any modules may alternatively be represented in hardware or firmware. In various embodiments, the modules described herein refer to logical modules that may be combined with other modules or divided into sub-modules despite their physical organization or storage.
The computing system 150 may also include memory 153. The memory 153 may include volatile data storage such as RAM or SDRAM. The memory may also include more permanent forms of storage such as a hard disk drive, a flash disk, flash memory, a solid state drive, or some other type of non-volatile storage, as described below.
The computing system 150 may also include or be interfaced to one or more display devices that provide information to the users. Display devices 155 may include a video display, such as one or more high-resolution computer monitors, or a display device integrated into or attached to a laptop computer, handheld computer, smartphone, smartwatch, wearable computer, computer tablet device, or medical scanner. In other embodiments, the display device 155 may include an LCD, OLED, or other thin screen display surface, a monitor, television, projector, a display integrated into wearable glasses, or any other device that visually depicts user interfaces and data to viewers. As described above, images and other information may be displayed to the user via the display devices 155 such that the user may efficiently view and interact with such images and information.
The computing system 150 may also include or be interfaced to one or more input devices 156 which receive input from users, such as a keyboard, trackball, mouse, 3D mouse, dial and/or knob (for example, a smartwatch crown), drawing tablet, joystick, game controller, touch sensitive surface (for example, capacitive or resistive touch screen), touchpad, accelerometer, video camera and/or microphone.
The computing system 150 may also include one or more interfaces 157 which allow information exchange between the computing system 150 and other computers and input/output devices using systems such as Ethernet, Wi-Fi, Bluetooth, as well as other wired and wireless data communications techniques.
The computing system 150 may include the audio input/output 158 for receiving, for example, audio commands or other input from the user. The audio system may also provide audio output to provide audio information to a user, for example via a speaker or headset. As described above, various sensors of the computing system 150 may include, for example, gyroscopes, accelerometers, cameras, Global Positioning System (GPS) transceivers and devices, near field communication (NFC) technology devices, Radio Frequency Identification (RFID) device, systems and devices utilizing WiFi, systems and devices utilizing Bluetooth such as iBeacons, and/or the like. The various sensors may provide input/data to the computing system 150 related to the location, position, orientation, and/or motion of a user, a user's appendage (such as an arm and/or hand), and/or another input device operated by the user. Such information may be processed by, for example, one or more software modules 151 (such as the rules engine 163) as described above, such that displayed image data (or other types of data) may be updated. Additionally, as described above, the system may also include, in some embodiments, external sensors 125 that may also provide data related to user inputs. External sensors 125 may include any of the sensors described herein, and may provide functionality similar to that described herein with reference to the various sensors. In various embodiments, the functionality provided by image storage 122, server 120, and/or CDS server 171, may reside within computing system 150.
The computing system 150 may communicate and/or interface with other systems and/or devices. In one or more embodiments, the computing system 150 may be connected to the computer network 190. The computer network 190 may take various forms. For example, the computer network 190 may be a wired network or a wireless network, or it may be some combination of both. The computer network 190 may be a single computer network, or it may be a combination or collection of different networks and network protocols. Additionally, the computer network 190 may include one or more local area networks (LAN), wide area networks (WAN), personal area networks (PAN), cellular or data networks, and/or the Internet.
Various devices and subsystems may be connected to the network 190. As shown in
Also connected to the network 190 may be a Picture Archiving and Communications System (PACS) and/or PACS workstation. The PACS System may be used for the storage, retrieval, distribution and presentation of images (such as those created and/or generated by the MRI scanner and/or CT Scanner). The medical images may be stored in an independent format, an open source format, or some other proprietary format. A common format for image storage in the PACS system is the Digital Imaging and Communications in Medicine (DICOM) format. In various embodiments, the stored images may be transmitted digitally via the PACS system, often reducing or eliminating the need for manually creating, filing, or transporting film jackets.
The network 190 may also be connected to a Radiology Information System (RIS). In an embodiment, the radiology information system may be a computerized system that is used by radiology departments to store, manipulate and distribute patient radiological information.
Also attached to the network 190 may be an Electronic Medical Record (EMR) system. The EMR system may be configured to store and make accessible to a plurality of medical practitioners computerized medical records. Also attached to the network 190 may be a Laboratory Information System. In an embodiment, the Laboratory Information System may be a software system which stores information created or generated by clinical laboratories. Also attached to the network 190 may be a Digital Pathology System that may be used to digitally manage and store information related to medical pathology.
Also attached to the network 190 may be one or more Computer Aided Diagnosis Systems (CAD) systems that are generally used to perform Computer-Aided Processing (CAP) such as, for example, CAD processes. In one embodiment, the CAD systems functionality may reside in a computing device and/or system separate from computing system 150 while in another embodiment the CAD systems functionality may reside within computing system 150.
Also attached to the network 190 may be one or more Processing Systems that may be used to perform computerized advanced processing such as, for example, computations on imaging information to create new views of the information, for example, volume rendering and/or other types of processing, for example image enhancement, volume quantification, blood-flow quantification, and the like. In one embodiment, such processing functionality may reside in a computing device and/or system separate from computing system 150 while in another embodiment the processing functionality may reside within computing system 150.
In other embodiments, other computing devices and/or systems that store, provide, acquire, and/or otherwise manipulate medical data may also be coupled to the network 190 and may be in communication with one or more of the devices illustrated in
Depending on the embodiment, other devices discussed herein may include some or all of the same components discussed above with reference to the computing system 150 and may perform some or all of the functionality discussed herein.
As mentioned above, various of the image storage 122, the server 120, the user preferences and rules database 124, the external sensor(s) 125, the imaging devices 170, the CDS server 171, and/or other components described above may or may not be considered a part of the computing system 150. For example, in some embodiments one or more of these components may be implemented as part of the computing system 150, may be in direct communication with the computing system 150, and/or may be in indirect communication (e.g., over network 190) with the computing system 150.
X. ADDITIONAL EMBODIMENTSVarious embodiments of the present disclosure may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or mediums) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.
For example, the functionality described herein may be performed as software instructions are executed by, and/or in response to software instructions being executed by, one or more hardware processors and/or any other suitable computing devices. The software instructions and/or other executable code may be read from a computer readable storage medium (or mediums).
The computer readable storage medium can be a tangible device that can retain and store data and/or instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device (including any volatile and/or non-volatile electronic storage devices), a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a solid state drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions (as also referred to herein as, for example, “code,” “instructions,” “module,” “application,” “software application,” and/or the like) for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. Computer readable program instructions may be callable from other instructions or from itself, and/or may be invoked in response to detected events or interrupts. Computer readable program instructions configured for execution on computing devices may be provided on a computer readable storage medium, and/or as a digital download (and may be originally stored in a compressed or installable format that requires installation, decompression or decryption prior to execution) that may then be stored on a computer readable storage medium. Such computer readable program instructions may be stored, partially or fully, on a memory device (e.g., a computer readable storage medium) of the executing computing device, for execution by the computing device. The computer readable program instructions may execute entirely on a user's computer (e.g., the executing computing device), partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.
Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart(s) and/or block diagram(s) block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions and/or modules into its dynamic memory and send the instructions over a telephone, cable, or optical line using a modem. A modem local to a server computing system may receive the data on the telephone/cable/optical line and use a converter device including the appropriate circuitry to place the data on a bus. The bus may carry the data to a memory, from which a processor may retrieve and execute the instructions. The instructions received by the memory may optionally be stored on a storage device (e.g., a solid state drive) either before or after execution by the computer processor.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. In addition, certain blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states relating thereto can be performed in other sequences that are appropriate.
It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions. For example, any of the processes, methods, algorithms, elements, blocks, applications, or other functionality (or portions of functionality) described in the preceding sections may be embodied in, and/or fully or partially automated via, electronic hardware such application-specific processors (e.g., application-specific integrated circuits (ASICs)), programmable processors (e.g., field programmable gate arrays (FPGAs)), application-specific circuitry, and/or the like (any of which may also combine custom hard-wired logic, logic circuits, ASICs, FPGAs, etc. with custom programming/execution of software instructions to accomplish the techniques).
Any of the above-mentioned processors, and/or devices incorporating any of the above-mentioned processors, may be referred to herein as, for example, “computers,” “computer devices,” “computing devices,” “hardware computing devices,” “hardware processors,” “processing units,” and/or the like. Computing devices of the above-embodiments may generally (but not necessarily) be controlled and/or coordinated by operating system software, such as Mac OS, iOS, Android, Chrome OS, Windows OS (e.g., Windows XP, Windows Vista, Windows 7, Windows 8, Windows 10, Windows Server, etc.), Windows CE, Unix, Linux, SunOS, Solaris, Blackberry OS, VxWorks, or other suitable operating systems. In other embodiments, the computing devices may be controlled by a proprietary operating system. Conventional operating systems control and schedule computer processes for execution, perform memory management, provide file system, networking, I/O services, and provide a user interface functionality, such as a graphical user interface (“GUI”), among other things.
As described above, in various embodiments certain functionality may be accessible by a user through a web-based viewer (such as a web browser), or other suitable software program). In such implementations, the user interface may be generated by a server computing system and transmitted to a web browser of the user (e.g., running on the user's computing system). Alternatively, data (e.g., user interface data) necessary for generating the user interface may be provided by the server computing system to the browser, where the user interface may be generated (e.g., the user interface data may be executed by a browser accessing a web service and may be configured to render the user interfaces based on the user interface data). The user may then interact with the user interface through the web-browser. User interfaces of certain implementations may be accessible through one or more dedicated software applications. In certain embodiments, one or more of the computing devices and/or systems of the disclosure may include mobile computing devices, and user interfaces may be accessible through such mobile computing devices (for example, smartphones and/or tablets).
Many variations and modifications may be made to the above-described embodiments, the elements of which are to be understood as being among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure. The foregoing description details certain embodiments. It will be appreciated, however, that no matter how detailed the foregoing appears in text, the systems and methods can be practiced in many ways. As is also stated above, it should be noted that the use of particular terminology when describing certain features or aspects of the systems and methods should not be taken to imply that the terminology is being re-defined herein to be restricted to including any specific characteristics of the features or aspects of the systems and methods with which that terminology is associated.
Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements, and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment.
Conjunctive language such as the phrase “at least one of X, Y, and Z,” or “at least one of X, Y, or Z,” unless specifically stated otherwise, is to be understood with the context as used in general to convey that an item, term, etc. may be either X, Y, or Z, or a combination thereof. For example, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list. Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y, and at least one of Z to each be present.
The term “a” as used herein should be given an inclusive rather than exclusive interpretation. For example, unless specifically noted, the term “a” should not be understood to mean “exactly one” or “one and only one”; instead, the term “a” means “one or more” or “at least one,” whether used in the claims or elsewhere in the specification and regardless of uses of quantifiers such as “at least one,” “one or more,” or “a plurality” elsewhere in the claims or specification.
The term “comprising” as used herein should be given an inclusive rather than exclusive interpretation. For example, a general purpose computer comprising one or more processors should not be interpreted as excluding other computer components, and may possibly include such components as memory, input/output devices, and/or network interfaces, among others.
While the above detailed description has shown, described, and pointed out novel features as applied to various embodiments, it may be understood that various omissions, substitutions, and changes in the form and details of the devices or processes illustrated may be made without departing from the spirit of the disclosure. As may be recognized, certain embodiments of the inventions described herein may be embodied within a form that does not provide all of the features and benefits set forth herein, as some features may be used or practiced separately from others. The scope of certain inventions disclosed herein is indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.
Claims
1. A database computing system for processing digital medical images, the computing system comprising:
- an electronic display;
- an input device;
- a non-transitory computer-readable storage medium configured to store software instructions; and
- one or more computer processors in communication with the electronic display, the input device, and the non-transitory computer-readable medium, the one or more computer processors configured to execute the software instructions in order to cause the computing system to: receive an indication of a selection of a first medical image of a first image series of a first medical exam associated with a patient, wherein the first image series includes a first plurality of medical images; determine, based on one or more matching rules, a comparison image series of a comparison medical exam for comparison with the first image series, wherein the comparison image series includes a second plurality of comparison images, and wherein the comparison medical exam is also associated with the patient and was acquired at a time different than the first medical exam; identify, based on one or more attributes of the first medical image, a first comparison image of the comparison image series for comparison with the first medical image; generate user interface data useable for displaying a user interface on the electronic display, the user interface including at least: the first medical image of the first image series displayed in a first image pane in a first location; and the first comparison image of the second image series displayed in a second image pane in a second location; in response to a first user input indicating a direction of movement within the first image series: update the user interface data such that, in the user interface, the first medical image is replaced with a second medical image of the first image series that is adjacent to the first medical image in the first image series, wherein the second medical image replaces the first medical image in the first image pane; identify, based on one or more attributes of the second medical image, a second comparison image of the comparison image series for comparison with the second medical image; and further update the user interface data such that, in the user interface, the first comparison image is replaced with the second comparison image, wherein the second comparison image replaces the first comparison image in the second image pane; and in response to a second user input indicating selection of a second image series of the first medical exam: determine, based on one or more attributes of the second medical image, a first medical image of the second image series for display; update the user interface data such that, in the user interface, the second medical image is replaced with the first medical image of the second image series, wherein the first medical image of the second image series replaces the second medical image of the first image series in the first image pane; identify, based on one or more attributes of the second image series, a second comparison image series of the comparison medical exam for comparison with the second image series; identify, based on one or more attributes of the first medical image of the second image series, a first comparison image of the second comparison image series for comparison with the first medical image of the second image series; and further update the user interface data such that, in the user interface, the second comparison image of the comparison image series is replaced with the first comparison image of the second comparison image series, wherein the first comparison image of the second comparison image series replaces the second comparison image of the comparison image series in the second image pane.
2. The computing system of claim 1, wherein the first comparison image of the comparison image series is further identified based on one or more matching rules.
3. The computing system of claim 1, wherein the second comparison image of the comparison image series is further identified based on one or more matching rules.
4. The computing system of claim 1, wherein the first comparison image of the second comparison image series is further identified based on one or more matching rules.
5. The computing system of claim 1, wherein the first location of the first image pane and the second location of the second image pane are adjacent to one another.
6. A database computing system for processing digital images, the computing system comprising:
- an electronic display;
- an input device;
- a non-transitory computer-readable storage medium configured to store software instructions; and
- one or more computer processors in communication with the electronic display, the input device, and the non-transitory computer-readable medium, the one or more computer processors configured to execute the software instructions in order to cause the computing system to: display, on the electronic display, a first medical image of a first image series of a first medical exam in a first image pane, wherein the first medical exam is associated with a patient; display, on the electronic display, a first medical image of a comparison image series of a second medical exam in a second image pane, wherein the comparison medical exam is also associated with the patient, wherein the second medical exam was acquired at a time different from the first medical exam, and wherein the first medical image of the comparison image series matches the first medical image of the first image series in at least one attribute such that the first medical image of the first image series and first medical image of the comparison image series may be usefully compared for diagnosis; in response to a first user input indicating a change from the first medical image of the first image series to another medical image of the first image series: replace, in the first image pane, the first medical image of the first image series with the another medical image of the first image series; and replace, in the second image pane, the first medical image of the comparison image series with another medical image of the comparison image series, wherein the another medical image of the comparison image series matches the another medical image of the first image series in at least one attribute such that the another medical image of the first image series and the another medical image of the comparison image series may be usefully compared for diagnosis; and in response to a second user input indicating a change from the first image series of the first medical exam to another image series of the first medical exam: replace, in the first image pane, the first medical image of the first image series with the another medical image of the another image series of the first medical exam; and replace, in the second image pane, the first medical image of the comparison image series with another medical image of another image series of the second medical exam, wherein the another medical image of the another image series of the second medical exam matches the another medical image of the another image series of the first medical exam in at least one attribute such that the another medical image of the another image series of the first medical exam and the another medical image of the another image series of the second medical exam may be usefully compared for diagnosis.
7. The computing system of claim 6, wherein the one or more computer processors are configured to execute the software instructions in order to further cause the computing system to:
- in response to a third user input, freeze at least the second image pane such that a medical image displayed in the second image pane is not replaced in response to the first and/or second user inputs.
8. The computing system of claim 6, wherein the first and second image panes are adjacent to one another.
9. The computing system of claim 6, wherein the second medical exam was acquired before the first medical exam.
10. A method comprising:
- by one or more computer processors configured to execute software instructions: displaying, on an electronic display, a first medical image of a first image series of a first medical exam in a first image pane, wherein the first medical exam is associated with a patient; displaying, on the electronic display, a first medical image of a comparison image series of a second medical exam in a second image pane, wherein the comparison medical exam is also associated with the patient, wherein the second medical exam was acquired at a time different from the first medical exam, and wherein the first medical image of the comparison image series matches the first medical image of the first image series in at least one attribute such that the first medical image of the first image series and first medical image of the comparison image series may be usefully compared for diagnosis; in response to a first user input indicating a change from the first medical image of the first image series to another medical image of the first image series: replacing, in the first image pane, the first medical image of the first image series with the another medical image of the first image series; and replacing, in the second image pane, the first medical image of the comparison image series with another medical image of the comparison image series, wherein the another medical image of the comparison image series matches the another medical image of the first image series in at least one attribute such that the another medical image of the first image series and the another medical image of the comparison image series may be usefully compared for diagnosis; and in response to a second user input indicating a change from the first image series of the first medical exam to another image series of the first medical exam: replacing, in the first image pane, the first medical image of the first image series with the another medical image of the another image series of the first medical exam; and replacing, in the second image pane, the first medical image of the comparison image series with another medical image of another image series of the second medical exam, wherein the another medical image of the another image series of the second medical exam matches the another medical image of the another image series of the first medical exam in at least one attribute such that the another medical image of the another image series of the first medical exam and the another medical image of the another image series of the second medical exam may be usefully compared for diagnosis.
11. The method of claim 10 further comprising:
- by the one or more computer processors configured to execute software instructions: in response to a third user input, freezing at least the second image pane such that a medical image displayed in the second image pane is not replaced in response to the first and/or second user inputs.
12. The method of claim 10, wherein the first and second image panes are adjacent to one another.
13. The method of claim 10, wherein the second medical exam was acquired before the first medical exam.
14. A database computing system for processing digital medical images, the computing system comprising:
- an electronic display;
- an input device;
- a non-transitory computer-readable storage medium configured to store software instructions; and
- one or more computer processors in communication with the electronic display, the input device, and the non-transitory computer-readable medium, the one or more computer processors configured to execute the software instructions in order to cause the computing system to: access a plurality of medical images associated with a plurality of medical exams of a patient; analyze the plurality of medical images to determine, for each medical image, respective attributes of the medical images; sort the plurality of medical images into a plurality of image series based on the attributes of the medical images, wherein the plurality of image series includes at least a first image series and a second image series; analyzing the first and second image series to determine pairs of medical images of the respective image series that are related to one another; generate user interface data useable for displaying a user interface on the electronic display, the user interface including at least: a first medical image of the first image series displayed in a first image pane in a first location; and a first medical image of the second image series displayed in a second image pane in a second location; and in response to a first user input indicating a direction of movement within the first image series: update the user interface data such that, in the user interface, the first medical image of the first image series is replaced with a second medical image of the first image series that is adjacent to the first medical image of the first image series in the first image series, wherein the second medical image of the first image series replaces the first medical image in the first image pane; identify, based on one or more attributes of the second medical image of the first image series, a second medical image of the second image series for comparison with the second medical image of the first image series; and further update the user interface data such that, in the user interface, the first medical image of the second image series is replaced with the second medical image of the second image series, wherein the second medical image of the second image series replaces the first medical image of the second image series in the second image pane.
15. The computing system of claim 14, wherein the one or more computer processors are further configured to execute the software instructions in order to cause the computing system to:
- in response to a second user input indicating a direction of movement within the first image series: update the user interface data such that, in the user interface, the first medical image of the first image series is replaced with a second medical image of the first image series that is adjacent to the first medical image of the first image series in the first image series, wherein the second medical image of the first image series replaces the first medical image in the first image pane; determine, based on one or more attributes of the second medical image of the first image series, that no medical image of the second image series corresponding to the second medical image of the first image series exists; and further update the user interface data such that, in the user interface, the first medical image of the second image series is replaced with a placeholder in the second image pane.
16. The computing system of claim 15, wherein the placeholder comprises at least one of a blank image or an absence of an image.
17. The computing system of claim 14, wherein analyzing the plurality of medical images to determine, for each medical image, respective attributes of the medical images comprises:
- analyzing the plurality of medical images to determine, for each medical image, respective view types of the medical images;
18. The computing system of claim 14, wherein each image series of the plurality of image series comprises a series of medical images having a common view type
19. The computing system of claim 14, wherein the plurality of medical images comprise chest radiograph images.
20. The computing system of claim 14, wherein analyzing the first and second image series to determine pairs of medical images of the respective image series that are related to one another comprises:
- analyzing the first and second image series to determine pairs of medical images of the respective image series that are from a same exam and/or acquired at a similar time.
21. The computing system of claim 14, wherein the one or more computer processors are further configured to execute the software instructions in order to cause the computing system to:
- determine the first and second locations of the respective first and second image panes based on one or more user preferences.
22. The computing system of claim 14, wherein the one or more computer processors are further configured to execute the software instructions in order to cause the computing system to:
- determine an association between the first image pane and the first image series based on one or more user preferences.
23. The computing system of claim 14, wherein the one or more computer processors are further configured to execute the software instructions in order to cause the computing system to:
- registering the medical images of the first image series so as to enable efficient comparison of the medical images of the first image series.
Type: Application
Filed: Apr 27, 2016
Publication Date: Feb 16, 2017
Inventors: Murray A. Reicher (Rancho Santa Fe, CA), Evan K. Fram (Paradise Valley, AZ)
Application Number: 15/140,363
