Method and Apparatus for Enhancing Images
A method and an apparatus for enhancing images are provided. An input image generated by an imaging device, such as a short-wavelength infrared (SWIR) camera, sensor, or any other imaging device, is processed to produce an input histogram representing the distribution of pixel intensities in the input image. Histogram start and end points are determined, and a variable plateau profile is calculated depending upon the type of enhancement desired. The plateau profile could take on any desired shape or size, and two or more plateau regions could be included in the profile. The input histogram is clipped along the variable plateau profile. A cumulative histogram is constructed from the clipped histogram, and is normalized. The input image is then transformed into an enhanced output image using the normalized cumulative histogram as a look-up table.
This application is a continuation application of and claims the benefit of priority to U.S. patent application Ser. No. 11/762,184 filed Jun. 13, 2007, now U.S. Pat. No. 8,218,868, the entire disclosure of which is expressly incorporated herein by reference.
BACKGROUND OF THE INVENTION1. Field of the Invention
The present invention relates to image enhancement, and more specifically, to a method and apparatus for enhancing images using variable plateau histogram equalization.
2. Related Art
In various imaging applications, such as military imaging applications, it is often desirable to enhance an image acquired by a camera or sensor to reveal additional detail in the image. For example, where images are taken in the presence of a bright light, details in shadows and in bright areas are often lost. Likewise, details are often lost in dark areas of dark images. As such, it is desirable to reveal such details without compromising overall image quality.
A number of image enhancement methods have, in the past, been developed. Examples of such methods include contrast stretching, histogram equalization, histogram projection, and plateau equalization. In contrast stretching, pixel intensity distributions are expanded (“stretched”) to occupy a full range of possible values. According to this method, minimum and maximum pixel intensity values are measured (or estimated), and offset and gain correction values are applied to the pixel intensity values corresponding to each pixel in the image, so that the full range of intensity values is occupied. In histogram equalization, the pixel intensity distribution is attenuated or flattened. According to this method, an input histogram is constructed, which includes the total number of pixels corresponding to each intensity level within the image. Then, a cumulative histogram is constructed and normalized, and each pixel in the image is transformed using the normalized cumulative histogram as a look-up table. In histogram projection, a histogram is constructed, consisting of occupied and unoccupied levels. A cumulative histogram is then constructed and normalized, and each pixel in the image is transformed using the normalized cumulative histogram as a look-up table. In plateau equalization, an input histogram is constructed and clipped at a specified level or “plateau.” Then, a cumulative histogram is constructed and normalized, and each pixel in the image is transformed using the normalized cumulative histogram as a look-up table.
The foregoing image enhancement methods suffer from a number of drawbacks. For example, existing image enhancement methods do not adequately reveal detail in specific regions of an image, such as shadows or bright areas. Rather, these methods uniformly enhance an entire image, which may result in distortion of other regions of the image for the sake of enhancing a desired region of the image. Further, existing image enhancement methods cannot be easily adapted, without manual adjustment, to enhance images in different environments (e.g., by dynamically altering enhancement characteristics in response to changes in camera sensitivity).
Accordingly, what would be desirable, but has not yet been provided, is a method and apparatus for enhancing images which addresses the foregoing limitations of existing image enhancement methods.
SUMMARY OF THE INVENTIONThe present invention relates to a method and apparatus for providing image enhancement, which can be utilized with any suitable imaging device (e.g., short wavelength infrared (SWIR) camera, sensor, imager, or any other device) to enhance images produced by the device. An input histogram corresponding to an image to be enhanced is constructed. The input histogram represents a pixel intensity distribution corresponding to the image. A variable plateau is established, and the input histogram is clipped at the plateau. The variable plateau could be formed in any desired size or shape, and can be varied to adapt to changing scene conditions, light levels, camera conditions, or in response to other conditions. Multiple plateaus can be established for enhancing desired regions of the image, such as dark regions (shadows) and bright areas of the image. A cumulative histogram is constructed from the clipped histogram, and is normalized to produce a normalized cumulative histogram. The normalized cumulative histogram is used as a look-up table to transform pixels of the input image to produce an enhanced output image. Automatic gain control (AGC) functions can be applied to provide additional enhancement of the image.
These and other important objects and features of the invention will be apparent from the following Detailed Description of the Invention, taken in connection with the accompanying drawings, in which:
The present invention relates to a method and apparatus for enhancing images. Desired aspects of an image, such as shadow regions or bright areas of the image, can be enhanced by the present invention to provide additional detail. An input image acquired by an imaging device, such as a short-wavelength infrared (SWIR) camera, sensor, or any other imaging device, is processed by the present invention to produce an input histogram representing the distribution of pixel intensities in the input image. Optionally, saturated and/or low pixels in the input image can be excluded. Histogram start and end points are determined, and a variable plateau is calculated depending upon the type of enhancement desired. The plateau may be varied along the histogram and can take on any desired shape, size, or number of plateau regions. When the plateau is calculated, the histogram is clipped therealong. A cumulative histogram is constructed from the clipped histogram, and is normalized. The input image is then transformed into an enhanced output image using the normalized cumulative histogram as a look-up table. Optional non-linear operations can be performed on the normalized cumulative histogram. The enhancement method can be automatically varied to adapt to various imaging device types as well as changes in scene conditions.
In process block 14, automatic gain control (AGC) processes are initiated for controlling the gain of the output image. In step 16a, the average pixel value of an image frame is calculated. Then, in step 16b, if the calculated average pixel value is at or near a pre-defined minimum or maximum pixel value, AGC processing is disabled. This prevents the present invention from remaining indefinitely in an AGC processing condition which can result if a camera is operating in a darkened room and lights are abruptly turned on and saturated pixels are removed from the image, or if a camera is operating in a bright-lit room and lights are abruptly turned off and low pixels are removed from the image. Parameters other than the average pixel value can be used for this process, including the number of saturated pixels expressed as a percentage of the total pixels collected, or the number of low pixels expressed as a percentage of the total pixels collected.
In step 16c, the number of saturated pixels in the input image is counted. In step 16d, the saturated pixels are used for AGC adjustment of an imager or camera, as discussed below with reference to
In step 18, histogram start and end values are determined, using a portion (fraction) of the number of histogram pixels. The start of the histogram is the first bin to exceed an input minimum or “floor” value and the end of the histogram is the last bin to exceed the input floor value. When the method 10 is first carried out, the input floor level could be set to a pre-defined level, which could be programmed by a user. Other methods to determine the start and end of the histogram are possible. For example, starting at the lowest histogram bin, the start of the histogram could be found when the cumulative pixels are at least a percentage (e.g., 1.5%) of the total number of pixels collected. Moving through the histogram, the end of the histogram could then be found when the cumulative pixels are at least 100% minus a percentage (e.g., 100%−1.5%=98.5%) of the total number of pixels collected. In this example, the histogram, from start to end, would contain approximately 97% of the total number of pixels collected. Such a method has been found to be less sensitive to noise. Another alternative is to use the input floor value, described above, for the number of pixels needed at each end of the histogram instead of using a percentage of the total number collected. Still further, other methods for calculating a floor value could be utilized, such as taking a fraction of the average number of pixels per bin, or a fraction of the average pixel value per bin.
The plateau profile could include two plateaus (illustratively, plateaus “A” and “B”) which correspond, respectively, to a “dark” plateau and a “light” plateau. The “dark” plateau represents a clipping level for enhancing details in shadows or dark regions of the input image. The “light” plateau represents a clipping level for enhancing details in light regions of the input image. In step 20, a plateau boundary is calculated, which represents the bin at which the clipping level transitions from plateau A to plateau B. Of course, the plateaus could take any shape and could be adjusted as desired. Importantly, the plateaus and the transitions between plateaus can be varied to highlight desired regions of the input image, and could also be automatically changed (e.g., by electronic means) to adjust to various operating parameters of different cameras or sensors, as well as to changing light levels or other scene conditions. Additionally, the plateau profile could vary from bin to bin of the histogram.
After the plateau boundary has been calculated in step 20, in step 22, the input histogram is clipped at the plateaus, such that all values above the clipping levels are reduced to the clipping levels. If two plateaus (such as plateaus A and B, discussed above) are defined, the to histogram is clipped at the two plateaus. After clipping, in step 24, a cumulative histogram is constructed from the clipped histogram. Each bin in the cumulative histogram includes the total number of pixels in the current bin plus the number of pixels in the preceding bin. The cumulative histogram is thus the integral of the distribution represented by the clipped input histogram.
In step 26, the cumulative histogram is normalized to the number of output levels in the output image. For example, if 256 possible output values are in the output image, the cumulative histogram would be normalized to the 256 output values from 0 to 255. Optional step 30 could then be carried out, wherein non-linear functions (discussed below with respect to
Once the desired enhancement type has been selected and the corresponding gain of the imager adjusted, a determination is made in step 49 as to whether gain adjustment is needed. If not, processing ends. If a positive determination is made, step 50 occurs, wherein a determination is made as to whether additional gain is required, even though the imager may already be adjusted to a maximum gain level. If a positive determination is made, optional step 52 occurs, wherein enhancement parameters are further adjusted for low light level conditions. In step 54, the gain of the imager is adjusted to the gain level set in either of steps 44, 46, or 48, or as modified in optional step 52.
In optional process 56, average AGC parameters for the desired number of frames of the input image can be determined, and adjusted as desired. Such a feature can operate in conjunction with the other processes shown in
The gain adjustment feature of the present invention allows for gain changes in steps of roughly 2.2. Look-up tables can be used for each gain range (operational setting or “OPR”) to set the various electrical parameters of the imager. If the gain is changed while the AGC parameters are averaged over n frames, the output of the averager will not immediately change in response to the step gain change. Thus, it is possible that the present invention will keep changing the gain until the change appears at the output of the averager, causing the AGC to overshoot and hunt or oscillate. In the present embodiment, for example, the camera runs at 60 frames per second (˜0.5 seconds for 32 frames) and the AGC algorithm runs every 0.1 second. One solution for addressing this possibility is to lower the AGC sample rate after a gain change and wait for the change to propagate through the averager before examining its output.
The AGC features of the present invention could be augmented by the inclusion of a “boxcar averager” to average AGC data from 1 to 32 frames, so as to provide for continuous gain adjustments that are immediately available in the output image. The boxcar averager includes a common timing scheme for all data streams. The boxcar averager could be implemented in the following steps: (1) after a gain change, wait a first predetermined period of time for electrical adjustments to settle (for example, 0.1 seconds per AGC period); (2) wait a second predetermined period of time for the camera hardware to acquire at least one new frame at the new gain setting (for example, 1/60 second), and calculate the new AGC parameters; (3) store the new AGC parameters (i.e., “fill” all the boxcars with the new AGC parameters); and (4) resume normal boxcar operation and normal AGC calculations. It has been found that such an implementation results in gain changes that can be quickly implemented in the output image, without oscillation, when operated with 32 frame averaging.
The interpolation process of the present invention could be advantageously implemented using two lookup tables, wherein the first lookup table contains the cumulative histogram data corresponding to the input image, and the second lookup table contains the differences between entries in the first lookup table. The difference calculation can be performed as the tables are created, to prevent the need for calculating in real time. If each pixel of the input image is twelve bits in length, the most significant 8 bits would be used as a lookup key for the lookup tables. A number proportional to the least significant 4 bits is added to the most significant 8 bits, resulting in an interpolated 12 bit pixel. This could be expressed mathematically as follows: interpolated output pixel value=(8-bit cumulative histogram bin value)*bin divider+(least significant 4 bits of input pixel)*difference. It has been found that for a small number of bins, e.g., 64, interpolation has improved the output image in many observed circumstances. For 256 bins, interpolation has also improved the image in a few observed circumstances.
As shown in
The various plateaus capable of being implemented by the present invention, such as those shown in
Enhancement control module 234 and contrast enhancement module 236 include the processing logic described herein for implementing variable plateau histogram processing of the image. The enhanced image is sent to a test pattern and stamp insert module 238, in addition to a frame statistics averager module 250. The frame statistics averager module 250 receives information from the statistics control module 248, which allows a user to specify frame or frames to which automatic gain control (AGC) should be applied. The AGC mode select module 254 and the AGC module 252 include the processing logic described herein for implementing enhanced AGC functionality.
The test pattern/stamp insert module 238 receives a test mode signal from the test mode control module 240, which allows a user to specify whether to insert a test pattern into the video stream. In optional video lookup module 244, a lookup table 246 could be utilized to construct the output image of the camera 200 by transforming the input image using the lookup table, for performing optional image correction functions such as gamma correction, etc. The output image is then stored in a video dual-port memory 262, and is subsequently sent to a digital-to-analog (DAC) converter 264 for conversion back to an analog video signal. The analog video signal could be adjusted using a composite synchronization adjustment 266, whereupon the adjusted signal is then available for use.
The camera 200 includes a digital data source selector 256 for allowing a user to select a desired digital signal which corresponds to a desired position along the video processing chain shown in
The image enhancement method of the present invention (and, in particular, the variable histogram plateau feature) can be automatically adjusted in response to various camera designs and changing environmental conditions. For example, as brightness levels decrease in a scene that is the subject of monitoring over time, the present invention could automatically generate a plurality of histogram plateaus (each of any desired shape or size) that are customized to compensate for changing light levels. Thus, for example, when dim conditions are detected, conventional histogram or plateau equalization processing could be implemented, wherein the post-processing (e.g., xp and ramp non-linear processes) of the present invention are disabled and a straight plateau is defined and implemented. Further, one or more user-defined plateaus, and/or post-processing settings could be programmed and implemented as desired.
If the present invention is implemented in a standalone system capable of interfacing with a plurality of imaging systems, such a system could be programmed with a plurality of plateaus, an optimal one of which can be automatically selected for a particular camera type or manufacture. Moreover, the present invention could be adapted to operate with one or more sensors which monitor optical conditions within the camera itself, such that the enhancement method is dynamically adapted to conform to such conditions.
Having thus described the invention in detail, it is to be understood that the foregoing description is not intended to limit the spirit and scope thereof. What is desired to be protected by Letters Patent is set forth in the appended claims.
Claims
1. A method for enhancing an image, comprising the steps of:
- constructing an input histogram corresponding to the image, the input histogram representing a pixel intensity distribution corresponding to the image;
- calculating a midpoint based upon an offset and a threshold value of the input histogram;
- establishing first and second horizontal plateaus to be applied to the input histogram, the first horizontal plateau extending to the midpoint and the second horizontal plateau extending from the midpoint;
- clipping the input histogram at the first and second horizontal plateaus to produce a clipped input histogram;
- constructing a cumulative histogram from the clipped input histogram;
- normalizing the cumulative histogram to produce a normalized cumulative histogram; and
- transforming the input image using the normalized cumulative histogram to produce an enhanced output image corresponding to the input image, wherein at least a portion of the input image is enhanced in the output image.
2. The method of claim 1, further comprising excluding saturated pixels in the image from the input histogram.
3. The method of claim 2, further comprising excluding low level pixels in the image from the input histogram.
4. The method of claim 1, further comprising performing non-linear operations on the normalized cumulative histogram prior to transforming the input image.
5. The method of claim 1, further comprising interpolating between bins to produce the enhanced output image.
6. The method of claim 1, further comprising selectively adjusting gain in the output image.
7. The method of claim 1, wherein the step of establishing the first and second horizontal plateaus further comprises establishing a dark plateau for enhancing dark regions in the image and a light plateau for enhancing light regions in the image.
8. The method of claim 1, further comprising identifying a start bin of the input histogram when cumulative pixels are at least a first percentage of a total number of pixels collected.
9. The method of claim 8, further comprising identifying an end bin of the input histogram when cumulative pixels are at least a second percentage.
10. An apparatus for enhancing an image, comprising:
- an imaging device for capturing an image;
- a contrast enhancement module for receiving and processing the image, wherein the contrast enhancement module constructs an input histogram corresponding to the image, calculates a midpoint based on an offset and a threshold value of the input histogram, constructs first and second horizontal plateaus for clipping the input histogram, the first horizontal plateau extending to the midpoint and the second horizontal plateau extending from the midpoint, and clips the input histogram along the first and second horizontal plateaus and generates a normalized cumulative histogram; and
- means for generating an output image based upon the normalized cumulative histogram.
11. The apparatus of claim 10, wherein the first horizontal plateau enhances dark areas of the input image and the second horizontal plateau enhances bright areas of the input image.
12. The apparatus of claim 10, further comprising an automatic gain control module for adjusting gain of the input image.
13. The apparatus of claim 12, wherein the automatic gain control modules includes user-selectable gain control modes for providing additional enhancement of the input image.
14. The apparatus of claim 10, wherein the contrast enhancement module excludes saturated pixels in the image from the input histogram.
15. The apparatus of claim 10, wherein the contrast enhancement module excludes low level pixels in the image from the input histogram.
16. The apparatus of claim 10, wherein the contrast enhancement module performs non-linear operations on the normalized cumulative histogram.
17. The apparatus of claim 10, further comprising means for interpolating between bins to produce the enhanced output image.
18. The method of claim 10, wherein the contrast enhancement module identifies a start bin of the input histogram when cumulative pixels are at least a first percentage of a total number of pixels collected.
19. The method of claim 18, wherein the contrast enhancement module identifies an end bin of the input histogram when cumulative pixels are at least a second percentage.
Type: Application
Filed: Jul 10, 2012
Publication Date: Nov 1, 2012
Inventors: Mark C. Stern (Princeton, NJ), Thomas Sudol (Yardley, PA)
Application Number: 13/545,468
International Classification: G06T 5/40 (20060101);