System and method for green imbalance compensation in images

A method and imaging device are provided. A pixel output signal is obtained for each green pixel in a group of pixels of an image. The group of pixels includes a plurality of pixel arrays, where each pixel array of the plurality of pixel arrays includes a first green pixel and a second green pixel. A green imbalance value is determined for the group of pixels based on the obtained pixel output signals. For each pixel array, a difference between the pixel output signal for the first green pixel and the pixel output signal for the second green pixel is calculated. An average of the calculated differences is determined. The pixel output signal of at least one of the green pixels in the group of pixels is adjusted based on the green imbalance value.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
CROSS-REFERENCE TO RELATED APPLICATIONS

This disclosure claims priority to U.S. Provisional Patent Application No. 61/858,455, filed on Jul. 25, 2013, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The technology described in this document relates generally to image signal processing (ISP) methods and more particularly to green imbalance compensation in imaging devices.

BACKGROUND

Semiconductor image sensors are used to sense radiation that includes, for example, visible light. Complementary metal-oxide-semiconductor (CMOS) image sensors and charge-coupled device (CCD) sensors are widely used in digital cameras and mobile phone cameras. These sensors utilize an array of pixels located in a substrate, where the pixels include photodiodes and transistors. The pixels absorb radiation projected toward the substrate and convert the absorbed radiation into electrical signals.

SUMMARY

The present disclosure is directed to an imaging device and a method of compensating for green pixel imbalance in an image captured by a pixel array. In an example method of compensating for green pixel imbalance in an image captured by a pixel array, a pixel output signal is obtained for each green pixel in a group of pixels of the image captured by the pixel array. The group of pixels includes a plurality of pixel arrays, where each pixel array of the plurality of pixel arrays includes (i) a first green pixel in a row that includes a red pixel, and (ii) a second green pixel in a row that includes a blue pixel. A green imbalance value is determined for the group of pixels based on the obtained pixel output signals. For each pixel array of the plurality of pixel arrays, a difference between the pixel output signal for the first green pixel and the pixel output signal for the second green pixel is calculated. An average of the calculated differences is determined, where the green imbalance value is equal to the average. The pixel output signal of at least one of the green pixels in the group of pixels is adjusted based on the green imbalance value.

In another example method of compensating for green pixel imbalance in an image captured by a pixel array, a pixel output signal for each green pixel in a group of pixels of the image captured by the pixel array is obtained. The group of pixels includes a plurality of non-overlapping pixel arrays. Each pixel array of the plurality of pixel arrays includes (i) a first green pixel in a row that includes a red pixel, and (ii) a second green pixel in a row that includes a blue pixel. A green imbalance value for the group of pixels is determined based on the obtained pixel output signals. For each pixel array of the plurality of pixel arrays, a difference between the pixel output signal for the first green pixel and the pixel output signal for the second green pixel is calculated. For each of the calculated differences, it is determined if the calculated difference is a result of image edge pixels in the group of pixels. The pixel output signal of at least one of the green pixels in the group of pixels is adjusted based on the green imbalance value.

In another example, an imaging device includes a pixel array and a processor for processing pixel output signals received from the pixel array. The processor is configured to obtain a pixel output signal for each green pixel in a group of pixels of an image captured by the pixel array. The group of pixels includes a plurality of pixel arrays, where each pixel array of the plurality of pixel arrays includes (i) a first green pixel in a row that includes a red pixel, and (ii) a second green pixel in a row that includes a blue pixel. The processor is also configured to, for each pixel array of the plurality of pixel arrays, calculate a difference between the pixel output signal for the first green pixel and the pixel output signal for the second green pixel. An average of the calculated differences is determined, where the average is equal to a green imbalance value for the group of pixels. The processor is further configured to adjust the pixel output signal of at least one of the green pixels in the group of pixels based on the green imbalance value.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 depicts an array of pixels using a Bayer pattern color filter.

FIG. 2 depicts an array of pixels including a group of pixels, where the group of pixels includes a 2×2 pixel array.

FIG. 3 is a flowchart of a process for compensating for green-green imbalance in an image captured by a pixel array.

FIG. 4 is a flowchart illustrating an example method for determining positive and negative thresholds used in compensating for green pixel imbalance in an image captured by a pixel array.

FIG. 5 is a flowchart illustrating an example method for compensating for green-green imbalance in an image captured by a pixel array.

FIG. 6 is a block diagram of an example imaging device that includes an image processing unit configured to compensate for green pixel imbalance in an image captured by a pixel array.

FIG. 7 is a flowchart illustrating an example method of compensating for green pixel imbalance in an image captured by a pixel array.

 DETAILED DESCRIPTION

FIG. 1 depicts an array of pixels 100 using a Bayer pattern color filter. In an example, the array of pixels 100 is part of a CMOS or CCD imaging device. In the CMOS or CCD imaging device, each pixel of the array 100 produces a signal that indicates an intensity of light impinging on the pixel, and a magnitude of the signal produced by each pixel is proportional to the intensity of the light impinging on the pixel. The signals produced by the pixels of the array 100 are digitized and stored in order to display a corresponding image on a monitor or to otherwise provide information about the captured image.

In order to allow capturing of a color image, the array of pixels 100 utilizes a Bayer pattern. As illustrated in FIG. 1, the Bayer pattern includes red, green, and blue pixels, where each pixel is sensitive to only one color or spectral band. Thus, photons having a wavelength corresponding to a red, green, or blue color are detected by respective red, green, or blue pixels of the array of pixels 100. To construct the Bayer pattern, a color filter array is placed in front of the pixel array 100 so that each pixel receives light of the color of its associated filter. The color filter array is arranged in the specific pattern depicted in FIG. 1.

The Bayer pattern shown in FIG. 1 includes an array of repeating red, green, and blue filters. Red pixels of the array 100 are associated with red filters of the Bayer pattern, green pixels are associated with green filters, and blue pixels are associated with blue filters. In the Bayer pattern, the red, green, and blue pixels are arranged so that alternating red and blue pixels are included in rows 102 of the pixel array 100, and alternating red and green pixels are included in rows 104 of the pixel array 100. The rows 102, 104 including such alternating pixels are repeated throughout the pixel array 100. Although FIG. 1 depicts the pixel array 100 having M rows and M columns, where M is equal to “8,” in other examples, the pixel array 100 includes any number of different rows and columns. Typically, the pixel array 100 includes hundreds or thousands of such rows and columns.

The use of the Bayer pattern color filter is intended to limit the color of light that is received by each pixel of the array 100. Specifically, the Bayer pattern color filter is intended to cause each pixel to receive light of a single color that is defined by a wavelength range of the pixel's associated filter (e.g., a red filter, a green filter, or a blue filter). However, due to the angle of incident light, a pixel often receives light of a color that is different from that of the intended color defined by the pixel's associated filter. For example, light may pass through a filter element of the Bayer filter at such an angle that the light strikes a neighboring pixel, despite the fact that the neighboring pixel is not associated with the filter element and is not intended to receive light of the color defined by the filter element. This condition is known as “cross-talk.” Cross-talk causes i) green pixels positioned next to red pixels to receive an amount of red-filtered light, and ii) green pixels positioned next to blue pixels to receive an amount of blue-filtered light. As a result of such cross-talk, the pixel array 100 has two different detection levels for the color green. Thus, the signal produced by the green pixels that are included in the rows 102 with the blue pixels is not the same as the signal produced by the green pixels that are included in the rows 104 with the red pixels. Ideally, there is uniformity in the detection level for the color green across the array 100.

The different detection levels for the color green is known as “green-green imbalance” or “green imbalance” and causes unwanted effects and degraded image quality in captured images. Because the green imbalance is caused at least in part by the angle of incident light impinging on the pixel array 100, green imbalance is location dependent, such that portions of the pixel array 100 experience a green imbalance that differs from that of other portions of the pixel array 100. With smaller pixel sizes and higher sensor resolution, the undesirable effects of green imbalance increase.

According to examples described herein, green imbalance is corrected by i) determining a green imbalance value for a portion of the pixel array 100 (e.g., determining a green imbalance value for a local neighborhood of pixels included in the pixel array 100), and ii) digitally correcting an image captured by the pixel array 100 based on the determined green imbalance. The portion of the pixel array 100 comprises, in an example, a 4 pixel×4 pixel group of pixels. Examples described herein further describe the consideration of image edge pixels in correcting green imbalance. When capturing an image with the pixel array 100, different detection levels for the color green (i.e., as between green pixels on rows with blue pixels and green pixels on rows with red pixels, as described above) result from both green imbalance and image edges. Thus, both the green imbalance and the image edge pixels are considered in the green imbalance correction methods described herein. The examples described herein are applicable to CMOS imaging devices, CCD imaging devices, and other imaging devices.

FIG. 2 depicts an array of pixels 200 including a group of pixels 202, where the group of pixels 202 includes a 2×2 pixel array 204. As described above, with reference to FIG. 1, the pixel array 200 is used to capture an image, with each of the pixels in the array 200 producing a pixel output signal that indicates an intensity of light impinging on the pixel. According to examples described herein, a green imbalance value for the group of pixels 202 is determined, and the green imbalance value is used to adjust a pixel output signal of at least one green pixel within the group of pixels 202, thereby compensating for the green pixel imbalance in the group 202. Further, as described below, the compensation procedure is configured to cause minimum degradation to edges and features of the captured image.

In the example of FIG. 2, the group of pixels 202 is a subset of pixels included in the pixel array 200. The group of pixels 202 is illustrated as a 4 pixel×4 pixel group of pixels for illustrative purposes only, and in other examples, the group of pixels 202 is of various other sizes and shapes. The group of pixels 202 includes a plurality of 2×2 pixel arrays, and an example 2×2 pixel array 204 is illustrated in FIG. 2. Each 2×2 pixel array of the plurality of 2×2 pixel arrays includes (i) a first green pixel in a row that includes a red pixel (e.g., the “Gr3” pixel of the example 2×2 pixel array 204), and (ii) a second green pixel in a row that includes a blue pixel (e.g., the “Gb3” pixel of the example 2×2 pixel array 204). Throughout this disclosure and in the figures, the first green pixels are referred to as “green-red” pixels or as “Gr” pixels. The second green pixels are referred to as “green-blue” pixels or as “Gb” pixels.

In compensating for the green pixel imbalance, the group of pixels 202 limits a sampling of pixel output signals to a selection of local values, such that the compensation procedure described herein is a local neighborhood operation. The use of such a local neighborhood operation involving the group of pixels 202 is in contrast to alternative green balance procedures that consider only two pixel values (e.g., Gr3 and Gb3, or other combinations of green pixels within a single 2×2 pixel array). In the compensation procedure described herein, a green imbalance value for the group of pixels 202 is determined based on the pixel output signals produced by the green pixels of the group 202, and the green imbalance value is used to correct at least one green pixel output signal from the group of pixels 202.

Specifically, for each 2×2 pixel array of the plurality of 2×2 pixel arrays included in the group 202, a difference between the pixel output signal for the first green pixel (i.e., the green-red or Gr pixel) and the pixel output signal for the second green pixel (i.e., the green-blue or Gb pixel) is calculated. After calculating the differences for the 2×2 pixel arrays of the group 202, an average difference is determined by summing the differences and then dividing by a number of differences calculated. The green imbalance value for the group 202 is set equal to the average difference, and a pixel output signal of at least one of the green pixels in the group of pixels 202 is adjusted based on the green imbalance value. An example application of this procedure for adjusting the pixel output signal of the at least one of the green pixels in the group of pixels 202 is described below with reference to FIG. 3. The example of FIG. 3 illustrates additional aspects of the green balance procedure, including the determination of whether a calculated green imbalance is a result of image edge pixels in the captured image.

FIG. 3 is a flowchart 300 of a process for compensating for green-green imbalance in an image captured by a pixel array. At 302, an image is acquired by the pixel array, and a group of pixels (i.e., a “kernel” of pixels) is defined within the image. With reference to FIG. 2, the array of pixels 200 represents the captured image, where the captured image results from pixel output signals for each of the pixels located within the array 200. The group of pixels 202, comprising a subset of the pixels included in the array 200, is defined within the captured image as illustrated in FIG. 2.

The group of pixels 202 includes a plurality of 2×2 pixel arrays, where each of the 2×2 pixel arrays includes (i) a first green pixel in a row that includes a red pixel (i.e., a green-red pixel), and (ii) a second green pixel in a row that includes a blue (i.e., a green-blue pixel). In the example of FIG. 2, the group of pixels 202 includes four 2×2 pixel arrays, where a first 2×2 pixel array includes green pixels Gr0 and Gb0, a second 2×2 pixel array includes green pixels Gr1 and Gb1, a third 2×2 pixel array includes green pixels Gr2 and Gb2, and a fourth 2×2 pixel array includes green pixels Gr3 and Gb3. In these 2×2 pixel arrays, the first green pixels are the pixels Gr0, Gr1, Gr2, and Gr3, and the second green pixels are the pixels Gb0, Gb1, Gb2, and Gb3. Although the group of pixels 202 is defined here as including four 2×2 pixel arrays, such that none of the 2×2 pixel arrays overlap one another, various other methods for defining 2×2 pixel arrays within a group of pixels are used in other examples. In another example, for instance, the defined 2×2 pixel arrays are allowed to overlap one another, such that a single pixel is included in multiple 2×2 pixel arrays, and a 4×4 pixel array similar to the array 200 of FIG. 2 includes more than four 2×2 pixel arrays.

In the example of FIGS. 2 and 3, the green pixels of interest are the green pixels Gb0 and Gr3 located in a center area of the group 202. Thus, a green imbalance existing between the green pixels Gb0 and Gr3 is corrected according to the procedure described below, where a green imbalance value for the entire group 202 is determined and thereafter used in correcting the green imbalance between the pixels Gb0 and Gr3.

At 304, differences are calculated between pixel output signals for green pixels in each of the 2×2 pixel arrays of the group. Thus, for each of the 2×2 pixel arrays, a difference is taken between the output signals for the first green pixel and the second green pixel. With reference to the example of FIG. 2, where four 2×2 pixel arrays are included in the group 202, the following differences between pixel output signals (Delta0, Delta1, Delta2, and Delta3) are calculated:
Delta0=Gr0−Gb0  (Eqn. 1)
Delta1=Gr1−Gb1  (Eqn. 2)
Delta2=Gr2−Gb2  (Eqn. 3)
Delta3=Gr3−Gb3  (Eqn. 4)
In an example, an estimated green imbalance value (Delta_est) between the pixels of interest, Gb0 and Gr3, is determined by taking an average of the calculated differences according to an equation:
Delta_est=(Delta0+Delta1+Delta2+Delta3)/4  (Eqn. 5)

At 306, for each of the calculated differences, it is determined if the calculated difference is caused by edges in the group of pixels. With reference to the example of FIG. 2, this determination is used to improve the accuracy of the estimated green imbalance value, Delta_est. The Delta_est value may lack accuracy because the calculated differences between the pixel output signals (Delta0, Delta1, Delta2, and Delta3) may be caused largely by edges in the group of pixels 202, rather than a true green imbalance within the group 202. To determine if the calculated differences are caused by edges in the group of pixels 202, the calculated differences are compared to positive and negative threshold values:
Delta—i=Gr—i−Gb—i(i=0,1,2,3)
if Gr—i−Gb—i>Pos_threshold, then Delta—i=Pos_threshold
if Gr—i−Gb—i<Neg_threshold, then Delta—i=Neg_threshold  (Eqn. 6)

In Equation 6 above, the differences between the pixel output signals for each of the four 2×2 pixel arrays are calculated according to Delta_i=Gr_i−Gb_i, where “i” is equal to 0, 1, 2, and 3. If the calculated difference Gr_i−Gb_i exceeds the positive threshold, then the calculated difference is considered to be an outlier caused by edges within the group 202, and the calculated difference is clamped to the positive threshold. Similarly, if the calculated difference Gr_i−Gb_i is less than the negative threshold, then the calculated difference is determined to be caused by edges within the group 202, and the calculated difference is clamped to the negative threshold. The upper and lower limit values defined by the positive and negative thresholds are predetermined values that are set via a calibration procedure that is described in greater detail below with reference to FIG. 4.

The use of the comparisons to the positive and negative thresholds reflects the fact that calculated differences Gr_i−Gb_i with unusually high values that exceed the threshold values are generally caused by edges within the group of pixels, rather than an actual green imbalance. Comparing the calculated differences to the thresholds and clamping the calculated differences, if necessary, reduces the error in the calculated differences. In an example, the positive threshold value is a positive number, and the negative threshold value is a negative number. In this example, if the calculated difference Gr_i−Gb_i is a positive value, then the calculated difference is compared to the positive threshold value, and if the calculated difference Gr_i−Gb_i is a negative value, then the calculated difference is compared to the negative threshold value. In the comparison, if the calculated difference has a magnitude that is greater than that of the threshold value, then the calculated difference is adjusted by setting the calculated difference equal to the threshold value. As described in further detail below, this adjusting of the calculated differences occurs prior to determining an average of the calculated differences.

At 308, an average of the calculated differences (after comparing the calculated differences to the threshold values and clamping the calculated differences, as necessary) is calculated to determine the green imbalance between the pixels of interest. As explained above, in the example of FIG. 2, the pixels of interest are the green pixels Gb0 and Gr3 located in a center area of the group 202. Thus, to determine the green imbalance between the green pixels Gb0 and Gr3, the following equation is used:
Green_imbalance=(Delta0+Delta1+Delta2+Delta3)/4  (Eqn. 7)
As explained above, in Equation 7, the calculated differences Delta0, Delta1, Delta2, and Delta3 have been adjusted, as necessary, based on the comparisons to the positive and negative thresholds. Thus, the Green_imbalance value of Equation 7 may differ from the Delta_est value of Equation 5.

At 310, the green imbalance between the pixels of interest within the group of pixels is corrected using the calculated green imbalance value. For the pixels of interest Gr3 and Gb0 of FIG. 2, the green imbalance is corrected by adjusting the pixel output signals for the pixels according to the following equations:
Gr3_adjusted=Gr3−(Green_imbalance/2)  (Eqn. 8)
Gb0_adjusted=Gb0+(Green_imbalance/2)  (Eqn. 9)
In the example of FIGS. 2 and 3, pixel output signals of two green pixels (i.e., Gr3 and Gb0) within the group of pixels 202 are adjusted based on the green imbalance value. Thus, in this example, the pixel output signals for the first green pixel (i.e., the green-red pixel) and the second green pixel (i.e., the green-blue pixel) of a single 2×2 pixel array of the group 202 is adjusted based on the green imbalance value, and the pixel output signals for other green pixels included in the group of pixels 202 are not adjusted. Further, in the example of FIGS. 2 and 3, the single 2×2 pixel array including the adjusted pixel output signals is located at a center of the group of pixels 202, as is evident from FIG. 2. In an example, the process described above with reference to FIG. 3 is repeated by moving the group of pixels 202 throughout the captured image until all green pixels in the captured image have been adjusted as necessary. In yet another example, in moving the group of pixels 202 throughout the captured image and correcting the green pixels in this manner, green pixels are corrected more than one time.

FIG. 4 is a flowchart 400 illustrating an example method for determining positive and negative thresholds used in compensating for green pixel imbalance in an image captured by a pixel array. As described above, with reference to FIG. 3, positive and negative threshold values are used in determining whether a calculated difference between pixel output signals is a result of image edge pixels in a group of pixels. In determining the positive and negative threshold values, a calibration procedure is performed. In an example of such a calibration procedure, at 402, a flat field image is acquired using the pixel array. The flat field image is a response of the pixel array to a source of uniform radiation, and the acquisition of the flat field image is performed to remove pixel-to-pixel sensitivity variations across the pixel array.

At 404, lens shading correction is performed on the flat field image to generate a corrected image. The acquisition of the image via the pixel array leads to situations where the image exhibits significant shading across the image. In an example, the image is bright in the center of the image, and the brightness decreases towards the edge of the image. In another example, the image is darker on the left side and lighter on the right side. The shading is caused by non-uniform illumination, non-uniform camera sensitivity, or dirt and dust on a lens surface. Lens shading correction is used to remove such effects from the image after the image has been acquired. After the lens shading correction, in the corrected image, the green-red, green-blue, red, and blue channels are at an equal, flat level.

At 406, the corrected image is divided into a plurality of locations. In an example, the corrected image is divided into a matrix of 64 locations×64 locations. At 408, for each location of the plurality of locations, the green imbalance in the corrected image is measured and recorded. As explained above, green imbalance is location dependent, such that the measured green imbalance values vary over the plurality of locations in the corrected image. At 410, the positive and negative threshold values are set equal to the measured green imbalance values plus or minus a margin value. Thus, the positive and negative threshold values are location dependent and vary across the different locations of the image, resulting in an array of local green imbalance thresholds (i.e., a positive and negative threshold for each location of the plurality of locations). If the Gr_i−Gb_i difference value exceeds the threshold, then the difference value is determined to be caused by an edge in the image, and the difference value is clamped to a particular local green imbalance threshold that is dependent on the portion of the image under consideration.

FIG. 5 is a flowchart 500 illustrating an example method for compensating for green-green imbalance in an image captured by a pixel array. At 502, the image is captured using the pixel array. At 504, a group of pixels (i.e., a kernel of pixels) that is a subset of the total number of pixels in the image is defined within the image. Although the example of FIG. 5 refers to the group of pixels as an “N×N kernel,” in other examples, the group of pixels is of different sizes and shapes and does not comprise a square shape. The group of pixels includes a plurality of 2×2 pixel arrays, where each 2×2 pixel array includes a Gr pixel (i.e., a green pixel on a row with a red pixel) and a Gb pixel (i.e., a green pixel on a row with a blue pixel). Differences are calculated between the Gr and Gb pixels for each of the 2×2 pixel arrays, such that values for Delta_i=Gr_i−Gb_i are determined for each of the 2×2 pixel arrays, where i is a variable that represents an identifier for a particular 2×2 pixel array.

At 506, each Delta_i=Gr_i−Gb_i value is compared to predetermined positive and negative thresholds. If the Delta_i=Gr_i−Gb_i value exceeds either of the thresholds, the Delta_i=Gr_i−Gb_i value is clamped to the threshold that is exceeded. A calibration process for determining the positive and negative thresholds is detailed at steps 512, 514, and 516 of FIG. 5. At 512, a flat field image is obtained using the pixel array. At 514, lens shading correction is performed on the flat field image to generate a corrected image. At 516, calibration is performed to obtain an array of positive and negative threshold values. In an example, the calibration performed at 516 involves determining green imbalance values at various locations of the corrected image. The array of positive and negative threshold values are determined using the green imbalance values, thus causing a plurality of positive and negative threshold values to be defined across the pixel array. In an example, the array of positive and negative threshold values are set equal to the green imbalance values at the various locations of the corrected image plus some margin value, as described above with reference to FIG. 4.

At 508, an average of the Delta_i=Gr_i−Gb_i values is computed. In an example where i=0, 1, 2, 3, the average is computed according to an equation, Delta=(Delta0+Delta1+Delta2+Delta3)/4. The green imbalance value for the group of pixels is equal to the computed average. At 510, at least one pixel output signal for the captured image is adjusted based on the green imbalance value for the group of pixels. In an example, the pixels of interest to be adjusted within the group of pixels are located at a center area of the group of pixels. With reference to FIG. 2, the pixels of interest at the center of the group of pixels 202 are the green pixels Gr3 and Gb0. As illustrated in FIG. 5 at 510, these pixels of interest are adjusted according to the equations Gr3=Gr3−Delta/2, and Gb0=Gb0+Delta/2. In other methods, additional or fewer pixel output signals are adjusted in the group of pixels.

FIG. 6 is a block diagram of an example imaging device 600 that includes an image processing unit 604 configured to compensate for green pixel imbalance in an image captured by a pixel array. Embodiments of the example imaging device shown in FIG. 6 are included in stand-alone digital cameras and in digital cameras embedded in other electronic devices, such as cellular phones, laptop computers, and hand-held personal computing devices. As shown in FIG. 6, example imaging device 600 includes an optical sensing unit 602, the image processing unit 604, a data storage unit 606, and an input/output interface 608. In other examples, the example imaging device 600 further includes memory (e.g., as part of the data storage unit 606, the input/output interface 608, or as a standalone component) and a bus that connects one or more of the components 602, 604, 606, 608.

The components of the imaging device 600 are configured to provide image acquisition and green imbalance correction as described herein. In providing the image acquisition, the optical sensing unit 602 includes a pixel array or other components used to form a complementary metal-oxide-semiconductor (CMOS) image sensor or charge-coupled device (CCD) image sensor. In providing the green imbalance correction, the image processing unit 604 includes one or more processors for processing image pixel output signals that are generated by the optical sensing unit 602. The one or more processors of the image processing unit 604 obtain the pixel output signals and perform procedures to adjust the pixel output signals as necessary for the green imbalance correction.

The data storage unit 606 and memory are configured to hold persistent and non-persistent copies of computer code and data. The computer code includes instructions that when accessed by the image processing unit 604 result in the imaging device 600 performing green imbalance correction operations as described above. The data includes data to be acted upon by the instructions of the code, and in an example, the data includes stored pixel output signals. The processing unit 604 includes one or more single-core processors, multiple-core processors, controllers, or application-specific integrated circuits (ASICs), among other types of processing components. The memory includes random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), or dual-data rate RAM (DDRRAM), among other types of memory.

The data storage unit 606 includes integrated or peripheral storage devices, such as, but not limited to, disks and associated drives (e.g., magnetic, optical), USB storage devices and associated ports, flash memory, read-only memory (ROM), or non-volatile semiconductor devices, among others. In an example, data storage unit 606 is a storage resource physically part of the imaging device 600, and in another example, the data storage unit 606 is accessible by, but not a part of, the imaging device 600. The input/output interface 608 includes interfaces designed to communicate with peripheral hardware (e.g., remote optical imaging sensors or other remote devices). In various embodiments, imaging device 600 has more or less elements or a different architecture.

FIG. 7 is a flowchart 700 illustrating an example method of compensating for green pixel imbalance in an image captured by a pixel array. At 702, a pixel output signal is obtained for each green pixel in a group of pixels of the image captured by the pixel array. The group of pixels includes a plurality of pixel arrays, where each pixel array of the plurality of pixel arrays includes (i) a first green pixel in a row that includes a red pixel, and (ii) a second green pixel in a row that includes a blue pixel. At 704, a green imbalance value is determined for the group of pixels based on the obtained pixel output signals. At 706, for each pixel array of the plurality of pixel arrays, a difference between the pixel output signal for the first green pixel and the pixel output signal for the second green pixel is calculated. At 708, an average of the calculated differences is determined, where the green imbalance value is equal to the average. At 710, the pixel output signal of at least one of the green pixels in the group of pixels is adjusted based on the green imbalance value.

This written description uses examples to disclose the invention, including the best mode, and also to enable a person skilled in the art to make and use the invention. The patentable scope of the invention includes other examples. Additionally, the methods and systems described herein may be implemented on many different types of processing devices by program code comprising program instructions that are executable by the device processing subsystem. The software program instructions may include source code, object code, machine code, or any other stored data that is operable to cause a processing system to perform the methods and operations described herein. Other implementations may also be used, however, such as firmware or even appropriately designed hardware configured to carry out the methods and systems described herein.

The systems' and methods' data (e.g., associations, mappings, data input, data output, intermediate data results, final data results, etc.) may be stored and implemented in one or more different types of computer-implemented data stores, such as different types of storage devices and programming constructs (e.g., RAM, ROM, Flash memory, flat files, databases, programming data structures, programming variables, IF-THEN (or similar type) statement constructs, etc.). It is noted that data structures describe formats for use in organizing and storing data in databases, programs, memory, or other computer-readable media for use by a computer program.

The computer components, software modules, functions, data stores and data structures described herein may be connected directly or indirectly to each other in order to allow the flow of data needed for their operations. It is also noted that a module or processor includes but is not limited to a unit of code that performs a software operation, and can be implemented for example as a subroutine unit of code, or as a software function unit of code, or as an object (as in an object-oriented paradigm), or as an applet, or in a computer script language, or as another type of computer code. The software components and/or functionality may be located on a single computer or distributed across multiple computers depending upon the situation at hand.

Claims

1. A method comprising:

obtaining a pixel output signal for each green pixel in a group of pixels of an image captured by a pixel array device, the group of pixels including a plurality of non-overlapping pixel arrays, wherein each pixel array of the plurality of pixel arrays includes (i) a first green pixel in a row that includes a red pixel, and (ii) a second green pixel in a row that includes a blue pixel;
determining a green imbalance value for the group of pixels based on the obtained pixel output signals, the determining of the green imbalance value including: for each pixel array of the plurality of pixel arrays, calculating a difference between the pixel output signal for the first green pixel and the pixel output signal for the second green pixel, and for each of the calculated differences, determining if the calculated difference is a result of image edge pixels in the group of pixels; and
adjusting the pixel output signal of at least one of the green pixels in the group of pixels based on the green imbalance value.

2. The method of claim 1, further comprising, after the calculating of the differences and before the adjusting, for each of the calculated differences:

comparing the calculated difference to a threshold; and
if the respective difference exceeds the threshold, then setting the difference to the threshold.

3. The method of claim 2, further comprising:

determining the threshold by: acquiring a flat field image using the pixel array device; performing lens shading correction on the flat field image to generate a corrected image; dividing the corrected image into a plurality of locations; measuring a calibration green imbalance value for a location, from among the plurality of locations, that includes the group of pixels; and setting the threshold to the calibration green imbalance value plus a margin value.

4. The method of claim 1, further comprising:

setting the green imbalance value to an average of the differences.

5. The method of claim 1, wherein the adjusting includes:

raising the pixel output signal of at least one of the first green pixels by half the green imbalance value and lowering the pixel output signal of at least one of the second green pixel by half the green imbalance value.

6. The method of claim 1, further comprising, after the calculating of the differences and before the adjusting, for each of the calculated differences:

comparing the difference to an upper threshold and to a lower threshold;
if the respective difference exceeds the upper threshold, then setting the difference to the threshold; and
if the respective difference is lower than the lower threshold, then setting the difference to the lower threshold.

7. The method of claim 1, wherein the adjusting includes using the determined green imbalance value to adjust the pixel output signal of (i) only one of the first green pixels from one pixel array of the plurality of pixel arrays and (ii) only one of the second green pixels from another pixel array of the plurality of the pixel arrays.

8. The method of claim 7, wherein said only one of the first green pixels and said only one of the second green pixels are diagonally adjacent to each other.

9. The method of claim 1, wherein the plurality of pixel arrays consists of four pixel arrays, and each pixel array is a square array of four pixels.

10. The method of claim 1, wherein the method is performed by an electronic device that includes the pixel array device.

11. A non-transitory processor-readable medium storing program instructions configured to be executed by a processor to:

obtain a pixel output signal for each green pixel in a group of pixels of an image captured by a pixel array device, the group of pixels including a plurality of non-overlapping pixel arrays, wherein each pixel array of the plurality of pixel arrays includes (i) a first green pixel in a row that includes a red pixel, and (ii) a second green pixel in a row that includes a blue pixel;
determine a green imbalance value for the group of pixels based on the obtained pixel output signals, the determining of the green imbalance value including: for each pixel array of the plurality of pixel arrays, calculating a difference between the pixel output signal for the first green pixel and the pixel output signal for the second green pixel, and for each of the calculated differences, determining if the calculated difference is a result of image edge pixels in the group of pixels; and
adjust the pixel output signal of at least one of the green pixels in the group of pixels based on the green imbalance value.

12. The processor-readable medium of claim 11, wherein the program instructions are configured to be executed by the processor to, after the calculating of the differences and before the adjusting, for each of the calculated differences:

compare the difference to a threshold; and
if the respective difference exceeds the threshold, then set the difference to the threshold.

13. The processor-readable medium of claim 12, wherein the threshold is determined by:

acquiring a flat field image using the pixel array device;
performing lens shading correction on the flat field image to generate a corrected image;
dividing the corrected image into a plurality of locations;
measuring a calibration green imbalance value for a location, from among the plurality of locations, that includes the group of pixels; and
setting the threshold to the calibration green imbalance value plus a margin value.

14. The processor-readable medium of claim 11, wherein the program instructions are configured to be executed by the processor to:

set the green imbalance value to an average of the calculated differences.

15. The processor-readable medium of claim 11, wherein the adjusting includes:

raising the pixel output signal of at least one of the first green pixels by half the green imbalance value and lowering the pixel output signal of at least one of the second green pixel by half the green imbalance value.

16. The processor-readable medium of claim 11, wherein the program instructions are configured to be executed by the processor to, after the calculating of the difference and before the adjusting, for each of the calculated differences:

compare the calculated difference to an upper threshold and to a lower threshold;
if the respective difference exceeds the upper threshold, then set the difference to the threshold; and
if the respective difference is lower than the lower threshold, then set the difference to the lower threshold.

17. The processor-readable medium of claim 11, wherein the adjusting includes using the determined green imbalance value to adjust the pixel output signal of only one of the first green pixels from one pixel array of the plurality of pixel arrays and only one of the second green pixels from another pixel array of the plurality of the pixel arrays.

18. The processor-readable medium of claim 17, wherein said only one of the first green pixels and said only one of the second green pixels are diagonally adjacent each other.

19. The processor-readable medium of claim 11, wherein the plurality of pixel arrays consists of four pixel arrays, and each pixel array is a square array of four pixels.

20. An system comprising:

a pixel array device configured to capture a pixel output signal for each green pixel in a group of pixels of an image, the group of pixels including a plurality of non-overlapping pixel arrays, wherein each pixel array of the plurality of pixel arrays includes (i) a first green pixel in a row that includes a red pixel, and (ii) a second green pixel in a row that includes a blue pixel;
a processor, and
a memory storing instructions configured to be executed by the processor to process the captured pixel output signals by: determining a green imbalance value for the group of pixels based on the obtained pixel output signals, the determining of the green imbalance value including: for each pixel array of the plurality of pixel arrays, calculating a difference between the pixel output signal for the first green pixel and the pixel output signal for the second green pixel, and for each of the calculated differences, determining if the calculated difference is a result of image edge pixels in the group of pixels; and adjusting the pixel output signal of at least one of the green pixels in the group of pixels based on the green imbalance value.
Referenced Cited
U.S. Patent Documents
6714239 March 30, 2004 Guidash
7561194 July 14, 2009 Luo
7830428 November 9, 2010 Jerdev et al.
7876363 January 25, 2011 Ovsiannikov
8005297 August 23, 2011 Hung et al.
8297858 October 30, 2012 Bledsoe et al.
8471921 June 25, 2013 Li
20080231735 September 25, 2008 Sheikh
20080252759 October 16, 2008 Jerdev et al.
20140375668 December 25, 2014 Shih
Patent History
Patent number: 9270960
Type: Grant
Filed: Jul 14, 2014
Date of Patent: Feb 23, 2016
Assignee: MARVELL INTERNATIONAL LTD. (Hamilton)
Inventor: Samson Huang (Saratoga, CA)
Primary Examiner: Timothy J Henn
Application Number: 14/330,416
Classifications
Current U.S. Class: Including Noise Or Undesired Signal Reduction (348/241)
International Classification: H04N 9/73 (20060101); H04N 5/04 (20060101); H04N 9/04 (20060101);