Method and Apparatus for Expanded Dynamic Range Imaging
A method and apparatus for enabling wide intensity range in digital imaging using a small amount of memory beyond the memory required for storing the relevant image. The methods and apparatus disclose exposing few-line segments within an image using two different exposure times before continuing with other segments of the image. In alternative configurations, sensors are sampled such that neighboring sensors having the same colors are sampled after being exposed for different durations.
The present invention relates to the field of electronic imaging in general, and to an apparatus and method for expanding the dynamic exposure range in images, in particular.
BACKGROUNDWhen capturing images, illumination conditions of the scene are often of wide range. Thus, for example when capturing a room with a window, areas within the room are relatively dark, while the view out of the window is more enlightened. Providing the same exposure to all areas of the scene will result in under exposure of the room, over exposure which may lead to saturation outdoors, or both.
When using electronic imaging equipment, the exposure time determines the length of time between starting to expose one or more sensors, and the time at which the sensor was measured. If the combination of exposure time and light amount is too high, the sensor will be saturated. However, if the combination is too low, the image or part thereof will be underexposed, and therefore too dark and will exhibit poor signal to noise ratio. If intermediate exposure time is used, the dark areas will be under exposed and introduce poor signal to noise ratio, while the brighter areas will be saturated.
A prior art solution to the problem includes a method in which the sensors are exposed to two or more different exposure times. The resultant images are then combined in various ways. The problems with such solution are in two areas. First, in order to process two frames taken with different exposures, at least the image taken with the first exposure must be saved in some memory device, which is expensive and size consuming. Second, transferring the image from the sensor to the memory requires time. The transfer duration enforces time difference between the end of the first exposure and the beginning of the second exposure. During this period, the imaging apparatus or the subject may move, and the correlation between the exposures would be lower. Moreover, the reading time of the data from the sensor increases the effect of what is known as Shutter Lag. Shutter Lag is the time difference between the beginning of the exposure at the first line, to the beginning of the exposure at the last line. If there is relative movement between the object and the imaging apparatus during the shutter lag, the image would appear distorted, since the areas depicted in the first line will not correspond to the areas depicted in the last lines, which is called the “shutter lag effect”. Another method relates to exposing alternate rows of sensitive areas to long and short periods. However, such method provides poor vertical resolution, since for each area only half of the pixels are sampled.
There is thus a need in the art for a method and apparatus that will enable imaging of scenes having wide intensity range. The method and apparatus should not require significant memory addition to an imaging device, and should enable the imaging device to function ordinarily when capturing images or parts thereof having relatively small intensity ranges.
SUMMARYA method and apparatus for dynamically expanding the illumination range within a digital image.
An aspect of an embodiment of the invention relates to a method for capturing one or more images of a scene by a capturing device comprising a sensor made up of multiple sensor cells, each image comprising a number of pixels, the scene having areas with substantially varying intensity levels, the method comprising: storing in a memory unit of the capturing device one or more first intensity values received from a multiplicity of first light-sensitive sensor cells under a first exposure time; receiving one or more second intensity values received from a multiplicity of second light-sensitive sensor cells under a second exposure time; integrating the first intensity values and the second intensity values into a combined value; and assigning a value associated with the combined value to a pixel in the image, wherein the memory unit is able of storing a number of values substantially smaller than the number of light-sensitive sensor cells in the sensor. The method can further comprise a step of determining whether an image is to be analyzed using two exposure times, or a step of determining the two exposure times. Optionally, the method comprises a step of determining one or more parts of the image to be analyzed. Within the method, the multiplicity of first light-sensitive sensor cells is optionally the multiplicity of second light-sensitive sensor cells. Within the method, the multiplicity of first light-sensitive sensor cells optionally comprises two or more vertically adjacent rows of light-sensitive sensor cells, and the multiplicity of second light-sensitive sensor cell comprises the same two or more vertically adjacent rows of light-sensitive sensor cells. Within the method, the multiplicity of first light-sensitive sensor cells and the multiplicity of second light-sensitive sensor cells can comprise alternating groups of two or more adjacent sensor cells. Optionally, each of the alternating groups of two or more adjacent sensor cells comprises horizontally adjacent sensor cells, and optionally vertically adjacent sensor cells are exposed using different exposure times. Optionally, each of the alternating groups of two or more adjacent sensor cells comprises vertically adjacent sensor cells, and optionally horizontally adjacent sensor cells are exposed using different exposure times.
Optionally, the first light-sensitive sensor cells and the second light-sensitive sensor cells constitute a row of sensor cells, and the row of light-sensitive sensor cells is read prior to reading another row of light-sensitive sensor cells. Within the method, horizontally adjacent light-sensitive sensor cells having the same color are optionally exposed for different exposure times. Within the method, vertically adjacent light-sensitive sensor cells having the same color are optionally exposed for equal exposure times. Within the method, a light-sensitive sensor cell is optionally exposed for equal exposure time as an adjacent light-sensitive sensor cell. Within the method, optionally substantially all sensor cells of a predetermined color on a first row of quadruplet sensor cells are exposed for the first exposure time, and substantially all sensor cells of the predetermined color on a second row of quadruplet sensor cells, the second row adjacent to the first row, are exposed for the second exposure time. Within the method, optionally each sensor cell in the first row or in the second row is exposed for exposure time as exactly one other sensor cell having the same column and adjacent row as the first sensor cell. Within the method, optionally alternating vertical pairs of sensor cells of a predetermined color on a first row are exposed for the first exposure time and substantially all sensor cells of the predetermined color on a second row adjacent to the first row are exposed for the second exposure time. Another aspect of the disclosure relates to an image signal processor for capturing one or more images of a scene, each image comprising a number of pixels, the scene having areas with substantially varying intensity levels, the image signal processor comprising: a sensor comprising an array of light-sensitive sensor cells, at least one light-sensitive sensor cell providing a first value related to a long exposure time and at least one light-sensitive sensor cell providing a second value related to a short exposure time; a memory unit for storing the first value or the second value, the memory unit able of storing a number of values substantially smaller than the number of light-sensitive sensor cells in the sensor; and a processing unit for integrating the first value and the second value. Within the image signal processor, the memory unit is able of storing at most a number of values which is about ten percents of the number of light sensitive sensor cells in the sensor.
Non-limiting embodiments of the invention will be described with reference to the following description of exemplary embodiments, in conjunction with the figures. The figures are generally not shown to scale and any sizes are only meant to be exemplary and not necessarily limiting. In the figures, identical structures, elements or parts that appear in more than one figure are preferably labeled with a same or similar number in all the figures in which they appear, in which:
Digital imaging generally uses a sensor comprising an array of sensor cells, preferably arranged in a matrix-like structure comprising rows and columns. All sensor cells may be sensitive to one color, or each sensor cell may be sensitive to one of a predetermined number of colors, for example red, blue, green or a subset thereof. When multiple sensor cell types are available, the sensor cells are preferably arranged in a predetermined manner, such as rows made up of sensor cell quadruples arranged as a 2×2 matrix.
In exemplary embodiments of the disclosed subject matter, image sensor cells are read at varying time exposures, and the resulting values are integrated to provide a single result for each sensor cell or multiple sensor cells, indicating the amount of light in the relevant part of the image.
A first preferred embodiment, is used with imaging device that comprises rows of light sensor cells sampled consecutively, i.e. one line after the other. The first embodiment relates to reading, i.e. sampling the sensor cells in a group of rows after being exposed for a relatively long time, which is preferably sufficient for dark areas, and then sampling the same rows after being exposed for a relatively short time, which will not cause lightened areas to be saturated. The number of rows in the group is relatively small to the number of rows in the overall image, so the time difference between the two samplings of the group is short relatively to the time difference between two consecutive images. After reading a group of rows twice, the next group is read, and so on until the last group of rows in the image. Using this arrangement requires storing the sampled values of only the few rows between the two samplings, which requires significantly less storage space than storing the whole image. Thus, the relatively little memory can be integrated within the Image Signal Processor (ISP) and does not require external memory or storage, thus saving on storage space and time.
Other preferred embodiments relate to imaging devices in which the light-sensitive sensor cells within each row are arranged in quadruplets of sensor cells, one of which is responsive to blue components of light (sensor cell B), two of which are responsive to green components (G1 and G2), and the fourth responsive to red components (R). Two exemplary configurations are disclosed, in which one or more rows are sampled in an interleaved manner after being exposed for a short time or for a long time, such that the sensor cells of each color are exposed using the two exposures, and integrating the results. In another configuration, two rows of quadruplets of sensor cells are sampled together. The sampling is done such that half of each row is sampled after short exposure, and the other half is sampled after long exposure, and inversely for the other row. In this embodiment, memory is required for storing the first available results (of the short exposure or of the long exposure) of up to two row-halves, i.e. one row, which is significantly less than the full image.
The two embodiments thus enable the evaluation of sensor cell values from two exposure times, while requiring little memory compared to the memory required to store the values of a full image. This enables to incorporate the memory into the ISP and enables expanded dynamic range imaging.
Referring now to
The image capturing device, generally referenced 100 comprises a sensor made up of sensor cells arranged in rows such as rows 116, 120, 124, 128, 132, 136, 140, 144 and 148. Traditionally, the rows are sampled sequentially from top to bottom or from bottom to top. In the disclosed embodiment, the rows are divided into M-line segments, wherein M is an integer number chosen to be small relatively to the number of rows in the image. M preferably varies between about 3 and about 50 rows of sensor cells. The first M rows, referred to as 104 are exposed for a first exposure period, the sensor cell values are sampled and stored in a memory device. Then, lines 104 are exposed again, this time for a second exposure period, and the two sets of sensor cell results are integrated into a single set. Exposing first with the long exposure time and then with the short exposure time accounts for relatively short time difference between the two samplings, so that the movement of objects within the imaged scene or of the imaging device between the two samplings is relatively small. In another configuration, if a sensor cell value can be sampled without affecting or resetting the sensor cell, then different sampling order can be used. The exposure starts, then after the short exposure time elapsed, a first sampling is performed, and the exposure continues until the long exposure time is completed, at which stage the sensor cells are sampled again.
After segment 104 is sampled, the M rows of segment 108 are sampled in the same manner, and so on until segment 112 which is the last segment of image 100.
This embodiment enables the integration of two results for each sensor cell within the image, with a short time difference between the two samplings for the same sensor cell, and with a short time difference between consecutive segments, thus avoiding discontinuities.
The memory required for temporary storage of the first exposure results should only be large enough for storing the results of M rows, which is significantly less than the whole image. Thus, such memory can be integrated into the ISP without significant increase in its size or cost, and provide expanded dynamic range imaging.
Referring now to
In prior art solutions, as shown in
In the configurations shown in
Thus, in the configuration shown in
In the configuration shown in
Referring now to
One configuration, named zigzag configuration is illustrated in
Another preferred configuration, referred to as half-row-interleave is depicted in
For the two configurations, and as described in association with
Referring now to
Referring now to
After processing an image in memory unit 530, the image is stored on storage 540 or another permanent storage device, or sent to a viewing device (not shown) or to any other destination. Determining the illumination values is further detailed in association with
Memory unit 530 preferably stores also information related to different exposures for the imaged areas. Such information may also be stored in storage 540. The illumination determination is performed by processor 520, which may also use any one or more of rules 560.
Processor 520 is preferably a computing platform such as a general purpose processor or a dedicated processor executing computer instructions for carrying out the disclosed methods using the disclosed configurations or similar ones. Alternatively, processor 520 can be implemented as firmware ported for a specific processor such as digital signal processor (DSP) or microcontrollers, or can be implemented as hardware or configurable hardware such as field programmable gate array (FPGA) or application specific integrated circuit (ASIC).
Memory unit 530 is preferably any kind of Random Access Memory, such as SDRAM.
Set of rules 560 may be stored on storage 540, or connect to memory 530 from an independent storage, and accessed by processor 520. The set of rules is preferably configurable, and rules may preferably be changed, deleted, or added by a user. The rules may relate to sensor types, luminance conditions, required resolution, or the like. After the image is stored and all other data is available, too, the values for the handled image are determined by processor 520 performs the relevant mathematical or logical operations on the image. Storage 540 may be a Random Access Memory (RAM), hard disk, magnetic media, memory card, flash memory, or the like.
Referring now to
The disclosed method and apparatus utilize the three operation configurations detailed in association with
In one preferred configuration, the method and apparatus disclose the exposure of some pixels for two or more exposure times, storing the first obtained value and integrating it with the second obtained value. Other configurations relate to exposing adjacent sensor cells of the same colors to different exposure levels, and integrating the results.
It will be appreciated by a person skilled in the art, that the methods and apparatus discussed above are preferably utilized only when the need arises, i.e. when there are significant differences in illumination levels within a single image. If the illumination levels can be obtained in a single exposure for all the sensor cells, then a single exposure should be used rather than the disclosed methods and apparatus.
Structure and acts described herein are replaceable by equivalents, which perform the same function, even if the structure or acts are different as known in the art. Therefore, only the elements and limitations as used in the claims limit the scope of the invention. When used in the following claims, the terms “comprise”, “include”, “have” and their conjugates mean “including but not limited to”.
Claims
1. A method for capturing an at least one image of a scene by a capturing device comprising a sensor made up of multiple sensor cells, the image comprising a number of pixels, the scene having areas with substantially varying intensity levels, the method comprising:
- storing in a memory unit of the capturing device an at least one first intensity value received from a multiplicity of first light-sensitive sensor cells under a first exposure time;
- receiving an at least one second intensity value received from a multiplicity of second light-sensitive sensor cells under a second exposure time; integrating the first intensity value and the second intensity value into a combined value; and
- assigning a value associated with the combined value to a pixel in the at least one image,
- wherein the memory unit is able of storing a number of values substantially smaller than the number of light-sensitive sensor cells in the sensor.
2. The method of claim 1 further comprising a step of determining whether the at least one image is to be analyzed using two exposure times.
3. The method of claim 1 further comprising a step of determining the two exposure times.
4. The method of claim 1 further comprising a step of determining an at least one part of the at least one image to be analyzed.
5. The method of claim 1 wherein the multiplicity of first light-sensitive sensor cells is the multiplicity of second light-sensitive sensor cells.
6. The method of claim 1 wherein the multiplicity of first light-sensitive sensor cells comprises at least two vertically adjacent rows of light-sensitive sensor cells, and the multiplicity of second light-sensitive sensor cell comprises the at least two vertically adjacent rows of light-sensitive sensor cells.
7. The method of claim 1 wherein the multiplicity of first light-sensitive sensor cells and the multiplicity of second light-sensitive sensor cells comprise alternating groups of at least two adjacent sensor cells.
8. The method of claim 7 wherein each of the alternating groups of at least two adjacent sensor cells comprises horizontally adjacent sensor cells.
9. The method of claim 8 wherein vertically adjacent sensor cells are exposed using different exposure times.
10. The method of claim 7 wherein each of the alternating groups of at least two adjacent sensor cells comprises vertically adjacent sensor cells.
11. The method of claim 10 wherein horizontally adjacent sensor cells are exposed using different exposure times.
12. The method of claim 1 wherein the at least one first light-sensitive sensor cell and the at least one second light-sensitive sensor cell constitute an at least one row of sensor cells, and wherein the at least one row of sensor cells is read prior to reading another row of light-sensitive sensor cells.
13. The method of claim 1 wherein horizontally adjacent light-sensitive sensor cells having the same color are exposed for different exposure times.
14. The method of claim 13 wherein vertically adjacent light-sensitive sensor cells having the same color are exposed for equal exposure times.
15. The method of claim 13 wherein a light-sensitive sensor cell is exposed for equal exposure time as an adjacent light-sensitive sensor cell.
16. The method of claim 1 wherein substantially all sensor cells of a predetermined color on a first row of quadruplet sensor cells are exposed for the first exposure time, and substantially all sensor cells of the predetermined color on a second row of quadruplet sensor cells, the second row adjacent to the first row, are exposed for the second exposure time.
17. The method of claim 16 wherein each sensor cell in the first row or in the second row is exposed for exposure time as exactly one other sensor cell having the same column and adjacent row as the first sensor cell.
18. The method of claim 1 wherein alternating vertical pairs of sensor cells of a predetermined color on a first row are exposed for the first exposure time and substantially all sensor cells of the predetermined color on a second row adjacent to the first row are exposed for the second exposure time.
19. An image signal processor for capturing an at least one image of a scene, the image comprising a number of pixels, the scene having areas with substantially varying intensity levels, the image signal processor comprising:
- a sensor comprising an array of light-sensitive sensor cells, at least one light-sensitive sensor cell providing a first value related to a long exposure time and at least one light-sensitive sensor cell providing a second value related to a short exposure time;
- a memory unit for storing the first value or the second value, the memory unit able of storing a number of values substantially smaller than the number of light-sensitive sensor cells in the sensor; and
- a processing unit for integrating the first value and the second value.
20. The image signal processor of claim 19 wherein the memory unit is able of storing at most a number of values which is about ten percents of the number of light sensitive sensor cells.
Type: Application
Filed: Nov 27, 2007
Publication Date: May 28, 2009
Patent Grant number: 8542315
Inventors: Noam Sorek (Yaakov), Llia Vitsnudel (Even Yeuda), Ron Fridental (Herzeliya)
Application Number: 11/945,540
International Classification: H04N 5/76 (20060101); H04N 5/225 (20060101); H04N 5/228 (20060101);