Method and apparatus for obtaining high dynamic range images
The application provides techniques for obtaining a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor exposed to incident light from the scene for capturing an image. The image sensor has a multiplicity of light-sensing elements in an array and each light sensing element has a particular one of a plurality of sensitivity levels to incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing elements and has a response function. Each light sensing element is responsive to incident light from the scene for producing a captured image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel position array. Each one of the multiplicity of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light sensing elements.
Latest The Trustees of Columbia University in the City of New York Patents:
- Cyclopropeneimines for capture and transfer of carbon dioxide
- SYSTEMS AND METHODS FOR AUGMENTED REALITY GUIDANCE
- Cross-circulation platform for recovery, regeneration, and maintenance of extracorporeal organs
- Non-reciprocal filter
- Systems and Methods for Battery Performance Monitoring and Management Using Discrete-Time State-Space Overpotential Battery Models
This application is a divisional of U.S. patent application Ser. No. 10/959,679, filed Oct. 5, 2004 now U.S. Pat. No. 7,999,858, which is a divisional of U.S. patent application Ser. No. 09/511,469, filed Feb. 23, 2000 now U.S. Pat. No. 7,084,905, both of which are incorporated herein by reference in their entirety.
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCHThis invention was made with government support under grant number N00014-97-1-0553 awarded by the Office of Naval Research. The government has certain rights in the invention.
BACKGROUND OF INVENTIONThe present invention relates to method and apparatus for capturing an image of a scene, and, more particularly, to method and apparatus for capturing a relatively high dynamic range image using a relatively low dynamic range image sensor.
Virtually any real world scene produces a very large range of brightness values. By contrast, known image sensing devices have very limited dynamic ranges. For example, it is typical for a video image sensor to provide only 8-bits or less of grey-scale or color information. In the case of grey-scale images, 8-bits allows only 256 discrete grey-scale levels, which is not sufficient to capture the fine details of most real life scenes.
A known solution to the problem of capturing high dynamic range images with a low dynamic range image sensor is to capture multiple images of each local area of the scene while varying the exposure to light from the scene. There are three general variations of this solution: (1) temporal variation of the exposure while capturing a sequence of images, (2) using multiple image sensors having different sensitivities to light from the scene and (3) spatial variation of the exposure.
Typically, temporal exposure variation involves sequentially capturing multiple images of the scene with different exposures and then combining the multiple captured images into a single high dynamic range image. The use of temporal exposure variation is disclosed, for example, in: Japanese Patent No. 08-331461 entitled “Driving Method for Solid-State Image Pickup Device” to Hamazaki, issued December 1996; Japanese Patent No. 07-254965 entitled “Electronic Image Pickup Device” to Saito, issued February 1995; U.S. Pat. No. 5,420,635 entitled “Video Camera, Imaging Method Using Video Camera, Method of Operating Video Camera, Image Processing Apparatus and Method, and Solid-State Electronic Imaging Device” to Konishi et al., issued May 1995; U.S. Pat. No. 5,455,621 entitled “Image Method for a Wide Dynamic Range and an Imaging Device for a Wide Dynamic Range” to Morimura, issued October 1993; Japanese Patent No. 06-141229 entitled “Image Picking Up and Synthesizing Method and Image Pickup Device with High Dynamic Range” to Morimura, issued May 1994; U.S. Pat. No. 5,801,773 entitled “Image Data Processing Apparatus for Processing Combined Image Signals in Order to Extend Dynamic Range” to Ikeda, issued September 1998; U.S. Pat. No. 5,638,118 entitled “Image Device with Diverse Storage Times Used in Image Composition” to Takahashi et al. issued June 1997; U.S. Pat. No. 5,309,243 entitled “Method and Apparatus for Extending the Dynamic Range of an Electronic Imaging System” to Tsai issued May 1994; Mann and Picard, “Being ‘Undigital’ with Digital Cameras: Extending Dynamic Range by Combining Differently Exposed Pictures,” Proceedings of IST's 48th Annual Conference, pp. 422-428, May 1995; Debevec and Malik, “Recording High Dynamic Range Radiance Maps for Photographs,” Proceedings of the ACM SIGGRAPH, 1997, pp. 369-378, August 1997; and Mitsunaga and Nayar, “Radiometric Self Calibration,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR 99), pp. 374-380, June 1999. However, known temporal exposure variation techniques all have the fundamental problem in that changes in the scene may take place during the sequential image capture at different exposures. Consequently, these techniques are useful only for static scenes where the radiance from the scene remain constant and where the position and orientation of the image sensor, and its geometric characteristics remain unchanged during sequential image capture at different exposures.
Another known technique for capturing multiple images of a scene for high dynamic range imaging uses multiple optically aligned image sensors and a beam splitter for providing incident light from the same view of the scene to the multiple image sensors. The image sensors having different sensitivities so that multiple images are simultaneously captured at different exposures. The multiple images captured by the image sensors are then combined to form a single high dynamic range image. Such techniques, for example, are described in: Japanese Patent No. 08-22341 entitled “Image Sensor” to Doi et al., issued August 1986; Japanese Patent No. 07-25495 entitled “Electronic Image Pickup Device” to Saito, issued February 1995; Japanese Patent No. 07-254966 entitled “Electronic Image Pickup Device” to Saito, issued October 1995; Japanese Patent No. 08-340486 entitled “Electronic Image Pickup Device to Saito, issued December 1996; Japanese Patent No. 10-069011 entitled “Image Pickup Device” to Kimura, issued March 1998; and U.S. Pat. No. 5,801,773 entitled “Image Data Processing Apparatus for Processing Combined Image Signals in Order to Extend Dynamic Range” To Ikeda, issued September 1998. The multiple image sensor techniques have the drawback in that they require high precision optics and a plurality of image sensors so that it is difficult to build compact and inexpensive imaging systems using these techniques.
Known spatial exposure variation techniques for high dynamic range imaging typically use a special image sensor having more than one (typically two) light-sensing elements having different sensitivities for each pixel. The light-sensing elements for each pixel are simultaneously exposed and their respective output signals are combined to derive a brightness value for the pixel. Such techniques are described, for example, in: U.S. Pat. No. 5,789,737 entitled “High Dynamic Range Segmented Pixel Sensor Array” to Street et al., issued August 1998; U.S. Pat. No. 4,623,928 entitled “High Dynamic Range CCD Detector/Imager” to Handy, issued November 1986; U.S. Pat. No. 4,873,561 entitled “High Dynamic Range Charge Coupled Device” to Wen, issued October 1989; Japanese Patent No. 08-331461 entitled “Driving Method for Solid-State Image Pickup Device” to Hamazaki, issued December 1996; Japanese Patent No. 59-217358 entitled “Charge Coupling Image Pickup Device” to Murakoshi, issued December 1994; and U.S. Pat. No. 5,420,635 entitled “Video Camera Imaging Method Using Video Camera, Method of Operating Video Camera, Image Processing Apparatus and Method, and Solid-State Electronic Imaging Device” to Konishi et al., issued May 1995. However, known spatial exposure variation techniques for high dynamic range imaging have the drawback of providing lower spatial resolution (typically a factor of two lower in the vertical direction) than the resolution of ordinary image sensors having the same number of light-sensing elements.
Accordingly, there exists a need for a method and apparatus for capturing a relatively high dynamic range images using a relatively low dynamic range image sensor, which overcomes the problems of the prior art as discussed above.
SUMMARY OF THE INVENTIONIn accordance with one aspect of the present invention, there is provided a method for capturing a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor exposed to incident light from the scene. The image sensor has a multiplicity of light-sensing elements in a linear or two-dimensional array, and each light-sensing element has a respective one of a plurality of sensitivity levels to incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing elements. Each of the light-sensing elements in the array in response to incident light from the scene produces a captured image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel position array, which may be a linear array having regularly spaced pixel positions in a row or a rectangular array having pixel positions at respective intersections of regularly spaced orthogonal pixel rows and pixel columns. In this manner each pixel position corresponds to a respective one of the plurality of sensitivity levels. The method comprises the step of estimating respective off-grid brightness values at a multiplicity of off-grid positions from the captured image brightness values at the multiplicity of pixel positions of the pixel position array, the off-grid positions being located in a regular array at respective interstices of the pixel position array. The method further comprises the step of deriving pixel brightness values of a relatively high dynamic range image of the scene from the estimated off-grid brightness values at the multiplicity of off-grid positions. Where the pixel position array is a linear array in which the brightness value at each pixel position is captured by a corresponding light-sensing element of a linear image sensor, the off-grid positions are in a regular linear array, and the relatively high dynamic range image is a linear image represented by a regular linear array of pixel brightness values. Where the pixel position array is defined by respective intersections of regularly spaced orthogonal pixel rows and pixel columns, the brightness values of the pixel positions in the array may be captured by corresponding light-sensing elements of a two-dimensional image sensor or may be a concatenation of the brightness values at pixel positions of linear pixel position arrays representing line images of a scanned scene successively captured by a linear image sensor.
In accordance with an exemplary embodiment of the method of the present invention, the off-grid positions are in a regular linear or rectangular array and the step of deriving pixel brightness values of a relatively high dynamic range image comprises using the estimated off-grid brightness values as the pixel brightness values of the relatively high dynamic range image.
According to another exemplary embodiment of the method of the present invention, the step of deriving pixel brightness values of the relatively high dynamic range image comprises resampling the estimated off-grid brightness values at the multiplicity of off-grid positions to derive respective resampled brightness values at the multiplicity of pixel positions of the pixel position array to thereby obtain the pixel brightness values of the relatively high dynamic range image.
In accordance with still another exemplary embodiment of the method of present invention, the step of estimating respective off-grid brightness values at the multiplicity of off-grid positions from the captured image brightness values at the multiplicity of pixel positions comprises the steps of estimating a sensitivity level off-grid brightness value for each individual one of the plurality of sensitivity levels at each one of the off-grid positions using only the captured image brightness values at pixel positions corresponding to the individual one of the sensitivity levels. The sensitivity level off-grid brightness value for each individual one of the sensitivity levels at each one of the off-grid positions is multiplied by a predetermined weighting factor for the individual one of the sensitivity levels to derive respective weighted sensitivity level off-grid brightness values for the different ones of the sensitivity levels at each one of the off-grid positions. The respective weighted sensitivity level off-grid brightness values for the different ones of the sensitivity levels at each one of the off-grid positions are added together to derive a weighted sum sensitivity level brightness value at each one of the off-grid positions. The weighted sum sensitivity level brightness value at each one of the off-grid positions is compensated by the inverse of a combined response function of the light-sensing elements to derive the respective estimated off-grid brightness values at the off-grid positions. The combined response function of the light-sensing elements is the sum of weighted response functions of light-sensing elements having different ones of the sensitivity levels, each weighted response function being a response function of a light-sensing element having a respective one of the sensitivity levels multiplied by a predetermined weighting factor for the respective one of the sensitivity levels.
According to yet another exemplary embodiment of the method of the present invention, the step of estimating respective off-grid brightness values at the multiplicity of off-grid positions from the captured image brightness values at the multiplicity of pixel positions comprises estimating an off-grid brightness value at each individual one of the off-grid positions by first compensating the captured image brightness value at each individual one of a predetermined group of pixel positions that are nearest neighbors to the individual one of the off grid positions by the inverse of the response function of a light-sensing element having a sensitivity level corresponding to the individual one of the nearest neighbor pixel positions. The compensated captured image brightness values at the nearest neighbor pixel position are then each compared with a predetermined low threshold value and a predetermined high threshold value, and a sum is taken of the compensated nearest neighbor pixel position brightness values that are greater than the predetermined low threshold value and less than the predetermined high threshold value. The sum is then divided by the number of compensated nearest neighbor pixel position brightness values included in the sum to derive an estimated off-grid brightness value for the individual one of the off-grid positions.
According to a further exemplary embodiment of the method of the present invention, interpolated on-grid brightness values at respective pixel positions of the pixel position array are derived directly from the captured image brightness values using an on-grid interpolation filter. The interpolated on-grid brightness value at each individual one of the pixel positions is derived by computing the product of the captured image brightness values at a predetermined group of pixel positions that are nearest neighbors to the individual one of the pixel positions and an on-grid interpolation filter kernel, the predetermined group of pixel positions having the same dimensions as the filter kernel. The interpolated on-grid brightness values are then each compensated by the inverse of a combined response function of light-sensing elements having different sensitivity levels to derive the pixel brightness values of the relatively high dynamic range output image.
In accordance with another aspect of the present invention, there is provided a system for capturing a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor adapted to be exposed to incident light from the scene. The system comprises an image sensor having a multiplicity of light-sensing elements in a linear or two-dimensional array. Each light-sensing element in the array having a respective one of a plurality of sensitivity levels to incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing elements. Each light-sensing element being responsive to incident light from the scene for producing a captured image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel position array, which may be a linear array having regularly spaced pixel positions in a row or a rectangular array having pixel positions located at respective intersections of regularly spaced orthogonal pixel rows and pixel columns. In this manner, each pixel position corresponds to a respective one of the plurality of sensitivity levels of the light-sensing elements. The system further comprises a captured image memory for storing the respective captured image brightness values at the multiplicity of pixel positions produced by the light-sensing elements, and an off-grid estimator for deriving from the captured image brightness values in the captured image memory respective estimated off-grid brightness values at a multiplicity of off-grid positions located in a regular array at respective interstices of the pixel position array. The system further includes an output image generator for deriving pixel brightness values of a relatively high-dynamic range output image from the estimated off-grid brightness values, and an output image memory for storing the pixel brightness values of the output image. Where the pixel position array is a linear array and the captured image memory stores brightness values captured by corresponding light-sensing elements of a linear image sensor, the off-grid positions are in a regular linear array and the output image is a linear image represented by a regular array of pixel brightness values. Where the pixel position array is a rectangular array defined by respective intersections of regularly spaced orthogonal pixel rows and pixel columns, the captured image memory may store brightness values captured by corresponding light-sensing elements of a two-dimensional image sensor or a concatenation of linear pixel position array brightness values of line images of a scanned scene successively captured by a linear image sensor. In the latter instance, the off-grid positions are in a regular two-dimensional array and the output image is a two-dimensional image represented by a regular two-dimensional array of pixel brightness values.
According to an exemplary embodiment of the system of the present invention, the output image generator provides the estimated off-grid brightness values derived by the off-grid estimator to the output image memory as the pixel brightness values of the output image.
According to another exemplary embodiment of the system of the present invention, the system includes an off-grid brightness value memory for storing the respective estimated off-grid brightness values derived by the off-grid estimator at the off-grid positions, and an on-grid resampler for deriving from the estimated off-grid brightness values in the off-grid brightness value memory respective resampled brightness values at the pixel positions of the pixel position array. The resampled brightness values are then provided to the output image memory for storage therein as the pixel brightness values of the output image.
According to still another exemplary embodiment of the system of the present invention, the off-grid estimator include an off-grid position generator for providing the multiplicity of off-grid positions and a sensitivity pattern memory for storing the respective sensitivity levels corresponding to the pixel positions. The off-grid estimator also includes a plurality of sensitivity level off-grid brightness value estimators, each corresponding to a respective one of the plurality of sensitivity levels of the light-sensing elements, for receiving off-grid positions from the off-grid position generator and sensitivity level data from the sensitivity pattern memory, and for deriving from the captured image brightness values in the captured image memory a plurality of sensitivity level off-grid brightness values each corresponding to a different one of the plurality of sensitivity levels at each one of the off-grid positions received from the off-grid position generator. The off-grid estimator also includes an accumulator for receiving the sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimators at each one of the off-grid positions. The accumulator comprises a weighting factor memory for storing respective ones of a plurality of predetermined weighting factors each corresponding to a respective one of the plurality of sensitivity levels, and a plurality of multipliers each for computing the product of the sensitivity level off-grid brightness value corresponding to a respective one of the plurality of sensitivity levels and the weighting factor corresponding to the same sensitivity level to provide a respective weighted sensitivity level off-grid brightness value corresponding to the respective one of the sensitivity levels at each one of the off-grid positions received from the off-grid position generator. The accumulator further includes an adder for summing the weighted sensitivity level off-grid brightness values corresponding to respective ones of the sensitivity levels at each one of the received off-grid positions computed by the plurality of multipliers to derive a respective weighted sum sensitivity level off-grid brightness value at each one of the received off-grid positions. The off-grid estimator further comprises a response function compensator for compensating each one of the weighted sum sensitivity level off-grid brightness values by the inverse of a combined response function of light-sensing elements to derive the respective estimated off-grid brightness values at the off-grid positions. The combined response function of the light-sensing elements is the sum of weighted response functions of light-sensing elements having different ones of the sensitivity levels, each one of the weighted response functions being a response function of a light-sensing element having a respective one of the plurality of sensitivity levels multiplied by a predetermined weighting factor corresponding to the respective one of the plurality of sensitivity levels.
In yet another exemplary embodiment of the system of the present invention, the off-grid estimator comprises a pixel position generator for providing the multiplicity of pixel positions, a sensitivity pattern memory for storing respective sensitivity levels corresponding to the multiplicity of pixel positions, and a response function compensator coupled to receive pixel positions from the pixel position generator and corresponding sensitivity levels from the sensitivity pattern memory. The response function compensator also receives captured image brightness values from the captured image memory at the pixel positions received from the pixel position generator and compensates the received captured image brightness values by the inverse of the response functions of light-sensing elements having sensitivity levels corresponding to the received pixel positions to provide respective compensated captured image brightness values at the pixel positions received from the pixel position generator. The off-grid estimator also includes a compensated on-grid brightness value memory for storing respective compensated captured image brightness values at respective pixel positions, an off-grid position generator for providing the multiplicity of off-grid positions and a pseudoinverse estimator. The pseudoinverse estimator receives off-grid positions from the off-grid position generator and derives from the compensated captured image brightness values in the compensated on-grid brightness value memory the respective estimated off-grid brightness values at the received off-grid positions.
In a further exemplary embodiment of the system of the present invention where the system includes an off-grid brightness value memory for storing the respective estimated off-grid brightness values derived by the off-grid estimator at the multiplicity of off-grid positions and an on-grid resample for deriving from the estimated off-grid brightness values in the off-grid brightness value memory respective resampled brightness values at the multiplicity of pixel positions of the pixel position array, the on-grid resampler comprises a pixel position generator for providing the multiplicity of pixel positions, an interpolator for receiving pixel positions from the pixel position generator and deriving a respective resampled brightness value for each one of the received pixel positions by computing the product of the estimated off-grid brightness values at a predetermined number of off-grid positions that are nearest neighbors to the received pixel positions and an interpolation kernel having the same dimensions as the predetermined number of nearest neighbor off-grid positions.
In a still further exemplary embodiment of the system of the present invention, there is provided a system for capturing a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor adapted to be exposed to incident light from the scene. The system comprising an image sensor having a multiplicity of light-sensing elements in a linear or two-dimensional array, each light-sensing element having a respective one of a plurality of sensitivity levels in accordance with a predetermined sensitivity pattern for the light-sensing element array, and being responsive to incident light from the scene for producing a captured image brightness value at a corresponding one of a multiplicity of pixel positions, where the pixel position array is rectangular, the captured image brightness values may be produced by corresponding light-sensing elements of a two-dimensional image sensor, or may be a concatenation of linear brightness value arrays of line images of a scanned scene successively captured by image sensor. The system further comprises a captured image memory for storing the captured image brightness values produced by the light-sensing elements at respective pixel positions of a pixel position array, which may be a linear array having regularly spaced pixel positions in a row or a rectangular array having pixel positions located at respective intersections of regularly spaced orthogonal pixel rows and pixel columns. The system also comprises a pixel position generator for providing the pixel positions of the pixel position array and an on-grid interpolation filter receiving pixel positions from the pixel position generator for deriving from the captured image brightness values in the captured image memory interpolated on-grid brightness values at the pixel positions received from the pixel position generator. The filter computes the product of respective captured image brightness values at a predetermined number of nearest neighbor pixel positions to each pixel position received from the pixel position generator and an on-grid interpolation filter kernel to derive an interpolated on-grid brightness value at each pixel position received from the pixel position generator, the dimensions of the on-grid interpolation filter being the same as the dimensions of the predetermined number of nearest neighbor pixel positions. Also included in the system is a response function compensator for compensating the interpolated on-grid brightness value at each one of the received pixel positions by the inverse of a combined response function of the light-sensing elements to derive a compensated interpolated on-grid brightness value at the received pixel position, and an output image memory for storing the compensated interpolated on-grid brightness values at the respective pixel positions of the pixel position array as the pixel brightness values of a high dynamic range output image. The combined response function of the light-sensing elements is the sum of weighted response functions of light-sensing elements having different ones of the plurality of sensitivity levels, each one of the weighted response functions being the response function of a light-sensing element having a respective one of the plurality of sensitivity levels multiplied by a predetermined weighting factor for the respective one of the plurality of sensitivity levels.
For a more complete understanding of the nature and benefits of the present invention, reference should be made to the following detailed description taken in conjunction with the accompanying drawings, in which:
Throughout the figures of the drawings the same reference numerals and characters are used to designate common features, components, parts, processes and steps.
DETAILED DESCRIPTIONReferring to
Turning to
Referring to
The image sensing part 1 includes a lens system 6, an aperture 7, a charge coupled device (CCD) image sensor 9 and a preamplifier 10. Incident light 5 from a scene passes through the lens system 6 and the aperture 7 before reaching the CCD image sensor 9. The CCD image sensor 9 has a predetermined spatially varying sensitivity pattern according to which the sensitivity level of each of the light-sensing elements (not shown) of the image sensor 9 is determined. Different exemplary ways of providing a spatially varying sensitivity pattern for the CCD image sensor 9 are described hereinbelow. The preamplifier 10 consists of a group of several analog circuits which are typically used in video cameras, such as an automatic gain control circuit for adjusting the amplitude of the CCD image sensor output and a correlated double sampling circuit for reducing noise.
The image sensing part 2 has an analog-to-digital (A/D) converter 11, a frame memory 12, a digital-to-analog (DIA) converter 13, a read-only memory (ROM) 14, a random-access memory (RAM) 15 and a processor unit 16. The A/D converter 11 digitizes the image signal from the image sensing part 1, and the digitized image data (captured image brightness values at respective pixel positions) are stored in the frame memory 12. Computations in the image processing part 2 are performed by the processing unit 16, which has access to the data in the frame memory 12, the ROM 14 and the RAM 15. Predefined data and programs to be used by the processing unit 16 to perform its computations are stored in the ROM 14. A portion of the RAM 15 is used for temporary storage of data during computations by the processing unit 16. Computed resampled brightness values at respective pixel positions, which are stored in the frame memory 12, are converted to an analog image signal by the D/A converter 13 and provided to the video output part 4.
The video output part 4 comprises a video encoder 17 and an output unit 18. The analog image signal from the image processing part 2 is encoded to a video signal by the video encoder 17, and the output unit 18 outputs the video output signal 19.
Image Sensing Part
The spatially varying sensitivity pattern of the CCD image sensor 9 may be implemented by using a fixed pattern mask or by on-chip techniques. Turning now to
Referring to
Turning to
Referring to
Turning to
Turning to
Referring to
Referring to
Turning to
In addition to the periodic application of a transfer trigger pulse to transfer trigger input 137, the photogenerated charge stored in each photodiode is periodically discharged by the periodic application of a reset pulse signal to a reset input 133. The application of each reset pulse voltage signal appropriately biases the junction between the p-well and a heavily doped n-type charge sink region 120 to cause substantially all of the photogenerated charge stored in the potential well of each photodiode of the CCD image sensor to be transferred to the charge sink region 120, as indicated by the arrow 139.
Turning now to
Transfer trigger pulses 198, which have the same width as transfer trigger pulses 195, are applied to the transfer gate electrode associated with photodiode PD3 at half the frequency as that of transfer trigger pulses 195. The phase of the transfer trigger pulses 198 is such that each transfer trigger pulse 198 immediately precedes the next reset pulse 194. The application of each transfer trigger pulse 198 causes an amount of photogenerated charge stored in photodiode PD3 represented by the corresponding blackened region 260 to be transferred to a respective stage of a corresponding vertical shift register, and the transferred charge is accumulated therein for the duration of the exposure pulse 193. Because transfer trigger pulses 198 occur at half the frequency of the transfer trigger pulses 195, the photogenerated charge accumulated in the vertical shift register stage receiving photogenerated charge from photodiode PD3 over the exposure duration will be half of that which is accumulated in the vertical shift register stage that receives photogenerated charge from photodiode PD4 over the same exposure duration. Operated in this manner, the effective sensitivity level of photodiode PD3 will be half that of photodiode PD4.
Transfer trigger pulses 261, which have the same width as the transfer trigger pulses 198 and 195, are applied to the transfer gate electrode associated with photodiode PD2. The phase of the transfer trigger pulses 261 is such that each transfer trigger pulse 261 immediately precedes the next reset pulse 194. The application of each transfer trigger pulse 261 causes the transfer of an amount of photogenerated charge stored in photodiode PD2 represented by the corresponding blackened region 263 to a respective stage of a corresponding vertical shift register, and the transferred charge is accumulated therein for the duration of the exposure pulse 193. Because the frequency of the transfer trigger pulses 261 is half that of the transfer trigger pulses 198 and one-quarter of that of the transfer trigger pulses 195, the total amount of photogenerated charge accumulated in the shift register stage that receives stored photogenerated charge from photodiode PD2 over the duration of the exposure pulse 193 will be half of that which is accumulated in the vertical shift register stage that receives stored photogenerated charge from photodiode PD3 and one-quarter of that which is accumulated in the shift register stage that receives stored photogenerated charge from photodiode PD4 over the same exposure duration. Consequently, the effective sensitivity level of photodiode PD2 is half of the effective sensitivity level of photodiode PD3 and one-quarter of the effective sensitivity level of photodiode PD4.
Transfer trigger pulses 264, which have the same width as transfer trigger pulses 261, 198 and 195, are applied to the transfer gate electrode associated with photodiode PD1 at half the frequency of that of transfer trigger pulses 261. The phase of the transfer trigger pulses 264 is such that each transfer trigger pulse 264 immediately precedes the next reset pulse 194. The application of each transfer trigger pulse 264 causes an amount of photogenerated charge stored in photodiode PD1 represented by the corresponding blackened region 266 to be transferred to a respective stage of a corresponding vertical shift register, and the transferred charge is accumulated therein for the duration of the exposure pulse 193. Because the frequency of the transfer trigger pulses 264 is half of that of transfer trigger pulses 261, one-quarter of that of transfer trigger pulses 198 and one-eighth of that of transfer trigger pulses 195, the total photogenerated charge accumulated in the vertical shift register stage receiving stored photogenerated charge from photodiode PD1 over the duration of the exposure pulse 193 is one-half of that which is accumulated in the shift register stage receiving stored photogenerated charge from photodiode PD2, one-quarter of that which is accumulated in the shift register stage receiving stored photogenerated charge from photodiode PD3 and one-eighth of that which is accumulated in the vertical shift register stage which receives stored photogenerated charge from photodiode PD4 over the same exposure duration. Consequently, the effective sensitivity level of photodiode PD1 is one-half of that of photodiode PD2, one-quarter of that photodiode PD3 and one-eighth of that photodiode PD4. Accordingly, different effective sensitivity levels may be obtained for the photodiodes of the CCD image sensor by applying respective transfer trigger pulses to their associated transfer gate electrodes at different frequencies in the manner described. Because photogenerated charge is accumulated in small amounts in respective vertical shift register stages upon the application of each transfer trigger pulse and the transfer trigger pulses are uniformly distributed over the exposure period, movement in the scene will be integrated in the same manner in the vertical shift register stages accumulating stored photogenerated charge from respective photodiodes. Thus, inconsistency of motion blur in the output images is suppressed by using the foregoing technique for obtaining a predetermined spatially varying sensitivity pattern for the CCD image sensor.
Image Processing Part
Returning to the dataflow diagram of
Referring to
Each one of the intermediate rows 269 is parallel to the pixel rows 267 and extends medially between a respective adjacent pair of the pixel rows 267. Each one of the intermediate columns 270 is parallel to the pixel columns 268 and extends medially between a respective adjacent pair of the pixel columns 267. For convenient reference of the pixel positions and off-grid positions, the bottom left corner of the captured image is defined as the origin of a coordinate system (0,0) with the pixel position nearest the origin having coordinates (0.5, 0.5) and the off-grid position nearest the origin having coordinates (1,1). The bottom right corner of the captured image is defined as having coordinates (xSize, 0), where xSize is the dimension of the captured image in the direction (x direction) of the pixel rows 267, and the upper left corner of the image is defined as having coordinates (0,ySize), where ySize is the dimension of the captured image in the direction (y direction) of the pixel columns 268.
Turning to
Referring again to
Turning back to
Referring to
Turning to
Turning now to
Referring to
s(E)=W1p1(E)+W2p2(E)+W3P3(E)+W4p4(E). (1)
The combining represented by equation (1) brings about a high dynamic range for sensing the radiation intensity E because the response functions at four sensitivity levels are combined. For example, p1(E) has sensitivity to a bright range of E and p4(E) has sensitivity to a dark range of E, while p2(E) and p3(E) cover the mid range of E. The combining of equation (1) also results in an increase in the number of grey scale levels. If each of the response functions p1(E), p2(E), p3(E) and p4(E) has a grey scale of 256 levels, s(E) has a grey scale equivalent to 1021 levels.
Turning to
It will be apparent to those skilled in the art that the nearest neighbor sampling function carried out by the resamplers 28, 29, 30 and 31 of
Turning now to
With reference to the block diagram of
Turning back to
Referring to
Referring to
Turning to
Referring to
While the interpolator 37 of the diagram in
In the embodiments of the present invention described thus far, the light-sensing elements of the image sensor have four different sensitivity levels. It will be understood that the present invention may be readily applied to embodiments where the light-sensing elements of the image sensor have a different number of sensitivity levels with only minor alterations to the four sensitivity level embodiments described hereinabove. For example, the light-sensing elements of the image sensor may have only two sensitivity levels, in which case the modifications to the embodiments are now described.
In the image sensing part where the spatially varying sensitivity pattern is defined by a mask, the mask will have cells with two different transparencies, namely a dark cell and a bright cell. For example, the mask may have a checkerboard pattern in which adjacent cells in a row and adjacent cells in a column will have different ones of the two transparencies. An example of such a mask 3500 is depicted in
In the image processing part, the off-grid position generators 32 in the diagram of
Referring to
Turning to
Referring to
The embodiment of the off-grid estimator represented by the data flow diagram 2400 in
Referring to
The data flow diagram of the on-grid resampler for the two sensitivity level case is the same as that for the four sensitivity level cases, as shown in
Turning to
Referring back to the data flow diagram 200 of
Turning to
Turning to
Referring now to
In accordance with the present invention, off-grid brightness values B(x,y) are obtained by applying an off-grid estimating function ƒoff(*) to the captured image brightness values Ic(x,y), i.e.:
B(x,y)=ƒoff(Ic(x,y)). (2)
In the embodiment of the off-grid estimator represented by
B(x,y)=s−1({tilde over (s)}(Ic(x,y))) (3)
where {tilde over (s)}(*) is a function for obtaining sensitivity level dependent off-grid brightness values by using a weighted sum of response functions of light-sensing elements having different sensitivity levels such as that computed using equation (1).
To obtain the output image, an on-grid resampling function ƒon(*) is used to compute brightness values at pixel positions by a conventional resampling function r(*). Therefore, the output image can be expressed as
It may be assumed that the same continuous function can be constructed from different resampled data sequences if the original continuous function is sufficiently smooth. Based on this assumption, is equivalent to Ĩo(x,y) is equivalent to Io(x,y), where
Ĩo(x,y)=s−1(r({tilde over (s)}(Ic(x,y))) (5)
Although Ĩo and Io are not equivalent when {tilde over (s)}(Ic(x,y)) is not smooth, Ĩo(x,y) can nevertheless be used as an approximation of Io(x,y).
The functions {tilde over (s)}(*) and r(*) may be combined into a single filter function because both functions are linear and invariant to phase shift. Therefore the embodiment represented by the diagram of
Referring to
The present invention may be applied to linear image sensors having linear arrays of light-sensing elements that capture line images. Such linear image sensors, including CCD linear image sensors, are available with a greater number of light-sensing elements along a given direction than two-dimensional image sensors. Accordingly, linear image sensors may be used to provide two-dimensional images of greater resolution in applications where scanning of a scene with the linear image sensor is feasible and sequentially captured line images of successive regularly spaced linear regions of the scene are concatenated. Examples of the ways that a linear image sensor may be used to scan a scene and sequentially capture line images that may be concatenated to for a two-dimension image of the scene are illustrated in
Turning to
In accordance with the present invention, the linear image sensor 276 has a spatially varying sensitivity pattern. Referring to
Each light-sensing element of the linear image sensor in response to incident light from the scene produces a captured image brightness value at a corresponding one of a multiplicity of regularly spaced pixel positions of a linear pixel position array, thereby capturing a line image. When successive line images captured by the linear image sensor are concatenated, there is obtained a two-dimensional array of pixel positions located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns orthogonal to the pixel rows, where each pixel position has a respective brightness value produced by a light-sensing element that corresponds to the same pixel position in each one of the concatenated line images, and corresponds to a respective one of the two sensitivity levels of the light-sensing elements in accordance with a two-dimensional sensitivity pattern as defined illustratively by the concatenated exposure mask of
Referring back to
The data flow diagram of
Referring to
Referring to
As an alternative to processing concatenated captured line images, each line image may be separately processed to derive a high dynamic range line image by estimating the brightness values at off-grid positions, which for a linear pixel position array are located between respective adjacent pairs of pixel positions. The processed line images may then be concatenated to obtain a high dynamic range two-dimensional image. Separate processing of each line image is less computationally intensive in comparison to the processing of a two-dimensional image, and therefore can be done in real time as the line images are captured. Turning to
B(x′)=W1Ic(x′−0.5)+W2Ic(x′+0.5), (6)
where W1 and W2 are weighting factors assigned to the two different sensitivity levels. The resulting off-grid brightness values represent a one-dimensional image with high dynamic range. The computed off-grid brightness values are each compensated by the inverse of a combined response function of the light-sensing elements. The combined response function may be expressed as
s(E)=W1P1(E)+W2P2(E), (7)
where P1(E) is the radiometric response function of a light-sensing element of the linear image sensor having one of the two sensitivity levels, P2(E) is the radiometric response function of a light-sensing element of the linear image sensor having the other of the two sensitivity levels, and W1 and W2 are the weighting factors assigned to the two different sensitivity levels. Compensation of the estimated off-grid brightness values by the inverse of the combined response function of the light-sensing elements using a mapper and a lookup table memory as in the two-dimensional case remaps the off-grid brightness values to ones that are linearly related to scene radiance.
Referring to
Turning back to
The respective estimated off-grid brightness values at the off-grid positions may be used directly as the pixel brightness values of an output image and stored in the output image memory 36 in
Referring back to
Another embodiment of the present invention for processing captured line images to derive on-grid brightness values of a line output image is represented by the data flow diagram 4700 of
Referring to
Turning to
While the present invention has been particularly described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various modifications and alterations may be made without departing of the spirit and the scope of the invention. For example, the image sensor need not be a charge coupled device, but may be any image sensor having an array of light-sensing elements, such as a CMOS image sensor. Accordingly, the disclosed embodiments of the invention are considered merely illustrative, and the invention is limited in scope only as specified in the appended claims.
Claims
1. A system for obtaining a relatively high dynamic range image of a scene using a relatively low dynamic range image sensor adapted to be exposed to incident light from the scene for capturing an image thereof comprising:
- an image sensor having a multiplicity of light-sensing elements in an array, each light-sensing element having a particular one of a plurality of sensitivity levels to incident light in accordance with a predetermined sensitivity pattern for the array of light-sensing elements and a respective response function, the predetermined sensitivity pattern of the array of light-sensing elements having at least two different predetermined sensitivity levels and corresponding to a repetitive pattern of groups of pixel positions, each one of the pixel positions in each group corresponding to a different one of the at least two predetermined sensitivity levels in a common predetermined positional order, each light-sensing element being responsive to incident light from the scene for producing a captured image brightness value at a corresponding one of a multiplicity of pixel positions of a pixel position array, whereby each one of the multiplicity of pixel positions corresponds to a particular one of the plurality of sensitivity levels of the light-sensing elements;
- a captured image memory for storing the captured image brightness values produced by the light-sensing elements at corresponding ones of the multiplicity of pixel positions of the pixel position array;
- an off-grid estimator for deriving from the captured image brightness values in the captured image memory respective estimated off-grid brightness values at a multiplicity of off-grid positions located at respective interstices of the pixel position array so as to form a regular off-grid position array; and
- an output image generator for deriving pixel brightness values of an output image from the estimated off-grid brightness values, comprising: an off-grid brightness value memory for storing the estimated off-grid brightness values derived by the off-grid estimator; an on-grid resampler for deriving from the estimated off-grid brightness values in the off-grid brightness value memory respective resampled on-grid brightness values at the pixel positions of the pixel position array; and an output image memory for storing the respective resampled on-grid brightness values derived by the on-grid resampler as the pixel brightness values of the output image.
2. The system of claim 1, wherein the array of light-sensing elements of the image sensor is a linear array for capturing a line image and the pixel position array is a linear array having the multiplicity of pixel positions located at respective regularly spaced positions in a pixel row so as to form a linear captured image brightness value array, the captured image memory for storing the captured image brightness values produced by the light-sensing elements.
3. The system of claim 2, wherein the off-grid positions of the off-grid position array are located between respective adjacent pairs of pixel positions of the linear pixel position array.
4. The system of claim 3, wherein the predetermined sensitivity pattern has a predetermined first sensitivity level and a predetermined second sensitivity level, and adjacent pixel positions of the linear pixel position array correspond to different ones of the predetermined first sensitivity level and the predetermined second sensitivity level.
5. The system of claim 1, wherein the array of light-sensing elements of the image sensor is a linear array and the pixel position array is a two-dimensional array having pixel positions at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the linear array of light-sensing elements of the image sensor is adapted to be sequentially exposed to successive regularly spaced linear regions of the scene in multiple exposures, each one of the exposures producing a respective linear captured image brightness value array, the respective linear captured image brightness value arrays produced by the multiple exposures being concatenated to form a two-dimensional captured image brightness value array having respective captured image brightness values at the pixel positions of the pixel position array, the concatenated linear captured image brightness value arrays being stored in the captured image memory as respective captured image brightness values at the pixel positions of the two-dimensional pixel position array.
6. The system of claim 5, wherein the off-grid positions of the off-grid position array are located at respective intersections of a plurality of regularly spaced intermediate columns and pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.
7. The system of claim 6, wherein the predetermined sensitivity pattern has a first predetermined sensitivity level and a second predetermined sensitivity level, and wherein adjacent pixel positions in each one of the pixel rows correspond to different ones of the first and second predetermined sensitivity levels, and adjacent pixel positions in each one of the pixel columns correspond to the same one of the first and second predetermined sensitivity levels.
8. The system of claim 1, wherein the image sensor has a two-dimensional array of light-sensing elements and the multiplicity of pixel positions are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, the captured image memory for storing the captured image brightness values produced by the light-sensing elements at corresponding ones at the multiplicity of pixel positions of the pixel position array.
9. The system of claim 8 wherein the off-grid positions of the off-grid position array are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each intermediate row being parallel to the pixel rows and extending between a respective adjacent pair of pixel rows and each intermediate column being parallel to the pixel columns and extending between a respective adjacent pair of pixel columns.
10. The system of claim 8, wherein the off-grid positions of the off-grid position array are located at respective intersections of a plurality of regularly spaced intermediate rows and the plurality of pixel columns, and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of pixel columns.
11. The system of claim 10, wherein the predetermined sensitivity pattern of the array of light-sensing elements has first and second predetermined sensitivity levels, and wherein adjacent pixel positions in each one of the pixel rows and adjacent pixel positions in each one of the pixel columns correspond to different ones of the first and second predetermined sensitivity levels.
12. The system of claim 1, wherein the output image generator comprises an output image memory for storing the respective estimated off-grid brightness values derived by the off-grid estimator as the pixel brightness values of the output image.
13. The system of claim 1, wherein the off-grid estimator comprises:
- an off-grid position generator for providing the off-grid positions of the off-grid position array;
- a sensitivity pattern memory for storing data indicative of the sensitivity level corresponding to each one of the pixel positions of the pixel position array;
- a plurality of sensitivity level off-grid brightness value estimators each corresponding to a respective one of the plurality of sensitivity levels of the light-sensing elements, and receiving off-grid positions from the off-grid position generator and sensitivity pattern data from the sensitivity pattern memory, each one of the sensitivity level off-grid brightness value estimators being responsive to an off-grid position received from the off-grid position generator and the sensitivity pattern data from the sensitivity pattern memory for deriving from the captured image brightness values in the captured image memory a respective sensitivity level off-grid brightness value for the corresponding sensitivity level at the received off-grid position; and
- an accumulator for combining the respective sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimators for the corresponding sensitivity levels at each off-grid position received from the off-grid position generator to derive a respective estimated off-grid brightness value at each off-grid position received from the off-grid position generator.
14. The system of claim 13, wherein the accumulator comprises an adder for adding the sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimator for corresponding sensitivity levels at each received off-grid position to derive the respective estimated off-grid brightness value at each off-grid positions received from the off-grid position generator.
15. The system of claim 13, wherein the accumulator comprises a weighting factor memory for storing a plurality of predetermined weighting factors each corresponding to a different one of the plurality of sensitivity levels of the light-sensing elements, a plurality of multipliers each for multiplying the sensitivity level off-grid brightness value corresponding to a respective one of the sensitivity levels at a received off-grid position by the weighting factor corresponding to the respective one of the sensitivity levels to provide a respective weighted sensitivity level off-grid brightness value for the corresponding sensitivity level at the received off-grid position, and an adder for summing the respective weighted sensitivity level off-grid brightness values provided by the plurality of multipliers to derive a respective estimated off-grid brightness value at the received off-grid position received from the off-grid position generator.
16. The system of claim 1, wherein the on-grid resampler comprises:
- a pixel position generator for providing the pixel positions of the pixel position array;
- an interpolator for receiving pixel positions provided by the pixel position generator and being responsive to a pixel position received from the pixel position generator for deriving from the estimated off-grid brightness values in the off-grid brightness value memory a respective interpolated on-grid brightness value at the received pixel position; and
- a response function compensator for compensating the respective interpolated on-grid brightness value derived by the interpolator at each pixel position received from the pixel position generator by the inverse of a combined response function of the light-sensing elements to derive a respective resampled on-grid brightness value at each pixel position received from the pixel position generator, the combined response function of the light-sensing elements being the sum of weighted response functions of light-sensing elements having different ones of the sensitivity levels, each one of the weighted response functions being a response function of a light-sensing element having a respective one of the sensitivity levels multiplied by a predetermined weighting factor for the respective one of the sensitivity levels.
17. The system of claim 16, wherein the response function compensator comprises a lookup table memory storing data representing the inverse of the combined response function of the light-sensing elements and a mapper for deriving from the respective interpolated on-grid brightness value derived by the interpolator at a received pixel position a respective index for the lookup table memory and providing a data value in the lookup table memory corresponding to the respective index to the output image memory, the data value being the pixel brightness value of the output image at the received pixel position.
18. The system of claim 16, wherein the interpolator is responsive to a pixel position received from the pixel position generator for computing a product of the estimated off-grid brightness values at a predetermined number of off-grid positions that are nearest neighbors to the received pixel position and an interpolation kernel having the same dimensions as the predetermined number of off-grid positions.
19. The system of claim 16, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel columns being orthogonal to the pixel rows, and the off-grid positions provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each intermediate column being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, and wherein the interpolator derives the interpolated on-grid brightness values at each one of the pixel positions (x,y) received from the pixel position generator using the relation I ( x - 2, y - 2 ) = ∑ i = 0 3 ∑ j = 0 3 B ( x - 1.5 + i, y - 1.5 + j ) ϕ ij, where x has values from 2.5 to xSize−2.5, y has values from 2.5 to ySize−2.5, I(x−2, y−2) is the interpolated on-grid brightness value at pixel position (x−2, y−2), B(x−1.5+i, y−1.5+j) is the off grid brightness value at off-grid position (x−1.5+i, y−1.5+j), φij is the value of a 4×,4 cubic interpolation kernel φ at indices i and j, indices i and j each has values from 0 to 3, xSize is the dimension of the captured image in the direction of the pixel rows, and ySize is the dimension of the captured image in the direction of the pixel columns.
20. The system of claim 16, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel columns and a plurality of regularly spaced pixel rows, the pixel rows being orthogonal to the pixel columns, and the off-grid positions provided by the off-grid generator are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, and wherein the interpolator derives the interpolated on-grid brightness values at each one of the pixel positions (x,y) received from the pixel position generator using the relation I ( x - 2, y - 2 ) = ∑ i = 0 3 ∑ j = 0 3 B ( x - 0.5 i - 0.5 j, y - 0.5 i + 0.5 j ) ϕ ij, where x has values from 2.5 to xSize−2.5, y has values from 2.5 to ySize−2.5, I(x−2, y−2) is the interpolated on-grid brightness value at pixel, position (x−2, y−2), B(x+0.5i−0.5j, y 1.5+0.5i+0.5j) is the off-grid brightness value at off-grid position (x+0.5i−0.5j, y 1.5+0.5i+0.5j), 4 is the value of a 4×4 cubic interpolation kernel φ at indices i and j, indices i and j each have values from 0 to 3, xSize is the dimension of the captured image in the direction of the pixel rows, and ySize is the dimension of the captured image in the direction of the pixel columns.
21. The system of claim 1, wherein the off-grid estimator comprises:
- an off-grid position generator for providing the off-grid positions of the off-grid position array;
- a sensitivity pattern memory for storing data indicative of the sensitivity level corresponding to each one of the pixel positions of the pixel position array;
- a plurality of sensitivity level off-grid brightness value estimators, each corresponding to a respective one of the sensitivity levels, for receiving off-grid positions from the off-grid position generator and sensitivity pattern data from the sensitivity pattern memory, each one of the plurality of sensitivity level off-grid brightness value estimators being responsive to an off-grid position received from the off-grid position generator and the sensitivity pattern data from the sensitivity pattern memory for deriving from the captured image brightness values in the captured image memory a respective sensitivity level off-grid brightness value for the corresponding sensitivity level at the received off-grid position;
- an accumulator for combining the respective sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimator for the corresponding sensitivity levels at each off-grid position received from the off-grid position generator to derive a respective combined sensitivity level off-grid brightness value at each off-grid position received from the off-grid position generator; and
- a response function compensator for compensating the respective combined sensitivity level off-grid brightness value at each off-grid position received from the off-grid position generator by the inverse of a combined response function of the light-sensing elements to derive a respective estimated off-grid brightness value at each off-grid position received from the off-grid position generator, the combined response function being the sum of weighted response functions of light-sensing elements having different ones of the plurality of sensitivity levels, each one of the weighted response functions being a response function of a light-sensing element having a respective one of the sensitivity levels multiplied by a predetermined weighting factor for the respective one of the sensitivity levels.
22. The system of claim 21, wherein the response function compensator comprises a lookup table memory storing lookup table data representing the inverse of the combined response function of the light-sensing elements, and a mapper for deriving from the respective combined sensitivity level off-grid brightness value at each off-grid position received from the off-grid position generator a respective index for the lookup table memory and providing a data value in the lookup table memory corresponding to the respective index as the respective estimated off-grid brightness value at the off-grid position received from the off-grid position generator.
23. The system of claim 21, wherein the output image generator comprises:
- an off-grid brightness value memory for storing the respective estimated off-grid brightness value derived by the response function compensator at each off-grid position received from the off-grid position generator;
- an on-grid resampler for deriving from the estimated off-grid brightness values in the off-grid brightness value memory respective resampled on-grid brightness values at the pixel positions of the pixel position array; and
- an output image memory for storing the respective resampled on-grid brightness values derived by the on-grid resampler as the pixel brightness values of the output image,
- and wherein the on-grid resampler comprises: a pixel position generator for providing pixel positions of the pixel position array; and an interpolator for receiving pixel positions provided by the pixel position generator and being responsive to a pixel position received from the pixel position generator for deriving from the estimated off-grid brightness values in the off-grid brightness value memory a respective resampled on-grid brightness value at the received pixel position.
24. The system of claim 23, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and the off-grid positions provided by the of grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, and wherein the interpolator derives the resampled on-grid brightness values at each one of the pixel positions (x,y) received from the pixel position generator by the relation I ( x - 2, y - 2 ) = ∑ i = 0 3 ∑ j = 0 3 B ( x - 1.5 + i, y - 1.5 + j ) ϕ ij, where x has values from 2.5 to xSize−2.5, y has values from 2.5 to ySize−2.5, I(x−2, y−2) is the resampled on-grid brightness value at pixel position (x−2, y−2), B(x−1.5+i, y−1.5+j) is the off-grid brightness value at off-grid position (x−1.5+i, y−1.5+j), φ is the value of a 4×4 cubic interpolation kernel φ at indices i and j, the indices i and j each have values from 0 to 3, xSize is the dimension of the captured image in the direction of the pixel rows, and ySize is the dimension of the captured image in the direction of the pixel columns.
25. The system of claim 24, wherein the 4×4 cubic interpolation kernel φ has the form:
26. The system of claim 23, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and the off-grid positions provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, and wherein the interpolator derives the resampled on-grid brightness values at each one of the pixel positions (x,y) received from the pixel position generator using the relation I ( x - 2, y - 2 ) = ∑ i = 0 3 ∑ j = 0 3 B ( x - 0.5 i - 0.5 j, y - 1.5 + 0.5 i + 0.5 j ) ϕ ij, where x has values from 2.5 to xSize−2.5, y has values from 2.5 to ySize−2.5, I(x−2,y−2) is the resampled on-grid brightness value at pixel position (x−2,y−2), B(x+0.5i−0.5j, y−1.5+0.5i+0.5j) is the off-grid brightness value at the off-grid position (x+0.5i−0.5j, y−1.5+0.5i+0.5j), φij is the value of a 4×4 cubic interpolation kernel φ at indices i and j, indices i and j each have values from 0 to 3, xSize is the dimension of the captured image in the direction of the pixel rows, and ySize is dimension of the captured image in the direction of the pixel columns.
27. The system of claim 23, wherein the pixel positions provided by the pixel position generator are located at respective regularly spaced pixel positions in a pixel row defining a linear pixel position array and the off-grid positions provided by the off-grid position generator are each located midway between a respective adjacent pair of pixel positions of the pixel position row, and wherein the interpolator derives the resampled on-grid brightness values at each one of the pixel positions x received from the pixel position generator using the relation I o ( x - 3 ) = ∑ k = - 1.5 k = 1.5 B ( x + k ) ψ ( k ), where x has values from 3.5 to xSize−3.5, I0(x−3) is the resampled on-grid brightness value at pixel position (x−3), B(x+k) is the estimated off-grid brightness value at off-grid position (x+k), k is a position of a 4×1 cubic interpolation kernel Ψ, Ψ(k) is the value of the cubic interpolation kernel at position k, the position k has values from −1.5 to 1.5, and xSize is the dimension of the captured line image.
28. The system of claim 27, wherein the 4×1 cubic interpolation kernel x has the form:
29. The system of claim 21, wherein the accumulator of the off-grid estimator comprises an adder for adding the respective sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimators for the corresponding sensitivity levels at each off-grid position received from the off-grid position generator to provide the combined sensitivity level off-grid brightness values at each off-grid position received from the off-grid position generator.
30. The system of claim 21, wherein the accumulator of the off-grid estimator comprises a weighting factor memory for storing a plurality of predetermined weighting factors each corresponding to a different one of the plurality of sensitivity levels of the light-sensing elements, a plurality of multipliers each for multiplying the sensitivity level off-grid brightness value corresponding to a respective one of the sensitivity levels at a received off-grid position by the weighting factor corresponding to the respective one of the sensitivity levels to provide a weighted sensitivity level off-grid brightness value for the corresponding sensitivity level at the off-grid position received from the off-grid position generator, and an adder for summing the weighted sensitivity level off-grid brightness values at the received off-grid position provided by the plurality of multipliers to derive the combined sensitivity level off-grid brightness value at the off-grid positions received from the off-grid position generator.
31. The system of claim 30, wherein the array of light-sensing elements of the image sensor is a linear array and the pixel position array is a linear array having a multiplicity of pixel positions at respective regularly spaced pixel positions in a pixel row, and wherein the off-grid positions of the off-grid position array are each located between a respective adjacent pair of pixel positions of the pixel position array and the sensitivity level off-grid brightness values derived by the plurality of sensitivity level off-grid brightness value estimators for an off-grid position received from the off-grid position generator are respective captured image brightness values at a predefined number of pixel positions that are nearest neighbors to the received off-grid position, each one of the predefined number of nearest neighbor pixel positions corresponding to a different one of the plurality of sensitivity levels of the light-sensing elements.
32. The system of claim 31, wherein each one of the off-grid positions of the off-grid position array is located midway between a respective adjacent pair of pixel positions of the linear pixel position array and the predetermined sensitivity pattern of the light-sensing elements has a first and a second predetermined sensitivity level, and wherein the combined sensitivity level off-grid brightness value at an off-grid position x′ received from the off-grid position generator is expressed as where B(x′) is the combined sensitivity level off-grid brightness value at the received off-grid position x′, W1 is the weighting factor for the sensitivity level corresponding to pixel position (x′−0.5), W2 is the weighting factor for the sensitivity level corresponding to pixel position (x′+0.5), Ic(x′−0.5) is the captured image brightness value at pixel position (x′−0.5) and Ic(x′+0.5) is the captured image brightness value at pixel position (x′+0.5), and the combined response function of the light-sensing elements is expressed as where P1(E) is the radiometric response function of a light-sensing element having the sensitivity level corresponding to pixel position (x′−0.5) and P2(E) is the radiometric response function of a light-sensing element having the sensitivity level corresponding to pixel position (x′+0.5).
- B(x′)=W1Ic(x′−0.5)+W2Ic(x′+0.5),
- S(E)=W1P1(E)+W2P2(E),
33. The system of claim 30, wherein the image sensor has a two-dimensional array of light-sensing elements and the multiplicity of pixel positions are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns defining a two-dimensional pixel position array, the pixel rows being orthogonal to the pixel columns, the multiplicity of off-grid positions are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each one of the intermediate rows being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.
34. The system of claim 33, wherein the predetermined sensitivity pattern of the array of light-sensing elements corresponds to a pixel position array having repetitive disposed groups of four nearest neighbor pixel positions, where each one of the four nearest neighbor pixel positions in each group corresponds to a respective one of four predetermined sensitivity levels in a common predetermined positional order.
35. The system of claim 30, wherein the image sensor has a two-dimensional array of light-sensing elements and the multiplicity of pixel positions are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns defining a two-dimensional pixel position array, the pixel rows being orthogonal to the pixel columns, and wherein the multiplicity of off-grid positions are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns, and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.
36. The system of claim 30, wherein the predetermined sensitivity pattern of the array of light-sensing elements has a first and a second predetermined sensitivity level, and adjacent pixel positions in each one of the pixel rows and adjacent pixel positions in each one of the pixel columns correspond to different ones of the predetermined first and second sensitivity levels.
37. The system of claim 30, wherein the array of light-sensing elements of the image sensor is a linear array and the multiplicity of pixel positions are located at respective regularly spaced pixel positions in a pixel row so as to form a linear captured image brightness value array, and wherein the linear array of light-sensing elements of the image sensor is adapted to be sequentially exposed to successive regularly spaced linear regions of the scene in multiple exposures, each one of the exposures producing a respective linear captured image brightness value array, the respective linear captured image brightness value arrays produced by the multiple exposures being concatenated to form a two-dimensional captured image brightness value array having respective captured image brightness values at pixel positions of a pixel position array located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the multiplicity of off-grid positions are located at respective intersections of the pixel rows and a plurality of regularly spaced intermediate columns, each one of the intermediate columns being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.
38. The system of claim 37, wherein the predetermined sensitivity pattern of the array of light-sensing elements has a first and a second predetermined sensitivity level, and wherein adjacent pixel positions in each pixel row correspond to different ones of the first and the second predetermined sensitivity levels and adjacent pixel positions in each pixel column correspond to the same one of the first and the second predetermined sensitivity levels.
39. The system of claim 21, wherein each one of the sensitivity level off-grid brightness value estimators is responsive to an off-grid position received from the off-grid position generator and the data indicative of the sensitivity level corresponding to each one of the pixel positions from the sensitivity pattern memory for determining a pixel position corresponding to the same sensitivity level as the sensitivity level off-grid brightness value estimator and located nearest the received off-grid position, and for estimating a sensitivity level off-grid brightness value at the received off-grid position to be equal to the captured image brightness value at the nearest pixel position corresponding to the same sensitivity level as the sensitivity level off-grid brightness value estimator.
40. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having an array of photodiodes in spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, the charge coupled device having respective integrated lenses formed over the photosensitive surfaces of predetermined ones of the photodiodes, each integrated lens concentrating the incident light onto the respective light-sensing surface, whereby the sensitivity level of each one of the photodiodes is determined by whether or not an integrated lens is formed over the light-sensing surface thereof.
41. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having an array of photodiodes arranged in spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, each one of the photodiodes having an integrated light filter formed above the light-sensing surface thereof, the transparency of each integrated light filter being fixed during fabrication of the charge coupled device, whereby the sensitivity level of each one of the photodiodes in the array is determined by the transparency of the integrated light filter above the light-sensing surface thereof.
42. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having an array of photodiodes arranged in spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, the charge coupled device being formed with an opaque layer overlying the array of photodiodes with a respective aperture formed in the opaque layer above the light-sensing surface of each one of the photodiodes, the respective size of the aperture above the light-sensing surface of each one of the photodiodes being fixed during fabrication of the charge coupled device, whereby the sensitivity level of each one of the photodiodes in the array is determined by the size of the aperture above the light-sensing surface thereof.
43. The system of claim 1, wherein the image sensor is an integrated circuit charge coupled device having a multiplicity of photodiodes in an array of spaced orthogonal rows and columns, each one of the photodiodes having a light-sensing surface adapted to be exposed to incident light and being operatively biased to store photogenerated charge therein, the charge coupled device having an interline structure in which the columns of photodiodes are each disposed adjacent a corresponding vertical shift register, each photodiode having an associated transfer gate electrode for controlling the transfer of stored photogenerated charge from the photodiode to the corresponding vertical shift register, the stored photogenerated charge in each one of the photodiodes being operatively periodically removed by the application of a periodic reset pulse to the device, the photogenerated charge stored in each one of the photodiodes being operatively transferred to a respective stage of the corresponding vertical shift register by the application of a transfer trigger pulse to the associated transfer gate electrode and being accumulated therein for an exposure interval, the transfer trigger pulse being applied immediately preceding a reset pulse, the respective transfer trigger pulses being operatively applied to the transfer gate electrodes associated with the photodiodes in the array at predetermined frequencies, whereby the sensitivity level of each one of the photodiodes in the array is determined by the frequency of transfer trigger pulses applied to the transfer gate electrode associated therewith.
44. The system of claim 1, wherein the off-grid estimator comprises:
- a pixel position generator for providing the pixel positions of the pixel position array;
- a sensitivity pattern memory for storing the sensitivity level corresponding to each one of the pixel positions;
- a response function compensator receiving pixel positions of the pixel position array from the pixel position generator, data indicative of the sensitivity levels corresponding to the received pixel positions from the sensitivity pattern memory and captured image brightness values at the received pixel positions from the captured image memory, and being responsive to a pixel position received from the pixel position generator and data indicative of the sensitivity level corresponding to the received pixel position for retrieving from the captured image memory the captured image brightness value at the received pixel position and for compensating the retrieved captured imaged brightness value at the received pixel positions by the inverse of a response function of a light-sensing element having the sensitivity level corresponding to the received pixel position to provide a respective compensated brightness value at the pixel position received from the pixel position generator;
- a compensated on-grid brightness value memory for storing respective compensated captured image brightness values derived by the response function compensator;
- an off-grid position generator for providing the off-grid positions of the off-grid position array;
- a pseudoinverse estimator receiving off-grid positions from the off-grid position generator and being responsive to an off-grid position received from the off-grid position generator for deriving from the compensated captured image brightness values in the compensated on-grid brightness value memory a respective estimated off-grid brightness value at the off-grid positions received from the off-grid position generator.
45. The system of claim 44, wherein the output image generator comprises
- an off-grid brightness value memory for storing the respective estimated off-grid brightness values derived by the pseudoinverse estimator;
- an on-grid resampler for deriving from the estimated off-grid brightness values in the off-grid brightness value memory respective on-grid resampled brightness values at the pixel positions of the pixel position array; and
- an output image memory for storing the respective resampled on-grid brightness values derived by the on-grid resampler as pixel brightness values of the output image, and wherein the on-grid resampler comprises: a second pixel position generator for providing the pixel positions of the pixel position array; and an interpolator receiving pixel positions from the pixel position generator and being responsive to a pixel position received from the second pixel position generator for deriving from the estimated off-grid brightness values in the off-grid brightness value memory the resampled on-grid brightness value at the pixel position received from the second pixel position generator.
46. The system of claim 45, wherein the pixel positions of the pixel position array provided by the second pixel position generator are located at respective regularly spaced pixel positions of a linear pixel position array, and the off-grid positions of the off-grid position array provided by the off-grid position generator are each located between a respective adjacent pair of pixel positions of the linear pixel position array.
47. The system of claim 46, wherein the off-grid positions provided by the off-grid position generator are each located midway between a respective adjacent pair of pixel positions of the linear pixel position array, the pixel positions provided by the second pixel position generator having coordinate x, where x has values from 3.5 to xSize−3.5 and xSize is the dimension of the captured line image.
48. The system of claim 47, wherein the interpolator is responsive to a pixel position received from the second pixel position generator for deriving a respective resampled on-grid position brightness value in accordance with the relation I o ( x - 3 ) = ∑ k = - 1.5 k = 1.5 B ( x + k ) ψ ( k ), where Io(x−3) is the resampled on-grid brightness value at pixel position (x−3), B(x+k) is the estimated off-grid brightness value at off-grid position (x+k), k is a position of a 4×1 cubic interpolation kernel Ψ, Ψ(k) is the value of the cubic interpolation kernel at position k, and the position k has values from −1.5 to 1.5.
49. The system of claim 48, wherein the 4×1 cubic interpolation kernel Ψ has the form:
50. The system of claim 45, wherein the pixel positions of the pixel position array provided by the second pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the off-grid positions of the off-grid position array provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intelluediate rows and a plurality of regularly spaced intermediate columns, each intermediate row being parallel to the pixel rows and extending between a respective adjacent pair of the pixel rows and each intermediate column being parallel to the pixel columns and extending between a respective adjacent pair of the pixel columns.
51. The system of claim 50, wherein the interpolator derives a respective resampled on-grid brightness value at a pixel position received from the second pixel position generator by computing the product of respective estimated off-grid brightness values at a predetermined number of off-grid positions that are nearest neighbors to the received pixel position and an interpolation kernel having the same dimensions as the predetermined number of off-grid positions.
52. The system of claim 51, wherein the plurality of intermediate rows each extend medially between a respective adjacent pair of pixel rows and the plurality of regularly spaced intermediate columns each extend medially between a respective adjacent pair of pixel columns, and wherein the pixel positions provided by the second pixel position generator have coordinates (x,y) where x has values from 2.5 to xSize−2.5 and, y has values from 2.5 to ySize 2.5, xSize is the dimension of the captured image in the direction of the pixel rows and ySize is the dimension of the captured image in the direction of the pixel columns.
53. The system of claim 52, wherein the interpolator is responsive to a pixel position (x,y) received from the second pixel position generator for deriving a respective resampled on-grid brightness value in accordance with the relation I ( x - 2, y - 2 ) = ∑ i = 0 3 ∑ j = 0 3 B ( x - 1.5 + i, y - 1.5 + j ) ϕ ij, where I(x−2, y−2) is the resampled on-grid brightness value at pixel position (x−2, y−2), B(x−1.5+i, y−1.5+j) is the estimated off-grid brightness value at off-grid position (x−1.5+i, y 1.5+j), φij is the value of a 4×4 cubic interpolation kernel φ at indices i and j, and indices i and j each has values from 0 to 3.
54. The method of claim 53, wherein the 4×4 cubic interpolation kernel φ has the form:
55. The system of claim 51, wherein the interpolation kernel is a 2-dimensional Gaussian kernel.
56. The system of claim 55, wherein the 2-dimensional Gaussian kernel has the form:
57. The system of claim 51, wherein the interpolation kernel is a bi-cubic B-spline interpolation filter kernel.
58. The system of claim 57, wherein the bi-cubic B-spline interpolation filter kernel has the form:
59. The system of claim 51, wherein the interpolation kernel is a bi-linear interpolation filter kernel.
60. The system of claim 59, wherein the bi-linear interpolation kernel has the form:
61. The system of claim 45, wherein the pixel positions of the pixel position array provided by the second pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the off-grid positions of the off-grid position array provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate pixel columns being parallel to the pixel columns and extending between a respective adjacent pair of pixel columns.
62. The system of claim 61, wherein the interpolator includes means for shifting the off-grid position array to coincide with the pixel position array, whereby the resampled on-grid brightness value at each pixel position of the pixel position array is equal to the estimated off-grid brightness value at a coincident off-grid position of the shifted off-grid position array.
63. The system of claim 45, wherein the pixel positions provided by the second pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and wherein the off-grid positions provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns, and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each intermediate column being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, the pixel positions provided by the second pixel position generator having coordinates (x,y) where x has values from 2.5 to xSize−2.5, y has values from 2.5 to ySize−2.5, xSize is the dimension of the captured image in the direction of the pixel rows and ySize is the dimension of the captured image in the direction of the pixel columns.
64. The system of claim 63, wherein the interpolator is responsive to a pixel position received from the second pixel position generator for deriving a respective resampled on-grid brightness value in accordance with the relation I ( x - 2, y - 2 ) = ∑ i = 0 3 ∑ j = 0 3 B ( x - 0.5 i - 0.5 j, y - 1.5 + 0.5 i + 0.5 j ) ϕ ij, where I(x−2,y−2) is the resampled brightness value at pixel position (x−2, y−2), B(x+0.5i−0.5j, y 1.5+0.5i+0.5j) is the off-grid brightness value at off-grid position (x+0.5i−0.5j, y 1.5+0.5i+0.5j), φij is the value of a 4×4 cubic interpolation kernel φ at indices i and j, and the indices i and j each have values from 0 to 3.
65. The system of claim 44, wherein the response function compensator comprises a lookup table memory storing separate lookup table data representing the inverse of respective response functions of light-sensing elements having different ones of the plurality of sensitivity levels, and a mapper receiving from the captured image memory the captured image brightness value at the pixel position received from the first pixel position generator for deriving therefrom a respective index for the lookup table data representing the inverse of the response function of a light-sensing element having the sensitivity level corresponding to the received pixel position, and providing a compensated on-grid brightness value corresponding to the index to the compensated on-grid brightness value memory.
66. The system of claim 44, wherein the pseudoinverse estimator comprises means responsive to an off-grid positions received from the off-grid position generator for retrieving from the compensated on-grid brightness value memory respective compensated on-grid brightness values at a predetermined number of pixel positions that are nearest neighbors to the received off-grid position, a comparator for comparing each one of the respective compensated on-grid brightness values at the nearest neighbor pixel positions to a predetermined low threshold value and a predetermined high threshold value, an adder means for computing a sum of compensated on-grid brightness values at the nearest neighbor pixel positions that are greater than the predetermined low threshold value and less than the predetermined high threshold value, and a divider for deriving the estimated off-grid brightness value at the received off-grid position by dividing the sum computed by the adder by the number of compensated on-grid brightness values included in the sum.
67. The system of claim 44, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, the pixel rows being orthogonal to the pixel columns, and the off-grid positions provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and a plurality of regularly spaced intermediate columns, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each intermediate column being parallel to the pixel columns and extending medially between a respective adjacent pair of pixel columns, and wherein the off-grid positions provided by the off-grid position generator have coordinates (x′,y′) where x′ has values from 1 to xSize−1 and y′ has values from 1 to ySize−1, xSize being the dimension of the captured image in the direction of the pixel rows and ySize being the dimension of the captured image in the direction of the pixel columns.
68. The system of claim 67, wherein the predetermined sensitivity pattern of the array of light-sensing elements corresponds to a pixel position array having repetitively disposed groups of four nearest neighbor pixel positions, where each of the four nearest neighbor pixel positions in each group corresponds to a respective one of four predetermined sensitivity levels in a common predetermined positional order.
69. The system of claim 67, wherein the pseudoinverse estimator for estimating the respective brightness value at an off-grid position (x′,y′) received from the off-grid position generator comprises: ϕ ij 2 ϕ ji I ( u, v ) ∑ ab ϕ ab 2 for each compensated on-grid brightness value I(u,v) at the sixteen nearest neighbor pixel positions (u,v) that is greater than the predetermined threshold value and less than the predetermined threshold value using the indices i and j calculated by the first computing means, where φmn is the value of the 4×4 interpolating kernel φ at indices m and n; ϕ ij 2 ϕ ji I ( u, v ) ∑ ab ϕ ab 2 calculated by the second computing means to derive a first sum s;
- a memory for storing a 4×4 cubic interpolator kernel φ;
- a comparator for comparing the compensated on-grid brightness value at each one of sixteen pixel positions (u,v) that are nearest neighbors to the received off-grid position (x′,y′) with a predetermined low threshold value and a predetermined high threshold value;
- a first computing means for calculating the indices i and j of the 4×4 cubic interpolation kernel φ using the relations i=x′−u−1.5 and j=y′−v−1.5 for each one of the sixteen nearest neighbor pixel positions (u,v) having a compensated on-grid brightness value I(u,v) that is greater than the predetermined low threshold value and less than the predetermined high threshold value, where u has values from x′−1.5 to x′+1.5 and v has values from y′−1.5 to y′+1.5;
- a second computing means for calculating the quantity
- a third computing means for calculating the quantities, for values of i and j calculated by the first computing means;
- a first adder for adding the quantities
- a second adder for adding the quantities calculated by the third computing means to derive a second sum w; and
- a divider for deriving the estimated off-grid brightness value at the received off-grid position (x′,y′) by dividing the first sum s by the second sum w.
70. The system of claim 44, wherein the pixel positions provided by the pixel position generator are located at respective intersections of a plurality of regularly spaced pixel rows and a plurality of regularly spaced pixel columns, and the off-grid positions provided by the off-grid position generator are located at respective intersections of a plurality of regularly spaced intermediate rows and the pixel columns, and at respective intersections of a plurality of regularly spaced intermediate columns and the pixel rows, each one of the intermediate rows being parallel to the pixel rows and extending medially between a respective adjacent pair of the pixel rows, each intermediate column being parallel to the pixel columns and extending medially between a respective adjacent pair of the pixel columns, the off-grid positions provided by the off-grid position generator having coordinates (x′,y′) where x′ has values from 1 to xSize−1 and y′ has values from 0.5 to ySize−0.5 for off-grid positions located at respective intersections of the pixel rows and the intermediate columns, and where x′ has values from 0.5 to ySize−0.5 and y′ has values from 1 to ySize−1 for off-grid positions located at respective intersections of the pixel columns and the intermediate rows, xSize being the dimension of the captured image in the direction of the pixel rows and ySize being the dimension of the captured image in the direction of the pixel columns.
71. The system of claim 70, wherein the predetermined sensitivity pattern of the array of light-sensing elements has first and second predetermined sensitivity levels, and adjacent pixel positions in each pixel row and adjacent pixel positions in each pixel column correspond to different ones of the first and second predetermined sensitivity levels.
72. The system of claim 70, wherein the pseudoinverse estimator for estimating the respective off-grid brightness value at an off-grid position (x′,y′) received from the off-grid position generator comprises: ϕ ij 2 ϕ ji I ( u, v ) ∑ ab ϕ ab 2, where the values of i and j are those used by the first computing means to calculate the coordinate (u,v), and φmn is the value of the 4×4 cubic interpolation kernel φ at indices m and n; ϕ ij 2 ϕ ji I ( u, v ) ∑ ab ϕ ab 2 calculated by the second computing means to derive a first sum s;
- a memory for storing a 4×4 cubic interpolation kernel φ;
- a first computing means responsive to an off-grid position (x′,y′) received from the off-grid position generator for computing coordinates (u,v) using the relations u=x′+0.5 i−0.5j and v=y′−1.5+0.5i+0.5j, where i and j are indices of the 4×4 cubic interpolation kernel dφ, and each one of the indices i and j has values from 0 to 3;
- means responsive to each pair of coordinates (u,v) computed by the first computing means for determining whether the coordinates (u,v) are those of a pixel position;
- a comparator responsive to the coordinates (uv) being coordinates of a pixel position for comparing the compensated on-grid brightness value I(u,v) at the pixel position (u,v) with a predetermined low threshold value and a predetermined high threshold value;
- a second computing means responsive to the coordinates (u,v) being the coordinates of a pixel position and the compensated on-grid brightness value I(uv) at the pixel position (u,v) being greater than a predetermined low threshold value and less than a predetermined high threshold value for calculating the quantity
- a third computing means for calculating the quantity φij2 for values of i and j used by the first computing means to calculate pixel position coordinates (uv) at which the compensated on-grid brightness value I(u,v) is greater than the predetermined low threshold value and less than the predetermined high threshold value;
- a first adder for adding the quantities
- a second adder for adding the quantities calculated by the third computing means to derive a second sum w; and
- a divider for deriving the estimated off-grid brightness value at the off grid position (x′,y′) received from the off-grid position generator by dividing the first sum s by the second sum w.
73. The system of claim 72, wherein the 4×4 cubic interpolation kernel φ has the form:
74. The system of claim 44, wherein the pixel positions provided by the pixel position generator are located at respective regularly spaced pixel positions in a linear pixel position array and the off-grid positions provided by the off-grid position generator are each located midway between a respective adjacent pair of pixel positions of the linear pixel position array, and wherein the predetermined sensitivity pattern of the light-sensing elements has a first and a second predetermined sensitivity level, and adjacent pixel positions of the linear pixel position array correspond to different ones of the first and second predetermined sensitivity levels.
75. The system of claim 74, wherein the pixel positions provided by the pixel position generator has values from 0.5 to xSize−0.5, where xSize is the dimension of the captured line image, and the off-grid positions provided by the off-grid position generator have values from 2 to xSize−2.
76. The system of claim 75, wherein the pseudoinverse estimator for estimating the respective brightness values at an off-grid position x′ received from the off-grid position generator comprises:
- a memory for storing a 4□1 Gaussian interpolation kernel G;
- a comparator for comparing the compensated on-grid brightness value I(k) at each one of four pixel positions k that are nearest neighbors to the received off-grid position x′ with a predetermined low threshold value and a predetermined high threshold value, where k has values from x′−1.5 to x′+1.5;
- a first computing means for calculating the quantity I(k)G(kW−x′) for each one of the compensated on-grid brightness values I(k) at the four nearest neighbor pixel positions k that is greater than the predetermined low threshold value and less than the predetermined high threshold value, where G(k−x′) is the value of the Gaussian interpolation kernel G at position (k′ x′);
- a first adder for adding the quantities I(k)G(k−x′) for all values of I(k) greater than the predetermined low threshold value and less than the predetermined high threshold value to derive a first sum p;
- a second adder for adding the Gaussian interpolation kernel values G(k−x′) for all values of k where I(k) is greater than the predetermined low threshold value and less than the predetermined high threshold value to derive a second sum q; and
- a divider for deriving the estimated off-grid brightness value B(x′) at the off-grid position x′ received from the off-grid position generator.
77. The system of claim 76, wherein the predetermined low threshold value is the noise level of a light-sensing element having a sensitivity level corresponding to the pixel position of the compensated on-grid brightness value being compared therewith compensated by the inverse of the response function of the light-sensing element, and the predetermined high threshold value is the saturation or near saturation brightness value of a light-sensing element having the sensitivity level corresponding to the pixel position of the compensated on-grid brightness value being compared therewith.
4479062 | October 23, 1984 | Kawasaki et al. |
4590367 | May 20, 1986 | Ross et al. |
4623928 | November 18, 1986 | Handy |
4652916 | March 24, 1987 | Suzaki et al. |
4868649 | September 19, 1989 | Gaudin |
4873561 | October 10, 1989 | Wen |
4918534 | April 17, 1990 | Lam et al. |
4996600 | February 26, 1991 | Nishida et al. |
5010504 | April 23, 1991 | Lee et al. |
5030985 | July 9, 1991 | Bryant |
5138458 | August 11, 1992 | Nagasaki et al. |
5185671 | February 9, 1993 | Lieverman et al. |
5193016 | March 9, 1993 | Cornuejols |
5282063 | January 25, 1994 | Deacon et al. |
5289286 | February 22, 1994 | Nakamura et al. |
5309243 | May 3, 1994 | Tsai |
5331420 | July 19, 1994 | Yamano et al. |
5373322 | December 13, 1994 | Laroche et al. |
5386228 | January 31, 1995 | Okino |
5416611 | May 16, 1995 | Tandon |
5420635 | May 30, 1995 | Konishi et al. |
5455621 | October 3, 1995 | Morimura |
5469216 | November 21, 1995 | Takahashi et al. |
5592219 | January 7, 1997 | Nakagawa |
5638118 | June 10, 1997 | Takahashi et al. |
5638119 | June 10, 1997 | Cornuejols |
5668596 | September 16, 1997 | Vogel |
5670280 | September 23, 1997 | Lawandy |
5703677 | December 30, 1997 | Simoncelli et al. |
5751352 | May 12, 1998 | Ogawa |
5767987 | June 16, 1998 | Wolff et al. |
5789737 | August 4, 1998 | Street |
5801773 | September 1, 1998 | Ikeda |
5812191 | September 22, 1998 | Orava et al. |
5828793 | October 27, 1998 | Mann |
5990950 | November 23, 1999 | Addison |
5990952 | November 23, 1999 | Hamazaki |
6028299 | February 22, 2000 | Hirama et al. |
6122408 | September 19, 2000 | Fang et al. |
6124974 | September 26, 2000 | Burger |
6211915 | April 3, 2001 | Harada |
6366318 | April 2, 2002 | Smith et al. |
6501504 | December 31, 2002 | Tatko et al. |
6753909 | June 22, 2004 | Westerman et al. |
6771312 | August 3, 2004 | Kamishima et al. |
6831692 | December 14, 2004 | Oda |
6864916 | March 8, 2005 | Nayar et al. |
6933971 | August 23, 2005 | Bezryadin |
6995791 | February 7, 2006 | Skow |
7084905 | August 1, 2006 | Nayar et al. |
7924321 | April 12, 2011 | Nayar et al. |
7999858 | August 16, 2011 | Nayar et al. |
20020027189 | March 7, 2002 | Murakami et al. |
20020050518 | May 2, 2002 | Roustaei |
20020126209 | September 12, 2002 | Yamada et al. |
4305807 | October 1994 | DE |
4420637 | December 1995 | DE |
19618476 | November 1997 | DE |
0503103 | September 1992 | EP |
0840502 | May 1998 | EP |
0912047 | April 1999 | EP |
0981245 | February 2000 | EP |
2687265 | August 1993 | FR |
2 255 465 | November 1992 | GB |
2 331 426 | May 1999 | GB |
59217358 | December 1984 | JP |
6070225 | March 1994 | JP |
06-141229 | May 1994 | JP |
6-153089 | May 1994 | JP |
06141229 | May 1994 | JP |
07-077700 | March 1995 | JP |
07-115643 | May 1995 | JP |
7-254965 | October 1995 | JP |
07254965 | October 1995 | JP |
07254966 | October 1995 | JP |
7264488 | October 1995 | JP |
8154201 | June 1996 | JP |
08223491 | August 1996 | JP |
08-340486 | December 1996 | JP |
08331461 | December 1996 | JP |
08340486 | December 1996 | JP |
9219824 | August 1997 | JP |
10069011 | March 1998 | JP |
1-0270673 | October 1998 | JP |
10-304235 | November 1998 | JP |
3074967 | August 2000 | JP |
WO 90/01844 | February 1990 | WO |
WO 93/14595 | July 1993 | WO |
WO94/18801 | August 1994 | WO |
WO 97/05742 | February 1997 | WO |
WO00/79784 | December 2000 | WO |
- U.S. Appl. No. 09/511,469, Jun. 26, 2006 Response to Supplemental Notice of Allowance.
- U.S. Appl. No. 09/511,469, Jun. 12, 2006 Supplemental Notice of Allowance.
- U.S. Appl. No. 09/511,469, Apr. 18, 2006 Amendment after Notice of Allowance.
- U.S. Appl. No. 09/511,469, Feb. 6, 2006 Notice of Allowance.
- U.S. Appl. No. 09/511,469, Sep. 8, 2005 Issue Fee Payment.
- U.S. Appl. No. 09/511,469, Jun. 16, 2005 Notice of Allowance.
- U.S. Appl. No. 09/511,469, May 9, 2005 Response to Ex-Parte Quayle Action.
- U.S. Appl. No. 09/511,469, Apr. 7, 2005 Ex-Parte Quayle Action.
- U.S. Appl. No. 10/959,679, Jul. 13, 2011 Issue Fee Payment.
- U.S. Appl. No. 10/959,679, Apr. 15, 2011 Notice of Allowance.
- U.S. Appl. No. 10/959,679, Jan. 10, 2011 Amendment and Request for Continued Examination (RCE).
- U.S. Appl. No. 10/959,679, Jul. 9, 2010 Final Office Action.
- U.S. Appl. No. 10/959,679, Apr. 1, 2010 Response to Non-Final Office Action.
- U.S. Appl. No. 10/959,679, Dec. 30, 2009 Non-Final Office Action.
- U.S. Appl. No. 10/959,679, Sep. 28, 2009 Response to Non-Final Office Action.
- Mann et al,, “On Being ‘Undigital’ With Digital Cameras: Extending Dynamic Range by Combining Differently Exposed Pictures”, M.I.T. Media Lab Perceptual Computing Section, p. 422-428, May 1995.
- Debevec et al, “Recovering High Dynamic Range Maps from Photographs”, SIGGRAPH '97 Conference Proceedings, Aug. 1997.
- Lee et al, “Scattered Date Interpolation with Multilevel B-Splines”, IEEE Transactions on Visualization and Computer Graphics, vol. 3, No. 3, p. 228-244, Jul.-Sep. 1997.
- Mitsunaga et al., “Radiometric Self Calibration”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR 99), p. 374-380, Jun. 1999.
- U.S. Appl. No. 10/959,679, Jun. 26, 2009 Non-Final Office Action.
- Pattanaik, et al., A Multiscale Model of Adaptation and Spatial Vision for Realistic Image Display:, In Proceedings of the 25th Annual Conference on Computer Graphics (SIGGRAPH '98), Orlando, FL, US, Jul. 19-24, 1998, pp. 278-298.
- Yamada, et al., “Effectiveness of Video Camera Dynamic Range Expansion for Lane Mark detection”, In Proceedings of the IEEE conference of intelligent Transportation System (ITSC '97), Boston, MA, US, Nov. 9-12, 1997, pp. 584-588.
- Brajovic, et al., “A sorting Image Sensor: an example of Massively Parallel Intensity-to-Time Processing for Low-Latency Computational Sensors”, Proceedings of the 1996 IEEE International Conference of Robotics and Automation, pp. 1638-1643 (1996).
- Burt, et al., “A Multiresolution Spline with Application to Image Mosiacs”, ACM Transactions on Graphics, 2(4):217-236 (1983).
- Burt, et al., “Enhanced Image Capture through Fusion”, Proceedings of International Conference on Computer Vision (ICCV), pp. 173-182 (1993).
- Chang, et al., “RGB Calibration for Color Image Analysis in Machine Vision”, IEEE Transactions Image Process., 5(10):1414-1422 (1996).
- Fujifilm, “Fujifilm Announces Super CCD EXR”, Press Release, Sep. 22, 2008, available at http://www.dpreview.com/news/2008/9/22/fujifilmEXR (downloaded on Feb. 13, 2013).
- Madden, “Extended Intensity Range Imaging”, Technical Report MS-CIS-93-96, Grasp Laboratory 366, University of Pennsylvania, (1993).
- Milgram, “Computer Methods for Creating Photomosiacs”, IEEE Transaction on Computers, 24(11):1113-1119 (1975).
- Nayar, et al., “High Dynamic Range Imaging: Spatially Varying Pixel Exposures”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1:472-479 (2006).
- Peleg, et al., “Panoramic Mosaics by Manifold Projection”, IEEE Computer Vision and Pattern Recognition (CVPR 97), pp. 338-343 (1997).
Type: Grant
Filed: May 10, 2011
Date of Patent: Dec 17, 2013
Assignees: The Trustees of Columbia University in the City of New York (New York, NY), Sony Corporation (Tokyo)
Inventors: Shree K. Nayar (New York, NY), Tomoo Mitsunaga (Kawasaki)
Primary Examiner: Aung S Moe
Assistant Examiner: Chriss Yoder, III
Application Number: 13/104,606
International Classification: H04N 5/228 (20060101);