IMAGE PROCESSING APPARATUS, IMAGE PICKUP APPARATUS HAVING THE SAME, AND IMAGE PROCESSING METHOD
An image processing apparatus for processing image signals that are obtained from an image pickup element. A correlation operation unit operates a correlation of the image signals of two images that are obtained from the image pickup element, and operates an image shift amount between images that are picked up by photoelectric conversion units, on the basis of the correlation. A detection unit detects whether or not an image signal, of the two images, reaches a predetermined level, and produces count information in a case that the image signal reaches the predetermined level. A controller controls the correlation operation unit, in accordance with the count information.
This application is a divisional application of copending U.S. patent application Ser. No. 13/859,174, filed Apr. 9, 2013, and published as U.S. Patent Application Publication No. 2013/0286275 on Oct. 21, 2013.
This application also claims the benefit of Japanese Patent Application No. 2012-102880, filed on Apr. 27, 2012, which is hereby incorporated by reference herein in its entirety.
BACKGROUND OF THE INVENTION1. Field of the Invention
The present invention relates to an image processing apparatus and, more particularly, to an image processing apparatus for performing a focus detection using a phase difference between two obtained pupil-divided images.
2. Description of the Related Art
In the related art, such a technique that pupil-divided images are obtained by an image pickup element in which a plurality of photoelectric conversion units correspond to one microlens, and a phase difference between the two obtained pupil-divided images is obtained, to perform a focus detection has been known.
For example, Japanese Patent Application Laid-Open No. 2001-83407 discloses such a technique that a phase difference is obtained from pupil-divided images, to perform a focus detection, and at the same time, all signals of photoelectric conversion units corresponding to a same microlens are added together and output as a signal of one pixel. Japanese Patent Application Laid-Open No. 2001-83407 discloses such a technique that by handling the image signals in a manner similar to that mentioned in the latter technique, the image signals are made substantially identical to the image signals obtained by an array of image pickup pixels in the related art and an image for appreciation is formed by an image processing technique in the conventional art.
If any one of the divided photoelectric conversion units has reached a saturation, even when the outputs of all of the photoelectric conversion units corresponding to the same microlens are added up, an added output does not exhibit a linear characteristic due to an influence of the saturation, so that a picture quality deteriorates.
However, according to the related art disclosed in the above Patent Literature, although a permissible saturation level in an image which is finally obtained is improved and the picture quality is improved, a saturation level of a pupil-separated image before the addition is not improved. Rather than that, a deterioration of an image shape is caused by an influence of the pixel which absorbed charge leaked out from a saturated pixel, and a correlation of images at the time of obtaining a phase difference deteriorates.
That is, an adverse influence occurs in the focus detection, and then a focus detection precision deteriorates or the apparatus enters a state where the focus detection is impossible.
Since it may be resulted in that the focus detection is impossible while a good image has been obtained, such a situation that a state where the focus detection can be performed and a state where the focus detection is impossible cannot be distinguished appears, and this may confuse the user.
SUMMARY OF THE INVENTIONIt is an aspect of the invention to provide an image processing apparatus and an image processing method which can attain to correctly obtain an image shift amount even in the case where a deterioration of an image shape due to saturation occurred in pupil-divided images.
To accomplish the above aspect of the invention, an image processing apparatus of the invention for processing image signals which are obtained from an image pickup element, comprises: a correlation operation unit configured to operate a correlation of the image signals of two images which are obtained from the image pickup element and operate an image shift amount between images picked-up by photoelectric conversion units on the basis of the correlation; a saturation detection unit configured to detect whether or not the image signal reaches a saturation level; and a controller configured to control the correlation operation unit, wherein the controller controls whether or not the correlation of the image signals from a plurality of photoelectric conversion units including the above-described photoelectric conversion units is used for the operation of the image shift amount which is performed by the correlation operation unit, and the control by the controller is made in accordance with a result of the saturation detection of the photoelectric conversion unit which is performed by the saturation detection unit.
Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the invention and, together with the description, serve to explain the principles of the invention.
Exemplary embodiments of the invention will be described in detail hereinbelow with reference to the drawings.
First, a construction of an image pickup apparatus according to the embodiment of the invention will be described.
In the diagram, a microlens 101 for converging light onto the pixel is provided and a color filter 102 is a filter generally constructed in such a manner that respective color filters of a plurality of colors such as RGB or the like are arranged at a predetermined period.
A wiring layer 103 of a semiconductor and photodiodes 104 and 105 for performing a photoelectric conversion are provided.
Although the photodiodes 104 and 105 are provided as one photoelectric conversion unit in the ordinary image pickup element, in a pixel structure serving as a prerequisite of the invention, pupil-divided images are obtained by dividing the photoelectric conversion unit. That is, in that pixel structure, two photoelectric conversion units correspond to one microlens.
In an image (image A) formed by a photoelectric conversion signal (image signal) only from the photoelectric conversion unit 104 and an image (image B) formed by an image signal only from the photoelectric conversion unit 105, a parallax occurs in both of those images since a pupil has been divided. The images having the parallax therebetween can be used in a field where a focus detection is performed, a stereo image is obtained, or the like.
Since a same pupil shape as that of a general pixel is re-formed by adding the image signal of the photoelectric conversion unit 105 (B-image pixel) and the image signal of the photoelectric conversion unit 104 (A-image pixel), if a general signal processing is executed to the added image signal, an image similar to that in the image pickup element in the conventional art can be obtained.
The problem of the saturation of the photoelectric conversion units will now be described with reference to
However, the light which is irradiated to the pixel is not uniform in a photosensitive surface thereof unless the pixel is positioned at the center of image height and is in an in-focus state.
A zigzag line 203 in
As a countermeasure for avoiding such an influence, there is a method of using the following pixel structure.
That is, the pixel is constructed such that if the photoelectric conversion unit 105 reaches the saturation, the charge generated there leaks into the photoelectric conversion unit 104. Thus, the signal obtained by adding the image signals of the photoelectric conversion units 105 and 104 exhibits the characteristic as shown by the zigzag line 201 in
According to the curve 205, since the photoelectric conversion unit 105 (characteristic 203) reaches the saturation and the charge therefrom flows into the photoelectric conversion unit 104, an inclination of the output to the incident light rises in the zigzag line 205.
Further, since there exists a parallax between the images A and B in accordance with a defocus amount, their levels differ largely in a blurred (defocused) image. However, since the saturated charge leaks, both of the images A and B are saturated in a highlight portion.
A defocus amount can be calculated by multiplying an image shift amount between the images A and B by a constant which is determined by a base line length. Since there is the image shift between the images A and B, the defocus amount is to be obtained. However, the image is not deviated in the portion where the saturation occurrs. Since an influence degree in this instance is large, a coincidence degree is largest at the foregoing position where no image shift occurs between the images A and B.
That is, since it is determined as a result of the correlation operation that no image shift occurs between the images A and B, it is erroneously decided that the pixel is in an in-focus state.
The correlation image is an image in which the images A and B are shifted little by little and a correlation amount obtained in accordance with a shift amount is set to an image. As a correlation amount, generally, a sum of the absolute values of the differences called SAD (Sum of Absolute Difference), a sum of the squares of the absolute values of the differences called SSD (Sum of Squared Difference), or the like is used as in index.
A position of a scale 0 at an axis of abscissa in
As mentioned above, when the charge leak caused by the saturation occurs, such a phenomenon that the correlation amount is largely affected by the defocus amount 0 caused by the charge leak irrespective of an actual defocus amount occurs.
First EmbodimentA case where an image processing apparatus according to the first embodiment of the invention is applied to the image pickup apparatus having the image pickup element constructed by the pixel illustrated in
In the diagram, a lens 501, an image pickup element 502, an A/D converter 503, an A and B-image addition circuit 504, an A and B-image separation circuit 505, a signal processing circuit 506, a correlation operation circuit 507, and a defocus amount calculation unit 508 are provided. Those component elements are controlled by a micro computer 509 for controlling the whole system.
A pixel of the image pickup element 502 has the structure as described by referring to
As mentioned above, by forming one pixel by combining the A-image pixel and the B-image pixel, a Bayer array of RGB is obtained.
Since A-image data and B-image data are sequentially output from the A/D converter 503, if adjacent pixel data of the same color is added by the A and B-image addition circuit 504, the pixels are returned to the Bayer array.
The signal processing circuit 506 forms a color image signal from the Bayer array.
The A-image data and B-image data are separated by the A and B-image separation circuit 505 and transferred to the correlation operation unit 507.
A shift amount between the images A and B is calculated as a correlation image by the correlation operation unit 507 and output to the defocus amount calculation unit 508.
In the defocus amount calculation unit 508, an image shift amount is obtained by analyzing the correlation image and a+defocus amount is obtained by multiplying the image shift amount by a constant which is determined by the base line length of the images A and B. The control micro computer 509 controls the lens 501 in accordance with the defocus amount so as to obtain the in-focus state.
A saturation threshold level 601 is a set value which is given so as to be changeable by the control micro computer 509. A level at which the saturation is started as a characteristic of a sensor is designated as a threshold value.
An A-image data input 602 and a B-image data input 603 are provided. The A-image data and the B-image data stored in a buffer at a front stage of the correlation operation unit are input to the input terminals 602 and 603 repeatedly while changing the shift amount one by one. An absolute value of a difference between the A-image data and the B-image data is calculated in a difference absolute value operation unit 605. A sum of the absolute values of the differences is obtained by an adding-up unit 608.
At a point of time when the inputs 602 and 603 of the A-image data and the B-image data have been finished for one shift amount, an SAD output (adding-up output) 610 is fetched at a post stage and the adding-up unit 608 is reset. The shift amount is changed and the A-image data and the B-image data are input from the beginning After the input of the image is finished, the SAD output 610 is fetched. By repeating the same operation as that mentioned above many times after that, a correlation image is obtained. In a saturation detection unit 604, which one of the A-image pixel and the B-image pixel reaches the saturation is detected by comparing with the threshold value 601.
When saturation detection unit 604 detects the saturation, a switch 606 is turned off. In a processing cycle in which the saturation is detected, the difference absolute value is not fetched into the adding-up unit 608. Thus, the influence of the saturation can be excluded from the correlation amount output 610. That is, whether or not the correlation of the image signals is used for the operation of the image shift amount can be controlled in accordance with a result of the saturation detection of the saturation detection unit 604.
In a counter circuit 607, a count value is incremented by one in a cycle in which the output of the saturation detection unit 604 is equal to 1. Thus, the number of times of saturation-caused exclusion (exclusion frequency) is counted by the counter circuit 607. A saturation-caused exclusion image can be obtained by fetching a saturation-caused exclusion counter value 609 in a cycle for fetching the output of the correlation amount of the SAD output 610.
In the diagram, a curve 801 indicates the correlation image according to the correlation amount added-up in the absolute value adding-up unit in
Normalized correlation amount=correlation amount*the number of field-of-view pixels/(the number of field-of-view pixels−saturation-caused exclusion counter value)
A weight is corrected by an amount in which the addition of the difference absolute value is excluded by the saturation.
The normalization is performed from the correlation image and the saturation-caused exclusion counter image by the defocus amount calculation unit 508, thereby obtaining the normalized correlation image as shown by the dotted line 802.
After that, in the defocus amount calculation unit 508, a sub-pixel matching is performed by a method called parabola fitting.
In the parabola fitting, since a sub-pixel matching position is calculated by using inclinations of the images before and after the minimum value of the correlation amount (for example, values at four points of the different shift amounts before and after the minimum value), a precision of the sub-pixel matching is improved by the normalization.
When a ratio occupied by the saturation pixels is large, unless the normalization is performed, the minimum value itself is disordered and the shift amount cannot be correctly calculated. Therefore, the correlation image is certainly normalized and used.
Second EmbodimentSubsequently, an image processing apparatus according to the second embodiment of the invention will be described with reference to
Although the correlation operation unit 507 is constructed by the operation unit and the like of the difference absolute value adding-up unit illustrated in
First, when the correlation operation is started in step S1001, a repetition of the correlation amount adding-up operation is started for the shift amount (it is assumed to be −100 to +100) in step S1002.
A repetition for the number of pixels (field-of-view length) of the field-of-view of the image A is started in step S1003.
The pixels at the B-image pixel position shifted by the shift amount from the A-image pixel position are fetched in step S1004, respectively.
Whether or not the A-image pixel or the B-image pixel is saturated is discriminated in step S1005. If it is saturated, the processing routine advances to step S1008, the count value of the exclusion frequency counter is incremented, and step S1010 follows.
If each of the A-image pixel and the B-image pixel is not saturated in step S1005, step S1007 follows. The absolute value of the difference between the A-image pixel and the B-image pixel is added-up as an SAD value and step S1010 follows.
The pixel positions of the images A and B are incremented by one (+1) (shifted by a distance of one pixel) in step S1010 and the processing routine is returned to step S1003.
After steps S1003 to S1010 are repeated for the field-of-view length (N in step S1003), step S1006 follows and the SAD value and the exclusion counter value are stored in a position corresponding to the shift amount in the array.
The shift amount is incremented by one (+1) (shifted by a distance of one pixel) in step S1009 and the processing routine is returned to step S1002.
By repeating steps S1002 to S1009 for the shift amount, a table of the correlation image for each shift amount and the exclusion counter is formed (N in step S1002) and step S1011 follows.
The normalization operation is executed in step S1011 and this processing routine is finished in step S1012.
The normalization is started in step S1011.
A repetition for the correlation image length (0 to 200) is started in step S1102.
The correlation amount and the exclusion counter are fetched from the formed table in step S1103.
The normalization is performed by multiplying the correlation amount by (field-of-view length (field-of-view length−exclusion counter value)) in step S1104.
A result of the normalization operation is stored into the table and a pointer is advanced one by one in step S1105 and the processing routine is returned to step S1102.
After steps S1102 to S1105 are repeated for the field-of-view length (N in step S1102), step S1106 follows and the processing routine is returned to the processing routine in
As mentioned above, a technical effect similar to that in the first embodiment can be also accomplished in the second embodiment. In the second embodiment, the functions of the processings shown in
However, the invention is not limited to the foregoing construction but all or a part of the functions of the processings shown in
The foregoing processings may be executed by a method whereby the program for realizing the functions of the processings shown in
“computer-readable storage medium” denotes a storage device such as flexible disk, magnetooptic disk, portable medium such as ROM, CD-ROM, or the like, hard disk which is built in the computer system, or the like. Further, it is assumed that “computer-readable storage medium” also incorporates a volatile memory (RAM) in the computer system serving as a server or a client in the case where the program has been transmitted through a network such as Internet or the like or a communication line such as a telephone line or the like. As mentioned above, the memory in which the program is held for a predetermined time is also incorporated in “computer-readable storage medium”.
The program may be transmitted from the computer system in which the program has been stored in the storage device or the like to another computer system through a transmission medium or by a transmission wave in the transmission medium. “transmission medium” for transmitting the program denotes a medium having a function for transmitting information like a network (communication network) such as Internet or the like or a communication line (communication wire) such as a telephone line or the like.
The foregoing program may be a program for realizing a part of the functions mentioned above. Further, the program may be what is called a differential file (differential program) which can realize the foregoing functions by a combination with the program which has already been recorded in the computer system.
A program product such as a computer-readable storage medium or the like in which the foregoing program has been recorded can be also applied as an embodiment of the invention. The foregoing program, storage medium, transmission medium, and program product are incorporated in the purview of the invention.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
Claims
1. An image processing apparatus for processing image signals that are obtained from an image pickup element, the image processing apparatus comprising:
- a correlation operation unit configured to operate a correlation of the image signals of two images that are obtained from the image pickup element, and to operate an image shift amount between images that are picked up by photoelectric conversion units, on the basis of the correlation;
- a detection unit configured to detect whether or not an image signal, of the two images, reaches a predetermined level, and to produce count information in a case that the image signal reaches the predetermined level; and
- a controller configured to control the correlation operation unit, in accordance with the count information.
2. The image processing apparatus according to claim 1, wherein the image pickup element is constructed such that the plurality of photoelectric conversion units are arranged correspondingly to a microlens.
3. The image processing apparatus according to claim 1, wherein the controller controls operation in such a manner that, when the two images are relatively shifted to operate the correlation, if the predetermined level of an image signal is detected, the correlation operation unit does not use the predetermined level of an image signal and a signal corresponding to the count information for the operation of the image shift amount.
4. The image processing apparatus according to claim 1, wherein the correlation operation unit normalizes a correlation result on the basis of the count information.
5. The image processing apparatus according to claim 1, wherein the correlation operation unit adds up a number of times of exclusion that the correlation of the image signals is not used for the operation of the image shift amount, in accordance with the control by the controller, based on the count information that is produced by the detection unit, and normalizes the correlation that is used for the operation of the image shift amount, on the basis of the added-up number of times of exclusion.
6. The image processing apparatus according to claim 5, wherein the correlation operation unit forms (i) a correlation image and (ii) an exclusion image corresponding thereto, from the correlation that is used for the operation of the image shift amount and the number of times of exclusion, and normalizes the correlation image on the basis of the exclusion image.
7. The image processing apparatus according to claim 1, wherein the detection unit detects whether or not an image signal, of the two images, reaches the predetermined level, on the basis of a result of a comparison between (i) a level of the image signal and (ii) a preset threshold value that is set by the controller.
8. An image pickup apparatus comprising:
- (a) an image pickup element in which a plurality of photoelectric conversion units are arranged in correspondence to a microlens; and
- (b) an image processing apparatus for processing image signals that are obtained from the image pickup element, the image processing apparatus including: (i) a correlation operation unit configured to operate a correlation of the image signals of two images that are obtained from the image pickup element, and to operate an image shift amount between images that are picked up by photoelectric conversion units, on the basis of the correlation; (ii) a detection unit configured to detect whether or not an image signal reaches a predetermined level, and to produce a count information in a case that the image signal reaches the predetermined level; and (iii) a controller configured to control the correlation operation unit, wherein the controller normalizes the correlation that is used for the operation of the image shift amount, on the basis of the count information.
9. The image pickup apparatus according to claim 8, further comprising:
- (c) an addition unit configured to add image signals from the plurality of photoelectric conversion units to produce an added signal; and
- (d) a signal processing unit configured to process the added signal.
10. The image pickup apparatus according to claim 8, wherein the controller controls operation in such a manner that, when the two images are relatively shifted to operate the correlation, if the predetermined level of an image signal is detected, the correlation operation unit does not use the redetermined level of an image signal and a signal corresponding to the count information for the operation of the image shift amount.
11. The image pickup apparatus according to claim 8, wherein the correlation operation unit normalizes a correlation result on the basis of the count information.
12. The image pickup apparatus according to claim 8, wherein the correlation operation unit adds up a number of times of exclusion that the correlation of the image signals is not used for the operation of the image shift amount, in accordance with the control by the controller, based on the count information that is produced by the detection unit, and normalizes the correlation that is used for the operation of the image shift amount, on the basis of the added-up number of times of exclusion.
13. The image pickup apparatus according to claim 12, wherein the correlation operation unit forms (i) a correlation image and (ii) an exclusion image corresponding thereto, from the correlation that is used for the operation of the image shift amount and the number of times of exclusion, and normalizes the correlation image on the basis of the exclusion image.
14. The image pickup apparatus according to claim 8, wherein the detection unit detects whether or not an image signal, of the two images, reaches the predetermined level, on the basis of a result of a comparison between (i) a level of the image signal and (ii) a preset threshold value that is set by the controller.
15. An image processing method for processing image signals that are obtained from an image pickup element, the image processing method comprising:
- a correlation operation step of operating a correlation of the image signals of two images that are obtained from the image pickup element and operating an image shift amount between images that are picked up by photoelectric conversion units, on the basis of the correlation;
- a detection step of performing a saturation detection to detect whether or not an image signal, of the two images, reaches a predetermined level, and producing count information in a case that the image signal reaches the predetermined level; and
- a control step of controlling the correlation operation step, in accordance with the count information.
16. The image processing method according to claim 15, wherein the image pickup element is constructed such that the plurality of photoelectric conversion units are arranged correspondingly to a microlens.
17. The image processing method according to claim 15, wherein the control step controls operation in such a manner that, when the two images are relatively shifted to operate the correlation, if the predetermined level of an image signal is detected, the correlation operation step does not use the predetermined level of an image signal and a signal corresponding to the count information for the operation of the image shift amount.
18. The image processing method according to claim 15, wherein the correlation operation step normalizes a correlation result on the basis of the count information.
19. The image processing method according to claim 15, wherein the correlation operation step adds up a number of times of exclusion that the correlation of the image signals is not used for the operation of the image shift amount, in accordance with the control by the control step, based on the count information that is produced in the detection step, and normalizes the correlation that is used for the operation of the image shift amount, on the basis of the added-up number of times of exclusion.
20. The image processing method according to claim 19, wherein the correlation operation step forms (i) a correlation image and (ii) an exclusion image corresponding thereto, from the correlation that is used for the operation of the image shift amount and the number of times of exclusion, and normalizes the correlation image on the basis of the exclusion image.
21. The image processing method according to claim 15, wherein the detection step detects whether or not an image signal, of the two images, reaches the predetermined level, on the basis of a result of a comparison between (i) a level of the image signal and (ii) a preset threshold value that is set in the control step.
Type: Application
Filed: Jul 16, 2015
Publication Date: Nov 5, 2015
Inventor: Takeshi Ogawa (Tokyo)
Application Number: 14/800,746