HDR IMAGE SENSOR EMPLOYING MULTI-TAP PIXEL ARCHITECTURE AND METHOD FOR OPERATING SAME
An image sensor may include control circuitry, a plurality of pixels, and an image processor. Each pixel includes a photodetector, at least first and second storage nodes, and transfer circuitry. The transfer circuitry is responsive to control signals generated by the control circuitry to transfer first charges generated by the photodetector during a first exposure time within a frame period to the first storage node. Second charges may be generated by the photodetector during a second, longer exposure time during the frame period, and transferred to the second storage node. The image processor may generate image frame data based on output voltage samples derived from the first and second charges of each of the plurality of pixels.
The present disclosure relates generally to image sensors and more particularly to image sensors for generating high dynamic range (HDR) images.
DISCUSSION OF THE RELATED ARTIn an HDR image sensor, dynamic range is increased relative to traditional imaging through multi-sampling of local regions of a scene, so that brighter and darker portions of the scene are digitized separately. Short exposure times are used to obtain samples in brighter portions of a scene, and longer exposure times are used to sample darker portions of the scene. Each region of the scene (e.g., a region defined by a single pixel or a group of adjacent pixels) may be determined as either a bright region, a dark region or, in some HDR schemes, an intermediate luminance region. ND conversion with a fixed number of bits may be used to digitize a selected sample(s) of each region, resulting in a higher resolution image and higher overall dynamic range.
To obtain multiple samples for the local regions, current methods may take different exposures of the same scene by either: (1) sampling two or more consecutive frames, each at a different exposure; or (2) exposing groups of pixels within the same frame at different exposure periods. Since the scene is sampled in different time periods, these methods cause motion-artifacts when merging the multi-exposure data into a single HDR image. In case (2), resolution is also compromised.
Another technique known as a dual gain conversion obtains two samples from a pixel using the same exposure but with different conversion gains to obtain bright condition and dark condition samples. (A low conversion gain sample is used for a bright condition and a high conversion gain sample is used for a dark condition sample.) This scheme reduces motion-artifacts but suffers from inflexibility as well as accuracy limitations due to the difficultly in precisely controlling the relative gains for each pixel. Further, the dynamic range extension is limited by the dual gain ratio, which is particularly constrained for small pixel sensors.
SUMMARYIn an aspect of the inventive concept, an image sensor includes a control circuit, a plurality of pixels, and an image processor. Each pixel includes a photodetector, at least first and second storage nodes, and transfer circuitry. The transfer circuitry is responsive to control signals generated by the control circuitry to transfer first charges generated by the photodetector during a first exposure time within a frame period, to the first storage node. Second charges are generated by the photodetector during a second, longer exposure time during the frame period, and transferred to the second storage node. The image processor may generate image frame data based on output voltage samples derived from the first and second charges of each of the plurality of pixels.
The image frame data may be a high dynamic range (HDR) image frame.
The first exposure time may be an integrated exposure time of a sequence of first sub exposure times, and the second exposure time may be an integrated exposure time of a sequence of second sub exposure times interspersed with the first sub exposure times. In this manner, since the first and second output voltage samples each represent incident light received in time intervals distributed over approximately the same overall timeframe, motion artifacts that may otherwise be present in the generated image frame may be reduced.
In another aspect, an operating method of an image sensor including a plurality of multi-tap pixels is provided. The method involves, for each of the multi-tap pixels: (i) during an exposure frame period, activating a photodetector responsive to incoming light for a plurality K of exposure periods and thereby accumulate K respective photocharge packets, at least two of the K exposure periods being of different lengths; and (ii) transferring the photocharge packets to K storage nodes within the respective pixel. Voltage samples are obtained from the plurality of multi-tap pixels, derived from the photocharge packets in the K storage nodes. HDR image frame data is generated based on the voltage samples.
In another aspect, a non-transitory computer-readable recording medium stores instructions that, when executed by at least one processor of an image sensor comprising a plurality of multi-tap pixels, implements a method for generating HDR image frame data. The method includes, for each of the multi-tap pixels: (i) during an exposure frame period, activating a photodetector responsive to incoming light for a plurality K of exposure periods and thereby accumulate K respective photocharge packets, at least two of the K exposure periods being of different lengths, where each of the K exposure periods is an integrated exposure period of a respective one of K sequences of a plurality n of sub exposure periods, each of the K sequences being interspersed with each of the other of the K sequences; and (ii) transferring the photocharge packets to K storage nodes within the respective pixel. Correlated double sampling (CDS) voltage samples are obtained, based on the photocharge packets in the K storage nodes in each of the multi-tap pixels. The CDS voltage samples are digitized, and HDR image frame data is obtained from the digitized CDS voltage samples.
The above and other aspects and features of the inventive concept will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings in which like reference characters indicate like elements or features. Various elements of the same or similar type may be distinguished by annexing the reference label with an underscore and second label that distinguishes among the same/similar elements (e.g., _1, _2), or directly annexing the reference label with a second label (e.g., 120i, 120j). However, if a given description uses only the first reference label (e.g., 120), it is applicable to any one of the same/similar elements having the same first reference label irrespective of the second label. Elements and features may not be drawn to scale in the drawings.
The following description, with reference to the accompanying drawings, is provided to assist in a comprehensive understanding of certain exemplary embodiments of the inventive concept disclosed herein for illustrative purposes. The description includes various specific details to assist a person of ordinary skill the art with understanding the inventive concept, but these details are to be regarded as merely illustrative. For the purposes of simplicity and clarity, descriptions of well-known functions and constructions may be omitted when their inclusion may obscure appreciation of the inventive concept by a person of ordinary skill in the art.
In embodiments of an image sensor and method for driving the same described below, the sensor includes a pixel array in which each pixel may be a multi-tap pixel having a single photodetector and multiple “taps” from the photodetector to respective storage nodes, e.g., floating diffusion (FD) nodes. Each tap includes transfer circuitry to transfer photocharge generated by the photodetector during an image frame to a different storage node. At least two of the taps transfer photocharge generated under different exposure time durations. Voltage produced at each storage node due to the transferred photocharge during a frame is read out as an image sample at the end of the frame. In this manner, multiple samples corresponding to at least long and short exposures (and in some cases, one or more medium exposures) are provided to an image processor each frame, enabling the image processor to generate HDR image data using a selected one of the samples per pixel for each frame. Further, to reduce motion artifacts due to motion within a frame period, each exposure time may be an integrated exposure time of a sequence of segregated sub exposure times. The sub exposure times of one integrated exposure time may be interspersed with the sub exposure times of other integrated exposure times. In this manner, since the storage node voltages each represent incident light received in time intervals distributed over approximately the same overall timeframe, motion artifacts that may otherwise be present in the generated image frame are reduced.
Control circuit 140 may cooperate with image processor 150 to provide control signals CNT to pixel array 110 to control transistor switches/gates within pixels 120. (Herein, the term “gate” may be used interchangeably with “transistor” when the transistor operates primarily as a switch and/or as a charge storage element.) The switches may be controlled so that each pixel 120 provides output voltage samples representing incident light upon the pixel 120. The control signals CNT control transfer circuitry to transfer photocharges (hereafter, “charges”) from the photodetector generated during the respective exposure times. The charges are transferred to a plurality of respective storage nodes to provide multiple voltage samples for each pixel every frame, where at least two samples correspond to different respective exposure times. As explained in detail below, each exposure time may be an integrated exposure time of a plurality of alternating sub exposure times, to reduce motion artifacts. Motion artifacts may be reduced by configuring each integrated exposure time (corresponding to one sample) to represent incident light distributed over approximately the same overall timeframe. The control signals CNT may also control outputting of the integrated voltages during a read-out period of the frame, and may control resetting of the storage nodes before and after the read-out for correlated double sampling (CDS) based noise reduction. Control circuit 140 may include a processor that executes instructions to at least partially carry out the control circuit's operations; the instructions may be read from an internal memory of control circuit 140, or from memory 170.
Control circuit 140 may also control operations of read-out circuit 130 by providing timing and control signals CTR to synchronize voltage read-out from the pixels of one row at a time, sequentially from column to column. For instance, pixel array 110 may have M columns of pixels 120 and M corresponding column line sets 1801 to 180M, where each column line set 180i (i=any integer from 1 to M) includes one or more column lines. (In
Read-out circuit 130 may include correlated double sampling (CDS) circuitry to cancel noise during the readout of the storage node voltages (which are analog voltages). Read-out circuit 130 may further include an analog to digital converter (ADC) to convert the noise reduced output voltages from the CDS circuitry to digital signals having corresponding digital values. Image processor 150 processes the digital signals to generate one or more HDR frames, depending on whether the application is for a still photo (single frame, in some cases) or video (composed of multiple frames). To this end, image processor 150 may combine short exposure samples from some pixels 120 of pixel array 110 and long exposure samples from other pixels 120 (and one or more medium exposure samples, if available) to generate a composite HDR image data frame. Short exposure samples for the HDR image are selected for pixels 120 determined to have received light within a high luminance range. Long exposure samples are selected for pixels determined to have received light within a low luminance range. Medium exposure samples may be selected for pixels determined to have received light in a medium luminance range between the low and high luminance ranges. By providing such n-bit data over two or more ranges of luminance rather than for the entire range of luminance, resolution enhancement and dynamic range expansion are achieved. Image processor 150 may execute instructions read from memory 170 to perform its operations. Image processor 150 may further output interim data as well as final HDR image data DI for storage in memory 170 and/or output the final HDR image data DI to a display system or a network. Image processor 150 may also execute a real time auto-exposure algorithm, discussed later, which may be used to dynamically control exposure times based on a measure of the illuminance environment.
Control signals CNT provided from control circuit 140 may include photogate signals PG (e.g., PGA, PGB, PGC and PGD discussed in an example below), transfer gate signals TG, storage gate signals SG, reset signals RG, select signals SEL and an overflow gate signal OG. Each photogate signal PG may be applied to a respective photogate of photodetector 210, where the photogate enhances PD 210 and guides photo-generated electrons (photocharge) collected by PD 210 into a respective storage area/storage node. Each signal PG may be a pulse or sequence of pulses that initiates a “tap” to transfer photocharge generated by PD 210. For example, the number of photogates of photodetector 210 may equal the number of storage nodes FD. In the tap mechanism, during the time that the pulse(s) PG is applied, photocharge may be transferred to a respective one of storage areas within transfer circuitry 220, and subsequently transferred to storage nodes FDA-FDD. The charge transfer may be controlled by transfer gate signals TG and storage gate signals SG. The time during which charges are generated in correspondence with a signal PG may be denoted an exposure time associated with the storage node FD to which the charge is subsequently transferred. When PG pulses are provided in a sequence during a single frame, charge may be built up in a storage area gradually over a sequence of sub exposure times, and then transferred from the storage area to a storage node FD. In this case, the exposure time associated with the storage node FD may be denoted an “integrated exposure time”.
Select signals SEL applied to output circuit 240 may be provided at the end of the exposure times during a frame to select one of output voltages VOUTA-VOUTD of the storage nodes FDA-FDD, respectively, for data sample read out. If correlated double sampling (CDS) is implemented, a noise sample at an FD node may be read out as VOUT just prior to the transfer of charge to that FD node. The noise sample may then be subtracted from the VOUT sample taken after the charge transfer to the FD node by CDS circuitry within readout circuit 130. One or more column lines of a column line set 180i connects to pixel 120i to route voltages VOUTA-VOUTD to read-out circuit 130. After read-out (and/or just before noise sample read-out), reset signals RG may applied to reset circuit 230 to reset the voltages at storage nodes FDA-FDD to initial values. Overflow gate signal OG is applied to photodetector 210 to discharge its parasitic capacitance as desired.
The K photocharge packets may be transferred to K respective storage nodes, e.g., FD nodes, in each pixel, in response to control signals (e.g., TG, SG) applied to the pixel (S304). In embodiments employing CDS/global shuttering, the transfer of the photocharge packets to the storage nodes may be a multi-stage process in which the packets are transferred to an interim storage area (e.g., an active area of an SGX transistor, discussed below) during an interim stage. In other cases, the photocharge packets are transferred directly from the photodetectors to the storage nodes. In either case, HDR image frame data may be generated based on digital data representing output voltage samples derived from the photocharge packets in the K storage nodes (S306). This operation may involve amplifying, by the output circuitry 240, a noise sample at the storage node, and a signal sample (“signal plus noise”) at the storage node, at different times, to generate a pair of output voltage samples. The dual sampling technique allows CDS circuitry to subtract the noise from the signal sample to provide a more accurate luminance sample. The noise from signal subtraction may be performed in either the analog or digital domain. A digital sample representing the noise-reduced voltage is applied to image processor 150, which then generates the HDR image data.
Referring collectively to
Photodetector 410 may include a photodiode 401 or other photosensitive element, and photogate field effect transistors (“photogates”) PGXA, PGXB, PGXC and PGXD. Photocharge packets generated in response to incident light upon PD 410 may be transferred by photogates PGXA-PGXD to respective storage areas SD during exposure times at which photogate signals PGA-PGD are respectively applied to the gate terminals of the photogates. One or more overflow transistors OGX, part of overflow circuitry 250 of
Transfer circuitry 420 (an example of the transfer circuitry 220 of
Storage node FDA may be designated for storing a voltage VFDA produced from a short exposure tap “S-tap”. As mentioned, a tap may be understood as a mechanism by which charge flows from the photodetector down a path to a storage node of the pixel and is represented by a signal (voltage/current) at an output node of the pixel for data readout. The term “tap” may also be used herein to refer to the circuit elements of the circuit path within the pixel that implement the charge flow and provide the output signal. A multi-tap pixel may have a configuration allowing charge to flow from the photodetector through multiple paths during the same frame period.
An S-tap in the example of
In the example of
The exposure time for the S-tap may be controlled by photogate signal PGA. As shown in
Accordingly, an exposure integration time EITS for the S-tap may be a sum of the sequence of n sub exposure times ETS, i.e.,
EITS=n×ETS.
Similarly, the first medium exposure tap M1-Tap in this example is controlled by photogate signal PGB applied as a sequence of n pulses Pb1-Pbn, each having a pulse width ETM1, referred to as a first medium sub exposure time. The first pulse Pb1 of the sequence may begin at time t1 coinciding with the falling edge of pulse Pa1, or shortly thereafter. The second medium exposure tap “M2-Tap” may be controlled by photogate signal PGC which includes a sequence of n pulses Pc1-Pcn, each having a width ETM2, referred to as a second medium sub exposure time. Each of pulses Pc1-Pcn may directly succeed a corresponding one of pulses Pb1-Pbn.
The long exposure tap “L-Tap” is controlled by photogate signal PGD, which may be applied as a sequence of n pulses Pd1-Pdn, each having a width ETL and each directly succeeding a corresponding one of pulses Pc1-Pcn. As illustrated in
Accordingly, the long sub exposure times ETL are interspersed with the short sub exposure times ETS during the frame, and are also interspersed with the first medium sub exposure times ETM1 and the second medium sub exposure times ETM2. Exposure integration times EITM1, EITM2 and EITL for the respective taps may be as follows:
EITM1=n×ETM1;
EITM2=n×ETM2;
EITL=n×ETL.
In one embodiment, the exposure integration times EITM1 and EITM2 are equal, and only a single medium exposure sample is used by image processor 150 (when the sample is within the corresponding medium luminance range) to generate the HDR image data for the pixel 120i. In this case, the single medium exposure sample may be an average of the samples VOUTB and VOUTC for the M1-Tap and M2-Tap, respectively. By averaging two samples in this manner, the signal to noise ratio (SNR) for the medium exposure sample may be improved. Alternatively, the exposure integration times EITM1 and EITM2 are unequal, and samples are provided for a total of four exposure ranges per pixel. This scheme may result in still higher resolution in the finally processed HDR image.
With continuing reference to
After the last sub exposure pulse Pdn, a read-out period may be designated during which one or more select signals, e.g., SEL1 and SEL2, may be applied to the output circuit 240 (
The integrated exposure times EITS . . . EITL and sub exposure times ETS ETL and number of sub exposure times per frame in
The example of
Further, in any of the above cases (including the embodiment of
Referring still to
A transfer period 511 may then follow at time t3 and extend to time t4, during which the second shuttle transistors TGX2 are turned ON by pulses 511 of shuttle gate signals TG2 applied to their gates. This transfers the photocharge packets (“signals”) stored by storage transistors SGX at storage areas SDA-SDD to the respective storage nodes FDA-FDD, causing the FD storage node voltages VFDA-VFDD to increase to levels defined by “signal plus noise”. Another set of SEL pulses, PS1−2 and PS2−2, may then be applied in succession between times t4 and t5, whereby the FD storage node voltages are amplified to produce output voltages VOUTA-VOUTD, respectively, representing the photocharge generated in the corresponding exposure integration times, plus noise. These “signal plus noise” voltages are sampled by readout circuit 130, which subtracts the level of the noise sample just taken to remove the noise, completing a correlated double sampling operation to obtain a more accurate luminance sample. For global shutter operations, a global reset period may follow at time t5 and extend to the end of the frame period. In the global reset period, both the RG signals and TG2 signals may transition high, so that residual charge is drained from both the SD areas and the FD nodes. Additionally, the OG pulse POGN may continue high from time t2 to the end of the global reset period, to continually drain unwanted photocharge from photodiode 410. (Note that rolling shutter operations are alternatively configurable, in which case the readout/resetting period may not be synchronized for each of the rows of the image sensor array.)
The pixel circuit design of
Exemplary embodiments of the inventive concept have been described herein with reference to signal arrows, block diagrams and algorithmic expressions. Each block of the block diagrams, and combinations of blocks in the block diagrams, and operations according to the algorithmic expressions can be implemented by hardware (e.g. logic/processing/control circuitry within image processor 150, control circuit 140 and/or readout circuit 130) accompanied by computer program instructions. Such computer program instructions may be stored in a non-transitory computer readable medium (e.g. memory 170 or an internal memory of control circuit 140) that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the computer readable medium is an article of manufacture including instructions which, when executed, implement the function/act specified in the flowchart and/or block/schematic diagram. The term “processor” as used herein is intended to include any processing device, such as, for example, one that includes a central processing unit (CPU) and/or other processing circuitry (e.g., digital signal processor (DSP), microprocessor, etc.). Moreover, a “processor” includes computational hardware and may refer to a multi-core processor that contains multiple processing cores in a computing device. Herein, the term “circuit” is used to denote either a stand-alone circuit or a circuit part (“circuitry”) of a higher level circuit.
While the inventive concepts described herein have been particularly shown and described with reference to example embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the claimed subject matter as defined by the following claims and their equivalents.
Claims
1. An image sensor comprising:
- control circuitry to generate control signals;
- a plurality of pixels each comprising: a photodetector that receives incident light; first, second, third and fourth storage nodes; and transfer circuitry responsive to the control signals to transfer first charges, second charges, third charges, and fourth charges generated by the photodetector during respective first, second, third and fourth exposure times within a frame period to the first, second, third and fourth storage nodes, respectively, the first exposure time being less than the second through fourth exposure times, the second and third exposure times being equal and less than the fourth exposure time, the first, second, third and fourth exposure times being respective integrated exposure times of sequences of first, second, third and fourth sub exposure times, respectively, the sequences being interspersed with one another during the frame period;
- and
- an image processor to generate image frame data based on output voltage samples derived from the first, second, third and fourth charges of each of the plurality of pixels, to generate high dynamic range (HDR) data of a high luminance range and a low luminance range based on the output voltage samples corresponding to the first charges and the fourth charges, respectively, and to average luminance data of output voltage samples corresponding to the second and third charges to generate HDR data of a single intermediate luminance range.
2. (canceled)
3. The image sensor of claim 1, wherein the first through fourth sub exposure times are each repetitive time intervals recurring at the same frequency within the frame period.
4. (canceled)
5. The image sensor of claim 1, further comprising a readout circuit coupled between the plurality of pixels and the image processor, to read out the output voltage samples during an end portion of the frame period; and
- wherein each said pixel comprises resetting circuitry to reset voltages at each of the first and second nodes following read-out of the output voltage samples.
6. The image sensor of claim 5, wherein the readout circuit comprises a correlated double sampling circuit and an analog to digital converter (ADC).
7. The image sensor of claim 1, wherein the photodetector comprises a photodiode and first through fourth photogates through which the first through fourth charges are transferred, respectively.
8. The image sensor of claim 1, wherein each of the plurality of pixels further comprises at least one overflow gate transistor, responsive to one of the control signals to discharge capacitance of the photodetector during a discharge time period within the frame period.
9. The image sensor of claim 8, wherein at least a portion of the discharge time period is within an end portion of the frame period following the first and second exposure times.
10. The image sensor of claim 9, wherein:
- a further portion of the discharge time period comprises a plurality of discharge time sub periods, each between a set of adjacent first through fourth sub exposure times and a succeeding set of adjacent first through fourth sub exposure times.
11-14. (canceled)
15. The image sensor of claim 1, wherein the image processor executes instructions to run a real time auto exposure algorithm and cooperates with the control circuitry to set the first through fourth exposure times in accordance with a result of the auto exposure algorithm.
16. The image sensor of claim 1, wherein the first through fourth storage nodes are each floating diffusion (FD) nodes, and the transfer circuitry further comprises a first storage transistor coupled between the photodetector and the first storage node in a first tap, a second storage transistor coupled between the photodetector and the second storage node in a second tap, a third storage transistor coupled between the photodetector and the third storage node in a third tap, a fourth storage transistor coupled between the photodetector and the fourth storage node in a fourth tap, each of the first through fourth storage transistors including storage areas for interim storage of the first through fourth charges, respectively.
17. An operating method of an image sensor comprising a plurality of multi-tap pixels and at least one overflow gate transistor, the method comprising:
- for each of the multi-tap pixels: during an exposure frame period, activating a photodetector responsive to incoming light for a plurality K of exposure periods and thereby accumulate K respective photocharge packets, at least two of the K exposure periods being of different lengths, wherein each of the K exposure periods is an integrated exposure period of a respective one of K sequences of a plurality n of sub exposure periods, each of the K sequences being interspersed with each of the other of the K sequences; transferring the photocharge packets to K storage nodes within the respective pixel; applying control signals to the at least one overflow gate transistor to discharge capacitance of the photodetector during a discharge time period within the frame period, wherein one portion of the discharge time period is within an end portion of the frame period following the K exposure periods, and a further portion of the discharge time period comprises a plurality of discharge time sub periods, each between a set of adjacent first through K sub exposure times and a succeeding set of adjacent first through K sub exposure times;
- obtaining voltage samples from the plurality of multi-tap pixels based on the photocharge packets in the K storage nodes of each of the multi-tap pixels; and
- generating high dynamic range (HDR) image frame data based on the voltage samples.
18. (canceled)
19. The operating method of claim 17, wherein said transferring the photocharge packets to K storage nodes comprises first transferring the photocharge packets from the photodetector to K storage areas of K respective storage transistors, and subsequently transferring the photocharge packets from the K storage areas to the K storage nodes following noise sampling of the K storage nodes.
20. A non-transitory computer-readable recording medium storing instructions that, when executed by at least one processor of an image sensor comprising a plurality of multi-tap pixels, implement a method that comprises:
- for each of the multi-tap pixels: during an exposure frame period, activating a photodetector responsive to incoming light for first through fourth exposure times and thereby accumulating first through fourth respective photocharge packets, the first exposure time being less than the second through fourth exposure times, the second and third exposure times being equal and less than the fourth exposure time, each of the first through fourth exposure times being an integrated exposure time of a respective one of first through fourth sequences of a plurality n of sub exposure times, each of the first through fourth sequences being interspersed with each of the other of the first through fourth sequences; transferring the first through fourth photocharge packets to first through fourth-storage nodes, respectively, within the respective pixel; obtaining correlated double sampling (CDS) voltage samples based on the photocharge packets in the first through fourth storage nodes of each of the plurality of multi-tap pixels; digitizing the CDS voltage samples; and generating high dynamic range (HDR) image frame data from the digitized CDS voltage samples, which comprises, for each of the multi-tap pixels, generating HDR pixel data of a high luminance range and a low luminance range based on the CDS voltage samples corresponding to the first photocharge packets and the fourth photocharge packets, respectively, and averaging luminance data of the CDS voltage samples corresponding to the second and third photocharge packets to generate HDR data of a single intermediate luminance range.
21-23. (canceled)
Type: Application
Filed: Apr 22, 2021
Publication Date: Oct 27, 2022
Inventors: Roee Elizov (Herzliya), Yoel Yaffe (Modiin), Amit Eisenberg (Kiryat Ono), Shy Hamami (Ganey-Tikva)
Application Number: 17/237,631