STRUCTURED LIGHT IMAGING SYSTEM
Structured light imaging method and systems are described. An imaging method generates a stream of light pulses, converts the stream after reflection by a scene to charge, stores charge converted during the light pulses to a first storage element, and stores charge converted between light pulses to a second storage element. A structured light image system includes an illumination source that generates a stream of light pulses and an image sensor. The image sensor includes a photodiode, first and second storage elements, first and second switches, and a controller that synchronizes the image sensor to the illumination source and actuates the first and second switches to couple the first storage element to the photodiode to store charge converted during the light pulses and to couple the second storage element to the photodiode to store charge converted between the light pulses.
Latest SEMICONDUCTOR COMPONENTS INDUSTRIES, LLC Patents:
- Self-aligned semiconductor device contacts having widths less than minimum lithography limits
- Semiconductor package with wettable flank and related methods
- Nonlinear, discrete time control of power factor correction power converter
- Propagation delay compensation and interpolation filter
- TRIMMING CIRCUIT FOR BANDGAP REFERENCES
This application claims priority of U.S. Provisional Patent Application No. 61/479,029, filed Apr. 26, 2011, which is incorporated fully herein by reference.
FIELDThe present invention is directed to imaging systems and, more particularly, structured light imaging systems.
BACKGROUNDStructured light imaging systems are commonly used for three-dimensional (3D) imaging. A structured light imaging system has two main parts: an illumination source and a sensing device including an array of pixels. The illumination source projects one or more light patterns onto an object being imaged and the pixels within the sensing device detect light reflected by the object. The detected light is then processed by the sensing device to generate a representation of the object.
The charge conversion transistor 58 has its gate connected to the storage node N and is connected between the array pixel supply voltage VAA and the row select transistor 60. The charge conversion transistor 58 converts the charge stored at the storage node N into an electrical output signal. The row select transistor 60 is controllable by a row select signal ROW for selectively outputting the output signal OUT from the charge conversion transistor 58. For each pixel 50, two output signals are conventionally generated, one being a reset signal Vrst generated after the storage node N is reset, the other being an image or photo signal Vsig generated after charges are transferred from the photosensor 52 to the storage node N.
Signals from the sensing device 200 are typically read out a row at a time using a column parallel readout architecture. The timing and control circuit 232 selects a particular row of pixels in the pixel array 230 by controlling the operation of a row addressing circuit 234 and row drivers 240. Signals stored in the selected row of pixels are provided to a readout circuit 242 in the manner described above. The signals are read twice from each of the columns and then read out sequentially or in parallel using a column addressing circuit 244. The pixel signals (Vrst, Vsig) corresponding to the reset pixel signal and image pixel signal are provided as outputs of the readout circuit 242, and are typically subtracted by a differential amplifier 260 in a correlated double sampling operation and the result digitized by an analog to digital converter 264 to provide a digital pixel signal. The digital pixel signals represent an image captured by pixel array 230. The digital pixel signals are processed in an image processing circuit 268 to produce an output image.
SUMMARY OF THE INVENTIONAn imaging method in accordance with one embodiment generates a stream of light pulses, converts the stream after reflection by a scene to charge, stores charge converted during the light pulses to a first storage element, and stores charge converted between light pulses to a second storage element. A structured light imaging system in accordance with one embodiment includes an illumination source that generates a stream of light pulses and an image sensor. The image sensor includes a photodiode, first and second storage elements, first and second switches, and a controller that synchronizes the image sensor to the illumination source and actuates the first and second switches to couple the first storage element to the photodiode to store charge converted during the light pulses and to couple the second storage element to the photodiode to store charge converted between the light pulses.
The invention is best understood from the following detailed description when read in connection with the accompanying drawings, with like elements having the same reference numerals. When a plurality of similar elements are present, a single reference numeral may be assigned to the plurality of similar elements with a small letter designation referring to specific elements. When referring to the elements collectively or to a non-specific one or more of the elements, the small letter designation may be dropped. The letter “n” may represent a non-specific number of elements. Also, lines without arrows connecting components may represent a bi-directional exchange between these components. This emphasizes that according to common practice, the various features of the drawings are not drawn to scale. On the contrary, the dimensions of the various features are arbitrarily expanded or reduced for clarity. Included in the drawings are the following figures:
In accordance with one use, the imaging system 300 generates a 3D representation of an object/3D scene 306. The illumination source 302 generates pulsed structured light 303, which is directed toward the object 306. The object reflects portions of the pulsed structured light 303 as reflected structured light 307. The sensing device 304, which is in sync with the illumination source 302, captures the reflected light 307 and generates a 3D representation of the object. The illustrated sensing device 304 includes an array of pixels 308 and each pixel includes multiple storage elements, which are discussed in further detail below. Through the use of multiple storage elements, the present invention facilitates the capture and storage of rapidly changing scenes.
In one embodiment, charge is accumulated in the photodiode 402 and then transferred to a storage element 406/410 (see, for example,
As indicated in the corresponding energy diagrams, applying a high signal level to the storage gate 708, signal SG, and a low signal level to the transfer switch, signal TX, allows charge to flow from the photodiode 702 to the storage element 704, but prevents the flow of charge to the floating diffusion region. Applying a low signal level to the storage gate 708 and a low signal level to the transfer switch isolates the charge stored in the storage element 704. Applying a low signal level to the storage gate 708 and a high signal level to the transfer switch allows charge to flow from the storage element to the floating diffusion region, but prevents the flow of charge from the photodiode 702 to the storage element 704.
The pixel 700 is coupled to conventional readout circuitry 752. The illustrated readout circuitry includes a row select transistor 790 and a voltage source, VAA. The row select transistor 790 is controlled by a row signal, ROW, to produce an output signal, OUT, from a selected row.
After the charge is accumulated in the first and second storage registers 704 and 706, the stored charge is read out of the storage registers and transferred to the floating diffusion regions 712/716 of the pixel during a Frame Valid period. In the illustrated embodiment, the charge is first read out of the first storage register 704 and, then, the charge is read out of the second storage register 706. Charge is read out of the first storage register by first applying a high row selection signal, ROW, to the row selection transistor 790 during the read out period, applying a reset signal, RST, pulse to the reset transistor 780 (which creates a Sample Reset pulse), and applying a transfer signal, TX1, pulse to the first transfer resistor 714. The readout circuitry 752 then reads the transferred charge during a Sample Signal period. A similar technique is applied to read out the charge stored in the second storage register.
As indicated in the corresponding energy diagrams, applying a high signal level to the storage gate 808, signal SG, allows charge to flow from the photodiode 802 to the storage element 804, but prevents the flow of charge to the floating diffusion region. Applying a mid-level signal to the storage gate 808 isolates the charge stored in the storage element 804. Applying a low signal level to the storage gate 808 allows charge to flow from the storage element to the floating diffusion region, but prevents the flow of charge from the photodiode 802 to the storage element 804.
The pixel 800 may be implemented using a circuit diagram such as the circuit diagram 750 and readout circuitry 752 described above with reference to
After the charge is accumulated in the first and second storage registers 804 and 806, the stored charge is read out of the storage registers and transferred to the floating diffusion regions 812/816 of the pixel during a Frame Valid period. In the illustrated embodiment, the charge is first read out of the first storage register 804 and, then, the charge is read out of the second storage register 806. Charge is read out of the first storage register by first applying a high row selection signal, ROW, to the row selection transistor 790 during the read out period, applying a reset signal, RST, pulse to the reset transistor 780 (which creates a Sample Reset pulse), and then applying a low pulse to the storage signal, SG1. The readout circuitry 752 then reads the transferred charge during a Sample Signal period. A similar technique is applied to read out the charge stored in the second storage register.
At step 1020, a stream of light pulses is generated. In one embodiment, the stream of light pulses is generated by an illumination source that generates a periodic stream of light pulses. The periodic stream of light pulses may have a rate of 5 kilo-Hertz. The stream of light pulses may be directed toward an object that is being imaged. In one embodiment, the stream of light pulses includes a single pattern of light. In other embodiments, the stream of light pulses include two or more distinct patterns. The patterns of light may include a pattern of random points or a zebra-stripe like pattern.
At step 1040, the source of the light pulses and an image sensor are synchronized. In one embodiment, the image sensor is synchronized to the illumination source using conventional techniques that will be understood by one of skill in the art from the description herein.
At step 1060, a reflection of the stream of light pulses is converted to charge. In one embodiment the stream of light pulses is reflected by an object that is being images/scanned and the reflected light is converted to charge by photodiodes within an array of pixels of the image sensor.
At step 1080, the converted charge is routed to storage elements. If the converted charge corresponds to a light pulse, processing proceeds at block 1090. If the converted charge corresponds to a period of time between light pulses, processing proceeds at block 1100. In one embodiment, the photodiode is coupled to the first storage element to transfer accumulated charge converted during the one or more pulses of light and is coupled to the second storage element to transfer accumulated charge converted between the two or more pulses of light. In another embodiment, the charge converted by the photodiode is streamed to the first storage element during conversion of the one or more pulses of light and streamed by the photodiode to the second storage element between conversion of the two or more pulses of light.
At step 1090, the converted charge corresponding to charge converted during one or more light pulses is stored in a first storage element of a pixel. At step 1100, the converted charge corresponding to charge converted between two or more light pulses is stored in a second storage element of a pixel. For example, charge converted from a first pulse may be stored to the first storage element, charge converted between the first pulse and a second pulse may be stored in a second storage element, charge converted from the second pulse may be stored to a third storage element, and charge converted between the second pulse and a third pulse may be stored to a fourth storage element.
In embodiments where each pixel includes more than two storage element, charge may be stored in different storage elements based on the pattern of the structured light and/or to enable simultaneous conversion and readout of previously converted/stored charge. For example, where the stream of light pulses includes pulse streams having different patterns (e.g., a pulse having a first pattern, another pulse having a second pattern, and another pulse having a third pattern), the storing step during the one or more pulses of light may include storing charge converted from at least one pulse having the first pattern to the first storage element, storing charge converted from at least one pulse having the second pattern to a third storage element, and storing charge converted from at least one pulse having the third pattern to a fourth storage element.
At step 1110, the system checks if additional cycles of the pulsed light are to be stored during an exposure period. If additional cycles are to be stored, processing proceeds at decision step 1080. If no more cycles are to be stored during the current exposure period, processing proceeds to step 1120.
At step 1120, charge is processed and read out of the storage elements. In an embodiment including four storage elements, charge stored in the first and second storage elements may be read out during the storage of charge in the third and/or fourth storage elements. Charge converted between light pulses may be subtracted from charge converted during light pulses. For example, the charge stored in a second storage element may be subtracted from charge stored in a first storage element, e.g., either at the pixel or column level, upon read out in the analog domain to further improve performance. In embodiments where each pixel includes additional storage element, charge in one storage element (e.g., a storage element for storing charge between light pulses may be subtracted from multiple storage elements (e.g., storage elements associated with pulses from different light patters. For example, if there are four storage elements, charge stored in a second storage element may be subtracted from the charge stored in a first storage element, the charge stored in a third storage element, and the charge stored in a fourth storage element.
Using conventional techniques that will be understood by one of skill in the art from the description herein, distance information may be obtained from the stored charge. In one example, the distortion of a projected pattern as imaged by the sensor can be used for an exact geometric reconstruction of the surface shape. By interleaving the captures of light and background frames and subtracting one from the other, the invention herein prevents degradation in depth resolution by rejecting the effects due to varying ambient illumination.
Although the invention is illustrated and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Rather, various modifications may be made in the details without departing from the invention.
Claims
1. A structured light imaging system comprising:
- an illumination source configured to generate a stream of light pulses; and
- an image sensor comprising; a photodiode that converts the stream after reflection by a scene to charge, a first storage element, a first switch coupled between the photodiode and the first storage element, a second storage element, a second switch coupled between the photodiode and the second storage element, and a controller coupled to the first and second switches, the controller configured to synchronize the image sensor to the illumination source, actuate the first switch to couple the first storage element to the photodiode to store charge converted during one or more of the light pulses, and actuate the second switch to couple the second storage element to the photodiode to store charge converted between one or more of the light pulses.
2. The system of claim 1, the image sensor further comprising:
- a readout circuit coupled to the controller;
- wherein the controller is further configured to transfer charge stored in the first and second storage elements to the readout circuit.
3. The system of claim 1, wherein the controller is further configured to subtract charge stored in the second storage element from charge stored in the first storage element.
4. The system of claim 1, the image sensor further comprising:
- a third storage element,
- a third switch coupled between the photodiode and the third storage element,
- a fourth storage element, and
- a fourth switch coupled between the photodiode and the fourth storage element, and
- wherein the controller is further coupled to the third and fourth switches.
5. The system of claim 4, wherein the stream of light pulses includes at least one pulse having a first pattern, at least one pulse having a second pattern, and at least one pulse having a third pattern, and wherein the controller is further configured to actuate the first switch to couple the first storage element to the photodiode to store charge converted from the at least one pulse having the first pattern, actuate the third switch to couple the third storage element to the photodiode to store charge converted from the at least one pulse having the second pattern, and actuate the fourth switch to couple the fourth storage element to the photodiode to store charge converted from the at least one pulse having the third pattern.
6. The system of claim 5, wherein the controller is further configured to subtract the charge stored in the second storage element from the charge stored in the first storage element, subtract the charge stored in the second storage element from the charge stored in the third storage element, and subtract the charge stored in the second storage element from the charge stored in the fourth storage element.
7. The system of claim 4, wherein the stream includes a first pulse, a second pulse adjacent the first pulse, and a third pulse adjacent the second pulse and wherein the controller is further configured to actuate the first switch to couple the photodiode to the first storage element to store charge converted from the first pulse, actuate the second switch to couple the photodiode to the second storage element to store charge converted between the first and second pulses, actuate the third switch to couple the photodiode to the third storage element to store charge converted from the second pulse, and actuated the fourth switch to couple the photodiode to the fourth storage element to store charge converted between the second and third pulses.
8. The system of claim 7, further comprising:
- a readout circuit coupled to the controller and the first, second, third, and fourth storage elements,
- wherein the controller is further configured to couple the readout circuit to the first and second storage elements when the photodiode is coupled to at least one of the third and fourth storage elements.
9. An image sensor for sensing a stream of light pulses from an illumination source as reflected by a scene, the sensor comprising:
- a plurality of pixels, each pixel comprising: a photodiode that converts the stream after reflection by a scene to charge; a first storage element; a first switch coupled between the photodiode and the first storage element; a second storage element; and a second switch coupled between the photodiode and the second storage element; and a controller coupled to the first and second switches, the controller configured to synchronize the image sensor to the illumination source, actuate the first switch to couple the first storage element to the photodiode to store charge converted during one or more of the light pulses, and actuate the second switch to couple the second storage element to the photodiode to store charge converted between one or more of the light pulses.
10. The sensor of claim 9, each pixel further comprising:
- a readout circuit coupled to the controller;
- wherein the controller is further configured to transfer charge stored in the first and second storage elements to the readout circuit.
11. The sensor of claim 9, each pixel further comprising:
- a third storage element,
- a third switch coupled between the photodiode and the third storage element,
- a fourth storage element, and
- a fourth switch coupled between the photodiode and the fourth storage element, and
- wherein the controller is further coupled to the third and fourth switches.
12. An imaging method comprising:
- generating a stream of light pulses;
- converting the stream after reflection by a scene to charge;
- storing charge converted during one or more of the light pulses to a first storage element; and
- storing charge converted between two or more of the light pulses to a second storage element.
13. The method of claim 12, wherein the generating step is performed by an illumination source and the converting and storing steps are performed by an image sensor, the method further comprising:
- synchronizing the image sensor to the illumination source.
14. The method of claim 12, further comprising;
- subtracting the charge stored in the second storage element from the charge stored in the first storage element.
15. The method of claim 12, wherein the stream of light pulses includes at least one pulse having a first pattern, at least one pulse having a second pattern, and at least one pulse having a third pattern, the storing step during the one or more pulses of light comprising:
- storing charge converted from at least one pulse having the first pattern to the first storage element;
- storing charge converted from at least one pulse having the second pattern to a third storage element; and
- storing charge converted from at least one pulse having the third pattern to a fourth storage element.
16. The method of claim 15, further comprising:
- subtracting the charge stored in the second storage element from the charge stored in the first storage element.
- subtracting the charge stored in the second storage element from the charge stored in the third storage element.
- subtracting the charge stored in the second storage element from the charge stored in the fourth storage element.
17. The method of claim 12, wherein the stream includes a first pulse, a second pulse adjacent the first pulse, and a third pulse adjacent the second pulse, the storing steps comprising:
- storing charge converted from the first pulse to the first storage element;
- storing charge converted between the first and second pulses to the second storage element;
- storing charge converted from the second pulse to a third storage element; and
- storing charge converted between the second and third pulses to a fourth storage element.
18. The method of claim 17, further comprising:
- reading out charge from the first and second storage elements during storage of charge in at least one of the third and fourth storage elements.
19. The method of claim 12, wherein the converting step includes accumulating charge in a photodiode and wherein the method further comprises:
- coupling the photodiode to the first storage element to transfer accumulated charge converted during the one or more pulses of light; and
- coupling the photodiode to the second storage element to transfer accumulated charge converted between the two or more pulses of light.
20. The method of claim 12, wherein the converting step is performed using a photodiode and wherein the method further comprises:
- streaming charge converted by the photodiode to the first storage element during conversion of the one or more pulses of light; and
- streaming charge converted by the photodiode to the second storage element between conversion of the two or more pulses of light.
Type: Application
Filed: Jun 3, 2015
Publication Date: Sep 24, 2015
Applicant: SEMICONDUCTOR COMPONENTS INDUSTRIES, LLC (Phoenix, AZ)
Inventors: Chung Chun WAN (Fremont, CA), Xiangli LI (Boise, ID), Gennadiy AGRANOV (San Jose, CA)
Application Number: 14/729,245