CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module
A CMOS imager in which a CMOS image sensor, a color image processing module and an image compression module are all provided on a single die. Both the color image processing module and the image compression module incorporate pipelined architectures to process the image data at a video rate in a massively parallel fashion.
Latest Patents:
The invention relates generally to improved semiconductor imaging devices and, more specifically, to a CMOS imager provided with an on-chip data compression module.
BACKGROUND OF THE INVENTIONA number of different types of semiconductor-based imagers exist, including charge coupled devices (CCDs), CMOS arrays, photodiode arrays, charge injection devices and hybrid focal plane arrays. Recently, however, CMOS imagers have gained popularity in use in a wide variety of electronic devices, because CMOS imagers offer a number of advantages over other types of imagers. CMOS imagers, for example, are compatible with integrated on-chip electronics (control logic and timing, image processing, and signal conditioning such as A/D conversion). CMOS imagers allow random access to the image data. CMOS imagers have lower fabrication costs as compared with the conventional CCD imagers, since standard CMOS processing techniques can be used. Additionally, CMOS imagers have low power consumption because only one row of pixels at a time needs to be active during the readout and there is no charge transfer (and associated switching) from pixel to pixel during image acquisition. On-chip integration of electronics is particularly advantageous because of the potential to perform many signal conditioning functions in the digital domain (versus analog signal processing) as well as to achieve a reduction in system size and cost.
CMOS imagers as discussed above are generally known as discussed, for example, in Nixon et al., “256×256 CMOS Active Pixel Sensor Camera-on-a-Chip,” IEEE Journal of Solid-State Circuits, Vol. 31(12) pp. 2046-2050, 1996; Mendis et al, “CMOS Active Pixel Image Sensors,” IEEE Transactions on Electron Devices, Vol. 41(3) pp. 452-453, 1994 as well as U.S. Pat. No. 5,708,263, U.S. Pat. No. 5,471,515, and U.S. Pat. No. 6,204,524, which are herein incorporated by reference.
Recent advances in CMOS image sensor technology include the integration of the imager and sophisticated image processing modules on a single die, as mentioned above. These systems-on-a-chip (SOCs) usually have cost, power consumption and form-factor advantages over the multi-chip solutions with the same functionality. Furthermore, image processing modules integrated with CMOS imagers can be fine tuned to the specific properties of the given imager and to the needs of the targeted applications. One segment of the market where the low power of CMOS imagers is most advantageous is the mobile devices market. Many cell-phone designs are incorporating image sensors in their architectures. However, bandwidth limitations imposed by wireless transmission, together with the desire to employ image sensors with large pixel counts, necessitates the use of image compression in the system. Many of the existing designs draw on the ability of the on-board CPU to perform image compression, but at speeds far below video rates. Other available solutions rely on an additional image processing chip to perform color processing and compression.
All of the known solutions require a frame buffer memory to allow for rate conversion between video rate of incoming uncompressed data and the rate at which compression can be performed by either system or dedicated CPU. This leads to increased cost of the imager module and reduced video throughput of the system. Accordingly, it would be desirable to provide a CMOS imager with on-board image compression circuitry which processes image data in real time, and thus eliminates the need for a frame buffer memory.
SUMMARY OF THE INVENTIONThe present invention provides a CMOS imager in which a CMOS image sensor, a color image processing module and an image compression module are all provided on a single die. Both the color image processing module and the image compression module incorporate pipelined architectures to process the image data at a video rate in a massively parallel fashion.
Additional advantages and features of the present invention will be apparent from the following detailed description and drawings, which illustrate preferred embodiments of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized, and that structural, logical and electrical changes may be made without departing from the spirit and scope of the present invention.
In the operation of the
With reference to the block diagram of
The present invention utilizes a unique SRAM addressing scheme to reduce the memory area and improve the utilization rate of memory to maximum. Referring back to
With reference to the block diagram of
Due to the nature of JPEG encoding, the output data rate of the encoder is variable (intermittent). This might present a problem for the overall system architecture, as in the absence of the frame buffer there would be a need to constantly monitor the availability of the compressed data for transfer to the system memory or direct transmission to the remote site. This type of system behavior would unnecessarily tie-up system resources (such as CPU cycles and data bus bandwidth). In order to address this problem, the present invention incorporates a relatively small memory buffer, FIFO 210, at the output of the encoder. This buffer, being much smaller that a full frame buffer, preserves the cost advantages of the system and allows for periods of fixed-rate data output interspersed with periods of inactivity.
In the preferred embodiment of the present invention, output buffer 210 is a dual-ported memory together with pointer control block, allowing the buffer to function as a FIFO (first-in-first-out). In this embodiment, the pointer control block allows for storage of the output of the encoder as soon as it becomes available, while allowing independent retrieval of the data based on the external requests, as long as FIFO is not empty. In yet another embodiment of this invention, the FIFO control block generates Half-Full and/or Almost Empty/Over X bytes flags describing the state of the content of the FIFO. These signals can then be used as an interrupt for external controller, prompting data retrieval cycle and relieving the external controller from the need to constantly monitor data availability in the FIFO.
In the preferred embodiment of the present invention, the INT signal is supplied to access the internal FIFO over the system data/memory bus. To read video data correctly from the internal FIFO, the CPU must know the current state of FIFO. The following FIFO conditions will generate the INT signal transition and will also be reflected as the corresponding INT status register bits. The CPU inquiries the INT status register to get the current FIFO states. Based on the FIFO status, the CPU initiates or stops the video data read process or processes the data.
Bit 0: End of video frame is in FIFO
Bit 1: FIFO overflow
Bit 2: FIFO empty
Bit 3: End of Frame is read from FIFO
Bit 4: X bytes in FIFO where X is programmable
In a preferred embodiment of the present invention, the FIFO outputs are connected directly to the system data/memory bus, allowing for the access to the imager system to be performed over standard system communication channel in a way similar to the memory access. In yet another embodiment of the invention, the access to the registers/control functions of the imager system itself is also performed through the same bus interface. In this embodiment, the control signals are provided to allow distinguishing between various traffic over the output pins: image access, register write and register read access.
In addition to providing real-time compressed data stream, the imager system of the present invention may also need to provide uncompressed video stream either in full frame format or in decimated format (such as VGA image decimated to CIF resolution). Accordingly, in another embodiment of the present invention, the encoder output (compressed data stream) and the uncompressed video can be multiplexed to the input port of the output memory buffer (FIFO).
A typical processor based system that includes a CMOS imager device according to the present invention is illustrated generally at 600 in
A processor system, such as a computer system, for example generally comprises a central processing unit (CPU) 644 that communicates with an input/output (I/O) device 646 over a bus 652. The CMOS imager 610 also communicates with the system over bus 652. The computer system 600 also includes random access memory (RAM) 648, and, in the case of a computer system may include peripheral devices such as a floppy disk drive 654 and a compact disk (CD) ROM drive 656 which also communicate with CPU 644 over the bus 652. As described above, CMOS imager 610 is combined with a pipelined JPEG compression module in a single integrated circuit.
The above description and drawings illustrate a preferred embodiment which achieves the objects, features and advantages of the present invention. It is not intended that the present invention be limited to the illustrated embodiments. Any modification of the present invention which comes within the spirit and scope of the following claims should be considered part of the present invention.
Claims
1-25. (canceled)
26. A CMOS imaging device integrated on a single chip, comprising:
- a CMOS image sensor;
- a color image processing module;
- an image storage module coupled to said color image processing module; and
- an image compression module coupled to said image storage module,
- wherein said image compression module has a pipelined architecture for processing image data in a parallel fashion at video rates without requiring a full frame memory buffer,
- wherein said image compression module implements JPEG compression at video rates, and
- wherein said image storage module comprises a plurality of line buffers at a front end of the compression module to allow two-dimensional processing.
27. A CMOS imaging device integrated on a single chip, comprising:
- a CMOS image sensor;
- a color image processing module;
- an image storage module coupled to said color image processing module; and
- an image compression module coupled to said image storage module,
- wherein said image compression module has a pipelined architecture for processing image data in a parallel fashion at video rates without requiring a full frame memory buffer,
- wherein said image compression module implements JPEG compression at video rates, and
- wherein said image storage module uses an SRAM addressing scheme to allow maximum utilization rate of memory, comprising reading and writing to one SRAM at the same rate without overwriting.
28. The CMOS imaging device of claim 27, wherein an address calculation formula is used in SRAM addressing.
29. A CMOS imaging device integrated on a single chip, comprising:
- a CMOS image sensor;
- a color image processing module;
- an image storage module coupled to said color image processing module;
- an image compression module coupled to said image storage module; and
- a buffer at the output of the image compression module to allow for periods of fixed-rate data output interspersed with periods of inactivity,
- wherein said image compression module has a pipelined architecture for processing image data in a parallel fashion at video rates without requiring a full frame memory buffer, and
- wherein said image compression module implements JPEG compression at video rates.
30. The CMOS imager device of claim 29, wherein the output buffer is a dual-ported memory provided with a pointer control block, such that the buffer functions as a FIFO memory.
31. The CMOS imager device of claim 30, wherein the pointer control block allows for the storage of the output of the encoder as soon as it becomes available, while allowing independent retrieval of data based on external requests.
32. The CMOS imager device of claim 30, further comprising a FIFO & Register & CPU control block to enable register access and FIFO access via a single system data/memory bus.
33. The CMOS imager device of claim 32, wherein the FIFO & Register & CPU control block generates an INT signal reflecting the current state of the FIFO.
34. The CMOS imager device of claim 30, wherein the FIFO pointer control block generates half-full and almost-empty/full flags reflecting the state of the content of the FIFO memory.
35. The CMOS imager device of claim 30, wherein the FIFO outputs are connected directly to a data/memory bus of a system, allowing for access to the system to be performed over standard system communication channels.
36. The CMOS imager device of claim 35, wherein access to registers and control functions of the imager system is also performed over said data/memory bus.
37. The CMOS imager device of claim 32, wherein control signals are provided to distinguish between image access, register write, and register read traffic over the output pins.
38. The CMOS imager device of claim 30, further comprising circuitry for multiplexing compressed and uncompressed video to an input port of the FIFO memory.
39. A process for processing image data from a CMOS imager device, comprising the steps of:
- converting an image to image data using a CMOS image sensor;
- color image processing said image data from a first format which is native to a CMOS imager to a second format more suitable for image compression;
- storing said processed image data in a memory; and
- compressing said stored image data using a processing module having a pipelined architecture for processing image data in a parallel fashion at video rates without requiring a full frame memory buffer,
- wherein said step of compressing the image comprises JPEG compression at video rates, and
- wherein said step of storing said processed image data in a memory comprises using an SRAM addressing scheme to allow maximum utilization rate of memory, said SRAM addressing scheme comprising reading and writing to one SRAM at the same rate without overwriting.
Type: Application
Filed: Sep 21, 2006
Publication Date: May 3, 2007
Applicant:
Inventors: Michael Kaplinsky (Sierra Madre, CA), Igor Subbotin (South Pasadena, CA), Xiuhong Lin (Pasadena, CA)
Application Number: 11/524,534
International Classification: H04N 9/04 (20060101);