Temporally Aligned Exposure Bracketing for High Dynamic Range Imaging
The invention provides an optical imaging system for temporally aligning bracketed exposures of a single image, the system comprising a light aperture, a prism and a image capturing device, where the prism is capable of splitting an incoming image from the light aperture into at least two temporally aligned images, and where the image capturing device captures the temporally aligned images at different levels of exposure.
This application claims the benefit of U.S. Provisional Patent Application No. 61/157,494, filed Mar. 4, 2009, the complete disclosure of which is incorporated herein, in the entirety.
COPYRIGHT NOTICEA portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the United States Patent and Trademark Office patent files and records, but otherwise reserves all other copyright rights.
BACKGROUND OF THE INVENTION1. Field of the Invention
This invention relates generally to imaging systems, and more particularly, to imaging systems that provide varying exposures for production of high dynamic range images.
2. Description of Related Art
High dynamic range imaging (HDRI) is a term applied in image processing, computer graphics and photography, and generally relates to systems or techniques for providing a greater dynamic range of exposures. HDRI is most commonly employed in situations where the range between light and dark areas is great, and subsequently a normal exposure, or even a digitally enhanced exposure, are not adequate to resolve all of the image area.
HDRI manipulates images and exposures to accurately represent the wide range of intensity levels found in real scenes, from direct sunlight to shadows. With HDRI, the user employs multiple exposures and bracketing with photo merging, to get greater detail throughout the tonal range.
More particularly, HDRI processing involves merging several exposures of a given scene into a, typically, 32-bit HDRI source file, which is then “tone mapped” to produce an image in which adjustments of qualities of light and contrast are applied locally to the HDRI source image.
HDRI images are best captured originally in a digital format with a much higher bit depth than the current generation of digital imaging devices. Current devices are built around an 8-bit per channel architecture. That means that both the cameras and output displays have a maximum tonal range of 8-bits per RGB color channel.
HDRI formats are typically 32-bits per channel. A few next generation cameras and displays are capable of handling this kind of imagery natively. It will probably be quite a few years until HDRI displays become common but HDRI cameras and acquisition techniques are already emerging.
HDRI images are typically tone-mapped back to 8-bits per channel, essentially compressing the extended information into the smaller dynamic range. This is typically done automatically with a variety of existing software algorithms, or manually with artistic input through programs like Adobe Photoshop.
So in a typical workflow for HDRI the artist first captures the HDRI image, and then the image is tone-mapped back to desired output device such as ink on paper, an 8-bit RGB monitor, or even a 32-bit HDRI monitor (requiring no tone mapping).
The real challenge with HDRI is not the file formats or computer algorithms to tone map them to 8-bit displays. Those challenges have already been largely met. For example, open EXR is an example of a robust open source HDRI format developed by Industrial Light and Magic. The hardest part of capturing HDR images is the physical devices used to capture the imagery. So far only two ways of capturing HDR images are available.
The first is to use exotic high end cameras with special imaging chips (CMOS or CCD) like the Spheron HDR. Both CCD (charge-coupled device) and CMOS (complimentary metal-oxide semiconductor) image sensors convert light into electrons, though CMOS sensors are much less expensive to manufacture than CCD sensors. These types of cameras are typically used by professionals in controlled environments for the primary purpose of creating spherical photos to illuminate computer generated images (another important use of HDRI). They are not point and shoot cameras and are not capable of motion photography.
The second is shooting multiple varying exposures in rapid succession (known as exposure bracketing) then combining those images taking the highlights from the underexposed images, mid tones from the normally exposed images, and shadows from the over exposed images to create a composite HDR image that retains massive detail in the highlights and shadows where normal cameras would lose detail.
Both of these techniques have substantial disadvantages. The second technique can be done with conventional hardware, but it is time consuming and takes substantial expertise to pull off. In addition, because the images are not temporally aligned, meaning they were taken one after another at different moments in time, there can be changes in the scene that produce artifacts when the HDRI software attempts to eliminate or synthesize the objects in motion across the frame. An example would be a car moving through the frame.
Even a slight movement of the camera between exposures will be noticeable in the resulting combined image. Moving objects will be “ghosted” in the HDRI image. As such this technique is totally useless for motion photography and can only be used with substantial success in still photography applications.
For this reason, exposure bracketed HDRI is typically restricted to still subjects, and any animals, cars, pedestrians, moving leaves or litter, clouds, etc., in fact anything that is shifting within the frame will preclude HDRI, or at the very least lead to unhappy results.
Further, producing HDRI from multiple images can be a time consuming and frustrating task. HDRI requires multiple, huge files, multiple steps, and typically specialized and complicated software.
The first technique is very expensive and requires exotic hardware or sophisticated electronic and software systems. While imaging chips are moving ever forward in sensitivity and dynamic range, they still do not produce the dramatic results that the first technique of changing exposures does. In addition, these special cameras are not capable of shooting higher frame rates required to shoot motion pictures. These products are used for narrow specialized purposes.
Proposed solutions to the problems associated with the second technique are reflected in various published patents at the United States Patent and Trademark Office. For example, United States Patent Application No. 20060221209, to McGuire, et al., published Oct. 5, 2006, teaches an apparatus and method for acquiring and combining images of a scene with multiple optical characteristics at multiple resolutions. Disclosed therein is a camera system that acquires multiple optical characteristics at multiple resolutions of a scene. The camera system includes multiple optical elements arranged as a tree having a multiple of nodes connected by edges. The system employs filters at the end of the chain, and lenses are placed in front of each of the sensors, creating additional sources of optical distortion.
United States Patent Application No. 20070126918, to Lee, published Jun. 7, 2007, discloses cameras that can provide improved images by combining several shots of a scene taken with different exposure and focus levels is provided. In addition, cameras are provided, which have pixel-wise exposure control means so that high quality images are obtained for a scene with a high level of contrast. The system is complicated, and employs light reducing filters to create exposures of varying intensity. Much of the light is lost, reducing clarity and introducing sources of distortion and noise to the images.
United States Patent Application No. 20080149812, to Ward, et al., published Jun. 26, 2008, discloses an electronic camera comprising two or more image sensor arrays. At least one of the image sensor arrays has a high dynamic range. The camera also comprises a shutter for selectively allowing light to reach the two or more image sensor arrays, readout circuitry for selectively reading out pixel data from the image sensor arrays, and, a controller configured to control the shutter and the readout circuitry. The controller comprises a processor and a memory having computer-readable code embodied therein which, when executed by the processor, causes the controller to open the shutter for an image capture period to allow the two or more image sensor arrays to capture pixel data, and, read out pixel data from the two or more image sensor arrays. This is essentially a total digital solution to the problem of controlling exposure levels for different images for high dynamic range processing.
Finally, United States Patent Application No. 20070177004, to Kolehmainen, et al., published Aug. 2, 2007, is directed to an image creating method and imaging device comprising at least two image capturing apparatus, each apparatus being arranged to produce an image. The apparatus is configured to utilize at least a portion of the images produced with different image capturing apparatus with each other to produce an image with an enhanced image quality. Multiple lenses are required to implement this method, which is expensive and creates parallax and optic imagery distortions with each lens addition.
None of the prior approaches have been able to provide a simple means for capturing multiple images that overcome the difficulties of temporal misalignment, and that are simple and quickly resolved into a high definition range image.
What is needed is an inexpensive solution that can be easily integrated into products with conventional form factors. This solution would ideally be easy to use, compact, and able to shoot at high frame rates with no introduction of temporal alignment problems and associated artifacts.
SUMMARY OF THE INVENTIONBy this invention is provided an optical imaging system for temporally aligning bracketed exposures of a single image, the system comprising a light aperture, a prism and a image capturing device, where the prism is capable of splitting an incoming image from the light aperture into at least two temporally aligned images, and where the image capturing device captures the temporally aligned images at different levels of exposure.
In one embodiment of the invention, the prism splits the intensity of said incoming image to achieve a desired EV output interval between temporally aligned images.
In a different embodiment, the capturing device further comprises image detection sensors, and the ISO of the sensors is adjusted to achieve a desired EV output interval between said images.
In another aspect of the invention, the system comprises an image processing device connected to said image capturing device.
In one embodiment, the image processing device comprises a computer processor.
In a different embodiment, the device further comprises a tone-mapping processor.
In a different aspect, the system comprises an eyepiece for viewing the image to be captured by the lens.
In a still further aspect, the system comprises a digital readout monitor.
In another embodiment, the prism is capable of splitting the image into three or more levels of exposure.
In a different embodiment, the three levels of exposure are about 14%, about 29% and about 57%, respectively, of the exposure level of the original image.
In a different embodiment, the three levels of exposure are about 5%, about 19% and about 76%, respectively, of the exposure level of the original image.
In a different embodiment, the three levels of exposure are about 1%, about 11% and about 88%, respectively, of the exposure level of the original image.
In a still different embodiment, the prism is capable of splitting the image into four or more levels of exposure.
In another embodiment, the prism is capable of splitting the image into five or more levels of exposure.
In a different aspect, the invention provides a method for temporally aligning bracketed exposures of a single image, the method comprising the steps of a) using a prism to split an incoming image from a light aperture into at least two temporally aligned images, and b) using an image capturing device to capture the temporally aligned images at different levels of exposure.
These and other features and advantages of this invention are described in, or are apparent from, the following detailed description of various exemplary embodiments of the apparatus and methods according to this invention.
A more complete understanding of the present invention and the attendant features and advantages thereof may be had by reference to the following detailed description when considered in conjunction with the accompanying drawings wherein:
The optical imaging system of the present invention provides an improvement to high dynamic range imaging, and assemblies therefore, that allows temporally aligned exposure bracketing. The system is simple, elegant, leverages existing technologies, allows for motion capture with no temporal distortion, and is relatively inexpensive to implement.
The present optical imaging system allows the user to capture light with confidence that the under and over exposed regions in the image will be imaged properly. The user simply captures all the available light with and image capturing device, and determines later how to map that information to the output device. With the optical imaging system the user can create stunning imagery that is otherwise impossible to capture, even with the most sophisticated of the current generation of normal photography equipment.
Before the present invention is described in greater detail, it is to be understood that this invention is not limited to particular embodiments described, as such may, of course, vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present invention will be limited only by the appended claims.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, the preferred materials are now described.
All publications and patents cited in this specification are herein incorporated by reference as if each individual publication or patent were specifically and individually indicated to be incorporated by reference and are incorporated herein by reference to disclose and describe the materials in connection with which the publications are cited. The citation of any publication is for its disclosure prior to the filing date and should not be construed as an admission that the present invention is not entitled to antedate such publication by virtue of prior invention. Further, the dates of publication provided may be different from the actual publication dates which may need to be independently confirmed.
It must be noted that as used herein and in the appended claims, the singular forms “a,” “an”, and “the” include plural referents unless the context clearly dictates otherwise. It is further noted that the claims may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely,” “only” and the like in connection with the recitation of claim elements, or use of a “negative” limitation.
As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present invention.
For example, although the foregoing drawings and references refer to color images and processors, the system and methods work equally well for black and white (grayscale) images and sensors. For instance, some applications for scientific or industrial use may prefer grayscale imagery.
Further, while unusual in present day camera art, it is possible to build an imaging apparatus without a primary lens (i.e., a pinhole camera or a slit scanner). These applications are more likely in industrial or scientific applications. The invention can easily be adapted for designs that don't include a front end lens, but rather a simple aperture or the like.
Generally speaking, the systems and methods utilize prism splitting by full spectrum brackets to several image detecting sensors of an image capturing device. The system eliminates exotic image sensors as a necessary feature. The system allows multiple exposures from existing commodity sensors simultaneously by simply dividing the incoming light for an image into multiple and different levels of exposure for the same image.
The temporally aligned imaging system can be analogized to Technicolor. Before color film stock was developed, Hollywood was in search of a way to shoot films in color. Technicolor, Inc. was the first company to develop a way to create color pictures from black and white film stock. It utilized three rolls of black and white film exposed simultaneously through a special set of beam splitters with red, green, and blue filters on them.
Simply put, each black and white film negative recorded just the red, green, or blue information. This process was done in reverse with a projector that ran all three rolls of film simultaneously with the correct color filter in front of each. When the images are aligned properly, a full color picture is realized.
As better color film stocks emerged, this process fell out of favor, until video cameras emerged. In the early days of video, color sensors were not very sharp, and had difficulty producing high resolution images, or good color saturation and reproduction. Black and white sensors were far sharper and had a higher dynamic range. So the Technicolor principle of using three image sensors and a beam splitter to feed each an identical simultaneous image was dusted off and put into use for a new generation of imaging products. Three black and white CCD were used with a new and vastly improved beam splitter called a trichroic prism.
This technique is used to this day in professional level video cameras, sometimes referred to as 3CCD sensor. The three red, green and blue sensors not only allow for sharper more saturated colors but also help enhance the dynamic range of the images they help create. But just as better color film stocks helped to usher out the era of the Technicolor process, better CMOS and CCD sensors are ushering out the era of 3CCD sensor systems in favor of full color single sensor systems. In fact some of the highest end professional cameras like the lineup from RED Digital Cinema Camera Company as well as every professional Digital SLR use only one full color sensor. It is quite apparent that sensor technology has progressed to the point where a single color sensor can replace and even outperform 3CCD sensor systems.
In one aspect, the temporally aligned exposure bracketing system employs trichroic prisms adapted to split the entire spectrum to each of multiple full color sensors, at different exposure levels, rather than splitting out the spectrum into different colors.
The system allows a color neutral change in the amount, rather than the spectrum, of light going to each sensor, by the application of such prisms for the temporal alignment of images for HRDI. By “color neutral”, it is meant that while the temporally aligned images created by the prism may vary in intensity between themselves, or between themselves and the incoming image, they are not substantially different from one another in color spectrum, i.e., the prism creates split images that are similar in color spectrum, or spectrally neutral, even if differing substantially in intensity.
All of the commonly understood color separation prism layouts may also be used for neutral separation. In reference now to
In
The film coatings 104 and 105 for the prism 100 may be of any of numerous coatings known to the art and capable of achieving a color neutral split, or separation, of the image, by reflection of the incoming light 30. Two examples of such spectrally neutral films include a thin film metallic coating, typically aluminum or silver, with or without a set of dielectric layers, and a set of dielectric layers consisting of high and low refractive index materials with the thin film stack designed to reflect a certain percentage of the incident light over the visible wavelength range. These and related types of thin film coatings 104 and 105 shall be termed “spectrally neutral film” or, alternatively, “neutral film.”
The following table provides a demonstration for calculating the percentages for such a system, using a prism for splitting a captured image into temporally aligned exposures 106, 107 and 108 at levels of 14.2857%, 28.5714% and 57.1429%, respectively.
Thus, with color image sensors that do not need the RGB color split, the prism is harnessed for the purpose of splitting out different exposures of the same image, that are temporally aligned (taken at the same moment).
Various means can be employed to adjust the EV (Exposure Value, commonly referred to as a “stop”) up and down with the intensity spectrum that would allow a camera to capture the images simultaneously. For instance, this can be accomplished by splitting the incoming light into different intensities directly in the prism, adjusting the ISO sensitivity in the sensors or some combination of the two.
At one extreme, the system could split the light intensity in the prism 100 into equal amounts of roughly 33% each and then adjust the ISO of the sensors 101, 102 and 103 respectively to achieve different EV output intervals. At another extreme, the system could split the light intensity within the prism 100 into the desired EV intervals for the light 106, 107 and 108. Thus, even while leaving the ISO of the sensors the same, the desired different EV output intervals are achieved for the recorded images. Any combination between these two extremes may be more or less desirable for various applications.
Systems for controlling the action of the lens and associated hardware, including light responsive software controllers, are well known to the art.
In addition, the individual sensors could benefit from some tuning for their respective exposure levels to reduce noise and other artifacts associated with under and over exposure, in ways known to the art.
A high quality standard camera lens 140 can be used with the system 10 to gather and focus light from the light aperture.
The system 10 also will typically include an eyepiece and/or monitor 150 for aligning the images for capture from the lens onto the sensors.
Additional features of the system typically would include mass storage for either the 8 bit tone mapped data 160, or the raw 32 bit HDRI data 170. Other HDRI formats are known, for instance 16 bit and 14 bit formats, though the standard is evolving toward the higher 32 bit format.
The ISO is a function of how sensitive the sensor/film is to light. The exposure generated by a particular aperture, shutter speed, and sensitivity combination can be represented by its exposure value “EV”. Zero EV is defined by the combination of an aperture of f/1 and a shutter speed of 1s at ISO 100.
The term “exposure value” is used to represent shutter speed and aperture combinations only. An exposure value which takes into account the ISO sensitivity is called “Light Value” or LV and represents the luminance of the scene. For the sake of simplicity, as is the case in this patent, Light Value is often referred to as “exposure value”, grouping aperture, shutter speed and sensitivity in one familiar variable. This is because in a digital camera it is as easy to change sensitivity as it is to change aperture and shutter speed.
Each time the amount of light collected by the sensor is halved (e.g., by doubling shutter speed or by halving the aperture), the EV will increase by 1. For instance, 6 EV represents half the amount of light as 5 EV.
Table 2 shows the additional variations possible for adjusting output intervals on top of the prismatic split, for +/−3EV, +/−2EV and +/−1EV.
The various exposure intervals can be modified or enhanced by using different combinations of prism splits with sensor sensitivity settings. This is accomplished by using differential exposure values (EV) to amplify the differences created by the prismatic split at the level of the sensors.
Table 3 shows results for a diagrammatic view of a system produced according to the invention that as shown in
Table 4 shows the results where variations to exposure intervals are shown using different combinations of prism splits and sensor sensitivity settings of +/−3EV, +/−2EV and +/−1EV. Table 4 shows the various ISO settings for each sensor that is used to produce alternative EV output intervals from each sensor (these settings are for +/−1EV input values only) as found in Table 3 (these settings are for +/−2EV input values only).
Table 5 is the results for a system produced according to the invention as depicted in
Table 6 is the settings for a system as would be configured for the Table 5 percentages, where variations to exposure intervals are shown using different combinations of prism splits and sensor sensitivity settings of +/−3EV, +/−2EV and +/−1EV.
The system depicted in
Further, while the use of a three-way prism is demonstrated in
Use of different prism splits will be desirable for different applications. In a very minimal configuration a 2-way configuration could work (
On the other hand, in some scientific or professional applications, the greater control from more elaborate splits possible from the four-way and five-way neutral prism splits shown in
While this invention has been described in conjunction with the specific embodiments outlined above, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, the preferred embodiments of the invention, as set forth above, are intended to be illustrative, not limiting. Various changes may be made without departing from the spirit and scope of this invention.
Claims
1. An optical imaging system for temporally aligning bracketed exposures of a single image, said system comprising a light aperture, a prism and a image capturing device,
- wherein said prism is capable of splitting an incoming image from said light aperture into at least two color neutral, temporally aligned images,
- whereby said image capturing device captures said temporally aligned images at different levels of exposure.
2. The system of claim 1 wherein said prism splits the intensity of said incoming image to achieve a desired EV output interval between said temporally aligned images.
3. The system of claim 1 wherein said image capturing device further comprises image detection sensors for said temporally aligned images.
4. The system of claim 1 wherein the ISO of said sensors is adjusted to achieve a desired EV output interval between said images.
5. The system of claim 1 wherein said prism further comprises at least one neutral film coating.
6. The system of claim 1 further comprising an image processing device connected to said image capturing device.
7. The system of claim 1 wherein said image processing device comprises a computer processor.
8. The system of claim 7 further comprising a tone-mapping processor.
9. The system of claim 8 wherein the image processing device and tone-mapping processor are contained on a single integrated circuit.
10. The system of claim 1 further comprising a digital readout monitor.
11. The system of claim 1 further comprising a lens associated with said aperture.
12. The system of claim 1 further comprising an eyepiece for viewing said incoming image.
13. The system of claim 1 wherein said prism is capable of splitting said image into at least three temporally aligned images having different levels of exposure.
14. The system of claim 13 wherein said three levels of exposure are about 14%, about 29% and about 57%, respectively, of the intensity of said incoming image.
15. The system of claim 13 wherein said three levels of exposure are about 5%, about 19% and about 76%, respectively, of the intensity of said incoming image.
16. The system of claim 13 wherein said three levels of exposure are about 1%, about 11% and about 88%, respectively, of the intensity of said incoming image.
17. The system of claim 1 wherein said prism is capable of splitting said image into at least four temporally aligned images having different levels of exposure.
18. The system of claim 1 wherein said prism is capable of splitting said image into at least five temporally aligned images having different levels of exposure.
19. A method for temporally aligning bracketed exposures of a single image, said method comprising the steps of
- a) using a prism to split an incoming image from a light aperture into at least two temporally aligned images, and
- b) using an image capturing device to capture said temporally aligned images at different levels of exposure,
- wherein said prism produces a color neutral split of said temporally aligned images.
20. The method of claim 19 wherein said prism splits the intensity of said incoming image to achieve a desired EV output interval between said temporally aligned images.
21. The method of claim 19 wherein said image capturing device further comprises image detection sensors for said temporally aligned images.
22. The method of claim 19 wherein the ISO of said sensors is adjusted to achieve a desired EV output interval between said images.
23. The method of claim 19 wherein said prism is capable of splitting said image into at least three temporally aligned images having different levels of exposure.
24. The method of claim 23 wherein said three levels of exposure are about 14%, about 29% and about 57%, respectively, of the intensity of said incoming image.
25. The method of claim 23 wherein said three levels of exposure are about 5%, about 19% and about 76%, respectively, of the intensity of said incoming image.
26. The method of claim 23 wherein said three levels of exposure are about 1%, about 11% and about 88%, respectively, of the intensity of said incoming image.
27. The method of claim 19 wherein said prism is capable of splitting said image into at least four temporally aligned images having different levels of exposure.
28. The method of claim 19 wherein said prism is capable of splitting said image into at least five temporally aligned images having different levels of exposure.
Type: Application
Filed: Mar 4, 2010
Publication Date: Sep 9, 2010
Inventor: Paul A. Wagner (Vancouver, WA)
Application Number: 12/717,765
International Classification: H04N 5/235 (20060101);
