Noise-constrained tone curve generation
Methods to generate, and devices to use, an adaptive tone curve generation operation are disclosed. Tone curves so generated have been found to produce stable tone mapping in a wide range of operational environments, including during multi-exposure fused images (e.g., during high dynamic range image fusion). In general, techniques are disclosed that use a content-load histogram in combination with image capture device noise information to adaptively bound a tone curve's slope and, therefore, the resulting tone curve. As used here, a content-load histogram is a histogram that captures spatial, color or other information carried by an image's pixels at various gray levels.
Latest Apple Patents:
- TECHNOLOGIES FOR PACKET FILTERING FOR PROTOCOL DATA UNIT SESSIONS
- TECHNOLOGIES FOR SIGNAL LEVEL ENHANCED NETWORK SELECTION
- DEBUGGING OF ACCELERATOR CIRCUIT FOR MATHEMATICAL OPERATIONS USING PACKET LIMIT BREAKPOINT
- CROSS LINK INTERFERENCE REPORTING IN 5G COMMUNICATION SYSTEMS
- CROSS LINK INTERFERENCE (CLI) CONFIGURATION AND MEASUREMENT
This disclosure relates generally to the field of image processing and, more particularly, to various techniques to generate noise-constrained tone curves for use in real-time image processing.
Tone mapping is the process of remapping gray levels from a first or input image to different levels in a second or output image. Tone mapping is generally used to adjust the input image to more closely match the visual appearance anticipated by the human eye on a particular output device, and/or to better preserve image information when mapping an input image to a smaller output dynamic range. Global tone mapping refers to the situation where there is a single tone curve that maps input gray levels to output gray levels. Local tone mapping refers to the case where a single gray level in the input image maps to multiple gray levels in the output image depending on the spatial location and configuration of the input image. Tone mapping is used in almost all image processing pipelines, but is of particular importance when rendering high dynamic range images for a limited dynamic range output device (e.g., created for example by fusing multiple exposures). Illustrative devices having a limited output dynamic range include most consumer desktop and portable displays.
SUMMARYIn general, the inventive concept provides methods, systems and computer readable media to generate noise-constrained tone curves. In one embodiment, each pixel in an image obtained from an image capture device may be evaluated to determine its contribution to a content-load histogram (e.g., a content-load measure). For example, each pixel may be used to identify a bin in the content-load histogram, where after the pixel (or pixel neighborhood) may be evaluated to identify spatial information corresponding to the pixel. In one embodiment, the spatial information may be generated in real-time as part of the pixel evaluation procedure. In another embodiment, the spatial information may be obtained from a separate image. However obtained, the information may be used to update the identified content-load histogram's bin. Once an image's pixels have been evaluated, the resulting content-load histogram may be adjusted based on the image capture device's noise characteristics. In particular, upper and lower bounds for the content-load histogram may be established. The adjusted content-load histogram may then be use to generate a noise-constrained tone curve which may be used to map an input image to an output image.
In another embodiment, each content-load measure may be adjusted or weighted to account for the presence of one or more anchor pixels. In one such embodiment, a first anchor pixel may be selected and each pixel in the input image evaluated in terms of that anchor pixel. For example, each pixel's content-load measure may be adjusted in accordance with a specified functional relationship between itself and the current anchor pixel. In one embodiment, the specified functional relationship may be based on the distance between the two pixels. At one extreme, every input image pixel may be selected as an anchor pixel. Alternatively, a specified subset of image pixels may be selected to be anchor pixels. Once a content-load histogram has been generated for each anchor pixel, the histograms may be adjusted to account for the image capture devices noise characteristics. Once adjusted, each content-load histogram may be converted to a noise-constrained tone curve and used to generate an output image.
This disclosure pertains to systems, methods, and computer readable media to generate noise-constrained tone curves. The disclosed tone curves have been found to produce stable tone mapping for high dynamic range images produced through multi-exposure fusion (e.g., during high dynamic range image fusion). In general, techniques are disclosed that use a content-load histogram in combination with an image capture device's noise information to adaptively bound the content-load histogram's slope and, therefore, the resulting tone curve. As used herein, a content-load histogram is a histogram where each bin corresponds to a value, or range of values, that pixels in an input image may assume (e.g., luma or color intensity), and where each bin's value represents the amount of spatial, color or other information carried by the input image's pixels at the bin's corresponding value or range of values (e.g., luma or color intensity).
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the inventive concept. As part of this description, some of this disclosure's drawings represent structures and devices in block diagram form in order to avoid obscuring the invention. In the interest of clarity, not all features of an actual implementation are described in this specification. Moreover, the language used in this disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. Reference in this disclosure to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention, and multiple references to “one embodiment” or “an embodiment” should not be understood as necessarily all referring to the same embodiment.
It will be appreciated that in the development of any actual implementation (as in any development project), numerous decisions must be made to achieve the developers' specific goals (e.g., compliance with system- and business-related constraints), and that these goals may vary from one implementation to another. It will also be appreciated that such development efforts might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the design an implementation of image processing systems having the benefit of this disclosure.
Referring to
Together the luma values and content-load values at each pixel in the image may be used to generate a content-load histogram (block 115). A content-load histogram may be thought of as a histogram in which each bin corresponds to a gray level in a luma image, and each bin's value represents the amount of spatial or color information carried by the pixels with the corresponding gray level. As such, it will be understood that the content-load histogram generally has a number of bins equal to the number of gray levels in the luma image. Sometimes, however, the content-load histogram's bin size can be different from one digital count, resulting in a histogram having a different size.
One illustrative method to generate a content-load histogram is shown in
H[i]=Y[i]S[i], EQ. 1
where H[i] represents the i-th bin of the content-load histogram, Y[i] represents the i-th element/pixel of the input luma image, S[i] represents the i-th element/pixel of the spatial image, and ‘’ represents a generic composition operator. In one embodiment, for example, the composition operator may represent a sum operator in which each spatial image value is added to that content-load histogram bin at the corresponding luma image's gray level (i.e., value). In another embodiment, the composition operator may yield a mean or standard deviation value. Substantially any function (linear or nonlinear) may be used that captures or characterizes the amount of information carried by the selected spatial image at each gray level present in the input luma image. Once the first luma pixel has been processed, a check can be made to determine if additional luma pixels remain (block 315). If the luma image has not been completely processed in accordance with blocks 305-310 (the “NO” prong of block 315), the next pixel from the luma image may be selected (block 320), where after operation 115 continues at block 305. If all pixels in the input luma image have been processed (the “YES” prong of block 315), operations continue at block 120 of
While spatial content image 225 may comprise a separate buffered image as suggested by
As an aside, the process outlined in
Returning to
where h[i] represents the i-th bin of the normalized and moderated content-load histogram, H[i] represents the i-th bin of the content-load histogram in accordance with block 115, ‘a’ represents a dampening factor, ‘n’ represents the number of bins in the content-load histogram, and the summation value represents a normalizing parameter. The factor a may be chosen to provide the desired stability and may, in general, be determined empirically.
Both system noise (block 125) and quantization noise (block 130) may be obtained and represented by a table or array with one element for each gray level or each gray level range. System or camera noise (represented as Ns) is typically a function of the type of image sensor and the particular image capture parameters used during image capture (e.g., exposure and gain settings). System noise may includes dark noise and shot noise. Quantization noise (represented as Ng) is typically a function of the image processing pipeline that takes the sensor's output and delivers the image or images being processed (e.g., luma image 200). When a single image is being processed, quantization noise Nq may be a constant (for that image). When multiple frames having different exposures are being combined or fused, such as during High Dynamic Range (HDR) or wide area-of-view image generation, the quantization error for each gray level must be estimated from both the original quantization error and the fusing weights for each exposure.
With normalized content-load histogram h, system noise table Ns, and quantization noise Nq tables known, the signal-to-noise ratio for each gray level may be expressed as follows:
where SNR[i] represents the signal-to-noise ratio at the i-th gray level, max( ) represents the algebraic maximum function, and ‘i’ runs from 0 to (n−1). See discussion above.
From EQ. 3, an upper-bound to the tone curve's slope at output gray level ‘i’ may be expressed as follows (block 135):
where TSNR represents the minimum allowed target signal-to-noise ratio (typically a tuning parameter that may be adjusted to suit the needs of the developer's particular implementation). In general, the larger the TSNR value, the less tolerant the resulting tone curve will be to noise. It may be recognized that in conventional histogram equalization, the slope at the i-th gray level is proportional to the luma histogram count at that level. In contrast, operation 100 (and, more specifically, block 135) constrains the maximum slope at any given gray level by incorporating the effect of noise.
When an input image has no, or very few, pixels at some gray levels (e.g., a snow scene without any dark objects), it is usually desirable to not allocate as many output gray levels to these input gray levels. It is also not desirable, however, to eliminate these gray levels altogether from the output image. Doing so could cause gradient areas in the output image to become unnatural, or overly exaggerate contrast at the dominant gray levels. It has been found that to avoid eliminating an input image's zero or low use gray levels during the tone mapping process, a lower-bound to the tone curve's slope may be set (block 140). While this lower-bound may be set arbitrarily, there is usually no need for it to be higher than one quantization level for a single exposure image or, in the case of a multi-exposure fused HDR image, higher than the estimated quantization step size. In another embodiment, lower-bound slope may be a function of the gray level.
Referring to
h[i]=max(SMIN,min(SMAX[i],h[i])) EQ. 5
to clip, and
to renormalize. Here h[i], SMAX[i], SMIN, and max( ) are as discussed above while min ( ) represents the algebraic minimum function.
As shown in
where T[i] represents the value of the i-th tone curve element/value.
Tone curve T[ ] may be used in a number of ways to generate an output image. Referring to
where G[i] represents the gain factor at the i-th pixel position, Yin[i] represents the input luminance image 535 value at the i-th pixel position, and Yout[i] represents the output luminance image 540 value at the i-th pixel position.
While the above describes how to create and use an adaptive global tone curve, the approach set-forth in operation 100 may also be used to provide adaptive local tone curves. In one embodiment, a content-load histogram may be generated for every pixel, with each histogram used to create a corresponding tone curve. In another embodiment, a content-load histogram may be generated for a subset of the image pixels and a tone curve generated for each histogram. In all embodiments disclosed herein, and unlike prior art region-based tones curves, every pixel in an image may contribute to every tone curve. Allowing each pixel to have an influence on each content-load histogram/tone curve has provided unexpectedly good results vis a vis output image quality.
Referring to
Next, a content-load histogram for the selected anchor pixel may be initialized (block 610), a “first” pixel from the image selected (block 615), and a content-load measure determined as described above (block 620). A weight factor may then be determined based on a specified relationship between the current anchor pixel and the selected image pixel (block 625). In general, this specified relationship may be linear or non-linear. In one embodiment, the specified relationship may be a Gaussian distribution of the distance between the anchor pixel and the currently selected image pixel. (For a particular example, see the discussion below in connection with
By way of example, consider the special-case embodiment illustrated in
In general, functional relationship 810 may be whatever a particular implementation needs and, as noted above, may be linear or non-linear. In one embodiment, functional relationship 810 may be a Gaussian distribution of the distance between an anchor pixel (e.g., pixel 705) and a pixel being evaluated (e.g., pixel 715). For example, this relationship may be expressed as follows:
where (xa, ya) represents the x and y coordinates of a first anchor pixel, (x1, y1) represents the x and y coordinates of a pixel from the input image being evaluated, f( ) represents a functional relationship between the pixels at (xa, ya) and (x1, y1), exp( ) represents the exponential function, ‘d’ represents a distance measure between the pixels at (xa, ya) and (x1, y1), and ‘σ’ represents the spread of the Gaussian distribution about the anchor pixel at (xa, ya). In general, distance measure d may represent any metric that makes sense for the particular implementation; not just distance. Because images are composed of pixels that are generally arranged in a rectangular array however, it is convenient to think of d as a distance metric. One example distance measure is a Euclidean distance that may be represented as:
d=(xa−x1)2+(ya−y1)2, EQ. 10
where (xa, ya) and (x1, y1) are as described above.
For the example distribution of EQ. 9, σ represents the spread of the Gaussian distribution about the anchor pixel located at (xa, ya). It can be seen that as σ increases, the spread of the modeled distribution increases which, in turn, gives pixels distal to the anchor pixel at (xa, ya) more influence on the generated values (compared to a smaller σ). Likewise, as σ decreases, the spread of the modeled distribution decreases which, in turn, gives pixels distal to the anchor pixel less influence on the generated values (compared to a larger σ). In one embodiment, a threshold may be applied to the output of the selected functional relationship so that any value below a certain threshold may return a zero and thus, pixels greater than a certain distance from the anchor pixel have no influence on it's histogram. In another embodiment, a floor and/or ceiling function may be applied to the output of the selected functional relationship. Once a anchor pixel's content-load histogram has been determined (e.g., in accordance with
Output image generation in accordance with block 660 in
where pout(x1, x1) represents the value for the pixel located at (x1, x1) in the output image, ω represents weighting factors for each local tone curve, pin (x1, x1) represents the value for the pixel located at (x1, x1) in the input image, T represents the local tone curve operator such that Ti(x1, x1) represents output from the i-th local tone curve corresponding to the value of the input image's pixel located at (x1, x1), and ‘M’ represents the number of local tone curves. It will be recognized that the summation in the denominator of EQ. 11 represents a normalization factor.
With respect to weighting factor ωi, values may be found by looking at some functional relationship (e.g., distance) between pout(x1, x1) and each of the pixels for which tone curves have been generated (two in
Referring to
In one embodiment histogram generation unit 950 produces a content-load histogram in accordance with
Referring to
Processor 1005 may execute instructions necessary to carry out or control the operation of many functions performed by device 1000 (e.g., such as the generation and/or processing of images in accordance with operations 100 and 600). Processor 1005 may, for instance, drive display 1010 and receive user input from user interface 1015. User interface 1015 can take a variety of forms, such as a button, keypad, dial, a click wheel, keyboard, display screen and/or a touch screen. Processor 1005 may be a system-on-chip such as those found in mobile devices and include a dedicated graphics processing unit (GPU). Processor 1005 may be based on reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architectures or any other suitable architecture and may include one or more processing cores. Graphics hardware 1020 may be special purpose computational hardware for processing graphics and/or assisting processor 1005 process graphics information. In one embodiment, graphics hardware 1020 may include a programmable graphics processing unit (GPU).
Sensor and camera circuitry 1050 may capture still and video images that may be processed to generate images in accordance with this disclosure and may, for example, incorporate raw image processing pipeline 940. Output from camera circuitry 1050 may be processed, at least in part, by video codec(s) 1055 and/or processor 1005 and/or graphics hardware 1020, and/or a dedicated image processing unit incorporated within circuitry 1050. Images so captured may be stored in memory 1060 and/or storage 1065. Memory 1060 may include one or more different types of media used by processor 1005, graphics hardware 1020, and image capture circuitry 1050 to perform device functions. For example, memory 1060 may include memory cache, read-only memory (ROM), and/or random access memory (RAM). Storage 1065 may store media (e.g., audio, image and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage 1065 may include one more non-transitory storage mediums including, for example, magnetic disks (fixed, floppy, and removable) and tape, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as Electrically Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory 1060 and storage 1065 may be used to retain computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 1005 such computer program code may implement one or more of the methods described herein.
It is to be understood that the above description is intended to be illustrative, and not restrictive. The material has been presented to enable any person skilled in the art to make and use the invention as claimed and is provided in the context of particular embodiments, variations of which will be readily apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with each other). The scope of the invention therefore should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.”
Claims
1. A non-transitory program storage device, readable by a processor and comprising instructions stored thereon to cause the processor to:
- obtain, from an image capture device, a first image comprising pixels wherein each pixel has a value;
- select one or more anchor pixels;
- select, for each of the one or more anchor pixels, every pixel in the first image one at a time, and with each selected pixel— identify a bin within a content-load histogram based, at least in part, on the value of the selected pixel, wherein a content-load histogram comprises a histogram in which each bin identifier corresponds to a value each pixel in the first image may assume, and whose bin values represent an amount of a specified structural information carried by the first image's pixels at a bin's corresponding bin identifier, determine a content-load value corresponding to the selected pixel, determine a weight factor for the selected pixel based, at least in part, on the selected pixel and the selected anchor pixel, update the identified content-load histogram bin based, at least in part, on the content-load value and the weight factor;
- adjust the content-load histogram based, at least in part, on noise characteristics of the image capture device;
- generate a tone curve based, at least in part, on the adjusted content load histogram; and
- generate an output image based, at least in part, on the tone curve.
2. The non-transitory program storage device of claim 1, wherein the instructions to cause the processor to obtain a first image comprise instructions to cause the processor to obtain a first luminance image.
3. The non-transitory program storage device of claim 1, wherein the instructions to cause the processor to select an anchor pixel comprise instructions to cause the processor to select each pixel, one at a time, as an anchor pixel.
4. The non-transitory program storage device of claim 1, wherein the instructions to cause the processor to determine a content-load value comprise instructions to cause the processor to determine a dynamic range value in a region around the selected pixel.
5. The non-transitory program storage device of claim 4, wherein the instructions to cause the processor to determine a dynamic range value comprise instructions to cause the processor to calculate a dynamic range value in real-time.
6. The non-transitory program storage device of claim 1, wherein the instructions to cause the processor to determine a content-load value comprise instructions to cause the processor to:
- identify a second image comprising the same number of pixels as the first image, wherein each pixel in the second image has a value and corresponds to one pixel in the first image; and
- select that pixel in the second image corresponding to the selected pixel from the first image, wherein the value of the pixel selected from the second image comprises the content-load value.
7. The non-transitory program storage device of claim 1, wherein the instructions to cause the processor to determine a weight factor comprise instructions to cause the processor to:
- determine a distance measure value between the selected pixel and the selected anchor pixel; and
- determine a weight factor based, at least in part, on the distance measure value.
8. The non-transitory program storage device of claim 7, wherein the instructions to cause the processor to determine a distance measure value comprise instructions to cause the processor to determine a Euclidean distance between the selected pixel and the selected anchor pixel.
9. The non-transitory program storage device of claim 8, wherein the instructions to cause the processor to determine a weight factor comprise instructions to cause the processor to apply the distance measure value to a Gaussian distribution.
10. The non-transitory program storage device of claim 1, wherein the instructions to cause the processor to update the identified content-load histogram bin comprise instructions to cause the processor to apply the determined weight factor to the content-load value to generate an update value.
11. The non-transitory program storage device of claim 10, wherein the instructions to cause the processor to apply the determined weight factor to the content-load value comprise instructions to cause the processor to multiply the weight factor and the content-load value.
12. The non-transitory program storage device of claim 10, wherein the instructions to cause the processor to update the identified content-load histogram bin comprise instructions to cause the processor to add the update value to the identified content-load histogram bin.
13. The non-transitory program storage device of claim 1, wherein the instructions to cause the processor to adjust the content-load histogram comprise instructions to cause the processor to:
- obtain noise characteristics for the image capture device;
- determine one or more of a maximum value and a minimum value for the content-load histogram based, at least in part, on the noise characteristics;
- limit values in the content-load histogram in accordance with the one or more of a maximum value and a minimum value to generate a second content-load histogram; and
- normalize the second content-load histogram to generate an adjusted content-load histogram.
14. The non-transitory program storage device of claim 13, wherein the instructions to cause the processor to limit and normalize are repeated a plurality of times.
15. The non-transitory program storage device of claim 13, wherein the instructions to cause the processor to determine one or more of a maximum value and a minimum value comprise instructions to cause the processor to set a minimum value equal to 1 quantization noise error level.
16. A method to generate an image using a noise-constrained tone curve, comprising:
- obtaining, from an image capture device, a first image comprising pixels wherein each pixel has a value; identifying a bin within a content-load histogram based, at least in part, on the value of the selected pixel, wherein a content-load histogram comprises a histogram in which each bin identifier corresponds to a value each pixel in the first image may assume, and whose bin values represent an amount of a specified structural information carried by the first image's pixels at a bin's corresponding bin identifier, determining a content-load value corresponding to the selected pixel, determining a weight factor for the selected pixel based, at least in part, on the selected pixel and the selected anchor pixel, updating the identified content-load histogram bin based, at least in part, on the content-load value and the weight factor;
- adjusting the content-load histogram based, at least in part, on noise characteristics of the image capture device;
- generating a tone curve based, at least in part, on the adjusted content-load histogram; and
- generating an output image based, at least in part, on the tone curve.
17. The method of claim 16, wherein selecting an anchor pixel comprises selecting each pixel, one at a time, as an anchor pixel.
18. The method of claim 16, wherein determining a content-load value comprises determining a dynamic range value in a region around the selected pixel.
19. The method of claim 16, wherein determining a content-load value comprises:
- identifying a second image comprising the same number of pixels as the first image, wherein each pixel in the second image has a value and corresponds to one pixel in the first image; and
- selecting that pixel in the second image corresponding to the selected pixel from the first image, wherein the value of the pixel selected from the second image comprises the content-load value.
20. The method of claim 16, wherein determining a weight factor comprises:
- determining a distance measure value between the selected pixel and the selected anchor pixel; and
- determining a weight factor based, at least in part, on the distance measure value.
21. The method of claim 20, wherein determining a distance measure value comprises determining a Euclidean distance between the selected pixel and the selected anchor pixel.
22. The method of claim 21, wherein determining a weight factor comprises applying the distance measure value to a Gaussian distribution.
23. The method of claim 16, wherein updating the identified content-load histogram bin comprises applying the determined weight factor to the content-load value to generate an update value.
24. The method of claim 23, wherein applying the determined weight factor to the content-load value comprises multiplying the weight factor and the content-load value.
25. The method of claim 16, wherein adjusting the content-load histogram comprises:
- obtaining noise characteristics for the image capture device;
- determining one or more of a maximum value and a minimum value for the content-load histogram based, at least in part, on the noise characteristics;
- limiting values in the content-load histogram in accordance with the one or more of a maximum value and a minimum value to generate a second content-load histogram; and
- normalizing the second content-load histogram to generate an adjusted content-load histogram.
26. The method of claim 25, wherein limiting and normalizing are repeated a plurality of times.
27. A non-transitory program storage device, readable by a processor and comprising instructions stored thereon to cause the processor to:
- obtain, from an image capture device, a first image comprising a first plurality of pixels, wherein each pixel has a value;
- for each of the first plurality of pixels— select a pixel, wherein the pixel identifies a bin in a histogram, and update the identified bin based, at least in part, on the value of the pixel;
- obtain noise characteristics of the image capture device;
- constrain maximum and minimum values of the histogram based, at least in part, on the noise characteristics;
- normalize the histogram following application of the constraints based on the image capture device's noise characteristics;
- generate a tone curve based, at least in part, on the constrained and normalized histogram; and
- generate an output image based, at least in part, on the tone curve.
28. The non-transitory program storage device of claim 27, wherein the instructions to cause the processor to constrain maximum and minimum values of the histogram comprise instructions to establish a maximum histogram value that is based, at least in part, on noise quantization of the image capture device.
29. The non-transitory program storage device of claim 28, further comprising instructions to cause the processor to establish a minimum histogram value that is based, at least in part, on a quantization step size determined after a multi-exposure fuse operation.
30. An electronic device, comprising:
- an image capture sensor;
- a memory communicatively coupled to the image capture sensor and having stored therein computer program code;
- a display element communicatively coupled to the memory; and
- one or more processors communicatively coupled to the image capture sensor, the memory, and the display element, the one or more processors adapted to execute the computer program code to: obtain, from the memory, a first image comprising pixels wherein each pixel has a value, select one or more anchor pixels; select, for each of the one or more anchor pixels, every pixel in the first image, one at a time, and with each selected pixel— identify a bin within a content-load histogram based, at least in part, on the value of the selected pixel, wherein a content-load histogram comprises a histogram in which each bin identifier corresponds to a value each pixel in the first image may assume, and whose bin values represent an amount of a specified structural information carried by the first image's pixels at a bin's corresponding bin identifier, determine a content-load value corresponding to the selected pixel, determine a weight factor for the selected pixel base, at least in part, on the selected pixel and the selected anchor pixel, update the identified content-load histogram bin based, at least in part, on the content-load value and the weight factor; adjust the content-load histogram based, at least in part, on noise characteristics of the image capture sensor; generate a tone curve based, at least in part, on the adjusted content load histogram; and store in the memory an output image based, at least in part, on the tone curve.
31. The electronic device of claim 30, wherein the instructions to cause the one or more processors to obtain a first image comprise instructions to cause the one or more processors to obtain a first luminance image from the memory.
32. The electronic device of claim 30, wherein the instructions to cause the one or more processors to select an anchor pixel comprise instructions to cause the one or more processors to select each pixel in the first image, one at a time, as an anchor pixel.
33. The electronic device of claim 30, wherein the instructions to cause the one or more processors to determine a content-load value comprise instructions to cause the one or more processors to determine a dynamic range value in a region around the selected pixel.
34. The electronic device of claim 33, wherein the instructions to cause the one or more processors to determine a dynamic range value comprise instructions to cause the one or more processors to calculate a dynamic range value in real-time.
35. The electronic device of claim 30, wherein the instructions to cause the one or more processors to determine a content-load value comprise instructions to cause the one or more processors to:
- identify a second image comprising the same number of pixels as the first image, wherein each pixel in the second image has a value and corresponds to one pixel in the first image; and
- select that pixel in the second image corresponding to the selected pixel from the first image, wherein the value of the pixel selected from the second image comprises the content-load value.
36. The electronic device of claim 30, wherein the instructions to cause the one or more processors to determine a weight factor comprise instructions to cause the one or more processors to:
- determine a distance measure value between the selected pixel and the selected anchor pixel; and
- determine a weight factor based, at least in part, on the distance measure value.
37. The electronic device of claim 36, wherein the instructions to cause the one or more processors to determine a distance measure value comprise instructions to cause the one or more processors to determine a Euclidean distance between the selected pixel and the selected anchor pixel.
38. The electronic device of claim 37, wherein the instructions to cause the one or more processors to determine a weight factor comprise instructions to cause the one or more processors to apply the distance measure value to a Gaussian distribution.
39. The electronic device of claim 30, wherein the instructions to cause the one or more processors to update the identified content-load histogram bin comprise instructions to cause the one or more processors to apply the determined weight factor to the content-load value to generate an update value.
40. The electronic device of claim 39, wherein the instructions to cause the one or more processors to apply the determined weight factor to the content-load value comprise instructions to cause the one or more processors to multiply the weight factor and the content-load value.
41. The electronic device of claim 39, wherein the instructions to cause the one or more processors to update the identified content-load histogram bin comprise instructions to cause the one or more processors to add the update value to the identified content-load histogram bin.
42. The electronic device of claim 30, wherein the instructions to cause the one or more processors to adjust the content-load histogram comprise instructions to cause the one or more processors to:
- obtain noise characteristics for the image capture sensor;
- determine one or more of a maximum value and a minimum value for the content-load histogram based, at least in part, on the noise characteristics;
- limit values in the content-load histogram in accordance with the one or more of a maximum value and a minimum value to generate a second content-load histogram; and
- normalize the second content-load histogram to generate an adjusted content-load histogram.
43. The electronic device of claim 42, wherein the instructions to cause the one or more processors to limit and normalize are repeated a plurality of times.
44. The electronic device of claim 42, wherein the instructions to cause the one or more processors to determine one or more of a maximum value and a minimum value comprise instructions to cause the one or more processors to set a minimum value equal to 1 quantization noise error level.
4731671 | March 15, 1988 | Alkofer |
6717698 | April 6, 2004 | Lee |
7506135 | March 17, 2009 | Mimar |
20030021488 | January 30, 2003 | Shaw et al. |
20040057632 | March 25, 2004 | Gindele |
20060182361 | August 17, 2006 | Ptucha et al. |
20070014470 | January 18, 2007 | Sloan |
20070252834 | November 1, 2007 | Fay |
20090100339 | April 16, 2009 | Wharton-Ali et al. |
20100157078 | June 24, 2010 | Attanossov |
20100226547 | September 9, 2010 | Criminisi |
20120170842 | July 5, 2012 | Liu |
20120188247 | July 26, 2012 | Cheung |
20130329093 | December 12, 2013 | Zhang et al. |
20140247870 | September 4, 2014 | Mertens |
- “FAQ—HDR Images for Photography.” http//:www.hdrsoft.com/resources/dri.html Accessed Mar. 16, 2012. pp. 1-8.
- Durand, Fredo and Julie Doresy. “Fast Bilateral Filtering for the Display of High-Dynamic-Range Images.” http//:people.csail.mit.edu/fredo/PUBLI/Siggraph2002/ Accessed Mar. 16, 2012. pp. 1-7.
- Zhang, Xuemei and Steven Hordley. “Noise-Constrained Tone Curve Generation.” Apple inc.—White Paper. pp. 3, Mar. 13, 2012.
Type: Grant
Filed: Jun 6, 2012
Date of Patent: Aug 18, 2015
Patent Publication Number: 20130329093
Assignee: Apple Inc. (Cupertino, CA)
Inventors: Xuemei Zhang (Mountain View, CA), Steven D. Hordley (Los Gatos, CA), Graham Finlayson (Norwich)
Primary Examiner: Tahmina Ansari
Application Number: 13/490,311
International Classification: G06K 9/48 (20060101); H04N 5/235 (20060101);