Compressed high dynamic range video

- Contrast, Inc.

Methods for the real-time, reversible compression of high dynamic range videos, allowing high dynamic range videos to be broadcast through standard dynamic range channels. The methods include streaming pixel values from at least one image sensor through a pipeline on a processor. The pipeline includes an HDR function that combines the streaming pixel values in real-time into an HDR stream and a transfer function that converts the HDR stream to an SDR video stream. The method further includes transmitting the SDR video stream to a receiver in real time for display as a video.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
TECHNICAL FIELD

The disclosure relates to high dynamic range video and related methods and devices.

BACKGROUND

Humans can see brightness variations over a dynamic range of about 10,000:1 simultaneously within a scene. Some cameras are able to capture images over that dynamic range and preserve the dynamic range by encoding the images digitally by storing pixel brightness values 14 bits per color. Human visual perception has a non-linear relationship to brightness, and brightness is often encoded digitally using a non-linear power law known as a gamma function.

Early research and technology such as cathode ray tubes suggested that video signals should be digitized using a gamma function using 8 bits per pixel for brightness. Accordingly, much contemporary digital video for consumer equipment and computer graphics uses 8 bit encoding. Existing television, cable, and satellite channels typically operate with 8 bit signals defining the standard dynamic range. Thus, while some modern cameras capture images at a higher dynamic range, video is typically broadcast and displayed at 8 bit standard dynamic range.

SUMMARY

The invention provides methods for the real-time, reversible compression of high dynamic range videos, allowing high dynamic range videos to be broadcast through standard dynamic range channels. A high-dynamic range video signal is streamed through a pipeline that processes the video stream in a pixel-by-pixel manner as the pixels stream through the pipeline. The pipeline includes a transfer function that that decreases a number of bits of each pixel to a standard dynamic range, in a reversible manner, to allow the high dynamic range video to be transmitted over existing, standard dynamic range channels. The transfer function may be an opto-electrical transfer function (OETF) such as S-Log that converts pixels from greater than 8 bits per color to at most 8 bits per color, and the inverse function may be applied by a receiver such as a display device to display the video with high dynamic range.

The processer process pixels in a pipeline in real-time to form the high dynamic range video. Pipeline processing of the pixels as they stream through the processor provides a high dynamic range (HDR) pixel stream and also uses the OETF to decrease the number of bits per pixel as the pixels stream through the pipeline in a frame-independent manner, allowing the live video to be captured and broadcast for display in real-time. Because the transfer function reversibly decreases the number of bits per pixel, methods are useful for reversibly compressing live high dynamic range video signal for transmission over standard dynamic range channels. After transmission over standard dynamic range channels, the compressed signal may be displayed at the low dynamic range or transformed back into, and displayed at, the high dynamic range.

In preferred embodiments, the pipeline processing is performed on a processor of a video camera that uses one or more sensors to capture incoming light. The sensors may use an optical splitting system with a beam splitter that splits incoming light onto the multiple image sensors. By using a beamsplitter with an asymmetric split ratio, the multiple image sensors capture images that are identical except for light level. That is, one of the sensors captures a bright version of an image while another captures the same image, at a lower light level. Those images are streamed from the sensors onto the processor where they are merged, in pipeline processing, to form a high dynamic range image. Portions of the bright image that are saturated can be replaced with corresponding portions from the less-bright image. A pipeline on the processing device can do this in a streaming, pixel-by-pixel manner so that a high dynamic range (HDR) stream is formed in real time as the camera is used to capture a scene. The pipeline also applies an opto-electrical transfer function (OETF) to the HDR stream to reversibly convert them into a standard dynamic range (SDR) video stream (e.g., 8 bit), which can be transmit or broadcast over existing 8 bit infrastructure. Thus embodiments of the disclosure include methods and devices for capturing a video signal and streaming it as pixel values in real time as an HDR stream, applying a transfer function such as s-log to compress the HDR stream into an SDR video stream, and broadcasting the compressed SDR video stream in real time with the capturing.

In certain aspects, the disclosure provides a video processing method. The method includes streaming pixel values from at least one image sensor through a pipeline on a processor. The pipeline includes an HDR function that combines the streaming pixel values in real-time into an HDR stream and a transfer function that converts the HDR stream to an SDR video stream. The method further includes transmitting the SDR video stream to a receiver in real time for display as a video.

Optionally, the method includes applying an inverse of the transfer function to the SDR video stream within the receiver to de-compress the video stream back into an HDR video for display, where the receiver is an HDR display device. Preferably, the HDR function and the transfer function are done by pipeline processing on a pixel-by-pixel basis while the streaming and transmitting steps are performed simultaneously.

Where the image sensor and processor are provided within a video camera, the pipeline processing allows the camera to capture the video of a live event for display by the receiver as live playback. The receiver may itself be an HDR display device. Preferably, the HDR stream comprises HDR pixels that use greater than 8 bits per color (e.g., 14 bit) and the SDR video stream comprises SDR pixels that use no more than 8 bits per pixel value (e.g., 8 bit). The transfer function may be an opto-electric transfer function such as S-Log 1, S-Log 2, or S-Log 3, and the method may include providing the receiver with an electro-optical transfer function (EOTF, an inverse of the OETF) for use by the receiver to convert the SDR video stream into an HDR video stream. The transfer function may be any suitable EOTF such as, for example, Filestream, S-Log, Panalog, Log C, and Canon Log.

Embodiments of the methods may include receiving incoming light through a lens and at least one beamsplitter, in which the beamsplitter splits the incoming light onto multiple image sensors such that the multiple image sensors each receive images that are identical but for light level. The HDR function in the pipeline may include a kernel operation that identifies saturated pixel values and a merge operation that merges the pixel values to produce the HDR stream. Merging may exclude at least some of the saturated pixel values from the HDR video signal.

Preferably, the multiple image sensors include at least a high exposure (HE) sensor and a middle exposure (ME) sensor, and wherein merging the sequences includes using HE pixel values that are not saturated and ME pixel values corresponding to the saturated pixel values. By using, for example, an asymmetric beamsplitter, at least 95% of the incoming light impinges on, or is captured by, the multiple image sensors. In some embodiments, streaming the pixel values through the kernel operation includes examining values from a neighborhood of pixels surrounding a first pixel on the HE sensor, finding saturated values in the neighborhood of pixels, and using information from a corresponding neighborhood on the ME sensor to estimate a value for the first pixel. Real-time, pipeline processing provides that at least some pixel values may be converted to SDR by the transfer function while the image sensors are still converting the incoming light into the pixel values for the same image, or frame. Sequences of the pixel values may be streamed through the processing device and merged without waiting to receive pixel values for an entire image from all pixel sites on the image sensors,

In related aspects, the disclosure provides a high dynamic range video camera. The camera includes at least one image sensor coupled to a processor and a pipeline on the processor that processes streams of pixel values from the sensor. The pipeline includes an HDR function that combines the streaming pixel values in real-time into an HDR video stream and a transfer function that converts the HDR video stream to an SDR video stream. Preferably, the HDR video stream comprises HDR pixel values with light levels encoded at greater than 8 bits per color, per pixel, and the SDR video stream comprises SDR pixel values with light levels encoded at no greater than 8 bits per color, per pixel. The HDR function and the transfer function may be performed in real-time on the streaming pixel values such that the SDR video stream can be received and displayed by a receiver as a live broadcast.

In preferred embodiments, the image sensor is provided in an optical splitting system that comprises a lens, at least one beamsplitter, and a plurality of image sensors. The plurality of image sensors may include at least a high exposure (HE) sensor and a middle exposure (ME) sensor, with the HE sensor, the ME sensor, the lens and the beamsplitter arranged to receive an incoming beam of light and split the beam of light into at least a first path that impinges and HE sensor and a second path that impinges on the ME sensor. The beamsplitter directs a majority of the light to the first path and a lesser amount of the light to the second path. The first path and the second path impinge on the HE and the ME sensor, respectively, to generate images that are optically identical but for light level.

In certain embodiments, the HDR function in the pipeline includes a kernel operation that identifies saturated pixel values and a merge module to merge the pixel values to form the HDR video stream. The kernel operation may be designed to operate on pixel values as they stream from each of the plurality of image sensors by examining, for a given pixel on the HE sensor, values from a neighborhood of pixels surrounding the given pixel, finding saturated values in the neighborhood of pixels, and using information from a corresponding neighborhood on the ME sensor to estimate a value for the given pixel. The pipeline may include (e.g., in the following order) a sync module to synchronize the pixel values as the pixel values stream onto the processing device from the plurality of image sensors; the HDR function comprising the kernel operation and the merge module; a demosaicing module; a tone-mapping operator; and the transfer function. The transfer function preferably includes an opto-electrical transfer function (OETF) such as an S-Log function.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 diagrams a video processing method.

FIG. 2 diagrams components of a high dynamic range video camera.

FIG. 3 shows the optical splitting system according to certain embodiments.

FIG. 4 shows the processor of the camera.

FIG. 5 shows transfer functions that may be used with methods of the disclosure.

FIG. 6 shows operation of the sync module.

FIG. 7 illustrates how the pixel values are presented to the kernel operation.

 DETAILED DESCRIPTION

The disclosure provides methods and devices that apply an EOTF (electro-optic transfer function) to a stream of pixels, to convert a video stream with for example more than 8 bits per color to a video stream with 8 bits per color. The EOTF provides a modified gamma function, such as may be used in an S-Log function. Methods may further include applying the inverse EOTF at the other end of the transmission to convert, e.g., the 8 bits per color video signal back into a signal with greater than 8 bits per color.

For example, a video stream may have 14 bits per color. Methods of the disclosure apply a specific S-log transfer function to that stream (preferably in a pipeline process, pixel-by-pixel) to convert each pixel to 8-bits per color. Then the 8-bit video signal may be sent via a standard 8-bit television, cable, or satellite channel to a receiver, where the receiver would apply an inverse of the transfer function to produce a 14-bit signal which could then be displayed on a 14-bit display. Thus, methods and devices of the disclosure are useful for transmitting HDR video over standard TV channels.

FIG. 1 diagrams a video processing method 101. The method 101 includes streaming 107 pixel values from at least one image sensor. The pixel are processed 111 through a pipeline on a processor. The pipeline includes an HDR function that combines 115 the streaming pixel values in real-time into an HDR stream and a transfer function that converts 119 the HDR stream to an SDR video stream. The method further includes transmitting 125 the SDR video stream to a receiver in real time for display as a video. The pipeline is preferably embodied within a processor which thus includes the HDR function and the transfer function. The processor may be connected to one or more image sensors within an apparatus such as a video camera according to certain embodiment.

FIG. 2 diagrams components of an apparatus 201 (e.g., a high dynamic range video camera) that includes at least one image sensor 211 coupled to a processor 219 such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A plurality of image sensors 265 are coupled to the processing device 219. The apparatus 201 is configured to stream pixel values 501 from each of the plurality of image sensors 265 in a frame independent-manner through a pipeline 231 on the processing device 219. The pipeline 231 includes an HDR function that combines the streaming pixel values in real-time into an HDR video stream and a transfer function that converts the HDR video stream to an SDR video stream. Various components of the apparatus 201 may be connected via a printed circuit board 205. The apparatus 201 may also include memory 221 and optionally a processor 227 (such as a general-purpose processor like an Intel chip). Apparatus 201 may further include one or more of an input-output device 239 or a display 267. The at least one sensor 211 may be provided as part of an optical splitting system that includes the plurality 265 of sensors.

FIG. 3 shows the optical splitting system 300 and an arrangement of the plurality 265 of sensors. The plurality 265 of sensors preferably include at least a high exposure (HE) sensor 213 and a middle exposure (ME) sensor 211. Each image sensor may have its own color filter array 307. The color filter arrays 307 may operate as a Bayer filter such as a repeating grid of red, green, blue, green filters. The optical splitting system 300 preferably includes a lens 311 and at least one beamsplitter 301. The HE sensor 213, the ME sensor 211, the lens 311 and the at least one beamsplitter 301 are arranged to receive an incoming beam of light 305 and split the beam of light 305 into at least a first path that impinges and HE sensor 213 and a second path that impinges on the ME sensor 211. The apparatus 201 uses a set of partially-reflecting surfaces to split the light from a single photographic lens 311 so that it is focused onto three imaging sensors simultaneously. In a preferred embodiment, the light is directed back through one of the beamsplitters a second time, and the three sub-images are optically identical except for their light levels. The optical splitting system 300 allows the apparatus 201 to capture HDR images using most of the light entering the camera. Generally, the image sensors will include analog-to-digital converters and when light 305 impinges upon the sensors 211, 213, pixel values will stream onto the processor 219.

FIG. 4 shows the processor 219 on the apparatus 201. Pixel values from the sensors stream through the pipeline 231 on the processing device 219. The pipeline 231 may include a sync module 405 to synchronize the pixel values 501 as the pixel values 501 stream onto the processing device 219 from the plurality of image sensors 265. The pipeline includes an HDR function 411 that combines the streaming pixel values in real-time into an HDR video stream, and a transfer function 423 that converts the HDR video stream to an SDR video stream. In preferred embodiments, the HDR function includes a kernel operation 413 and a merge module 421. The kernel operation 413 identifies saturated pixel values 501 and the merge module merges the pixel values 501 to produce an HDR stream. The pipeline may further include a demosaicing module 425, a tone-mapping operator 427, one or more auxiliary module 431 such as a color-correction module, or combinations thereof.

In preferred embodiments, the kernel operation 413 operates on pixel values as they stream from each of the plurality 265 of image sensors by examining, for a given pixel on the HE sensor, values from a neighborhood of pixels surrounding the given pixel, finding saturated values in the neighborhood of pixels, and using information from a corresponding neighborhood on the ME sensor to estimate a value for the given pixel. Alternatively, the pipeline 231 may include—in the order in which the pixel values flow: a sync module 405 to synchronize the pixel values as the pixel values stream onto the processing device from the plurality of image sensors; the HDR function 411 comprising the kernel operation 413 and the merge module 421; a demosaicing module 425; a tone-mapping operator 427; and the transfer function 423.

Preferably, the transfer function 423 provides an opto-electrical transfer function (OETF) such as an S-Log function.

FIG. 5 describes two exemplary transfer functions that may be used with methods and devices of the disclosure. The curves depicted as S-Log 1 and S-Log 2 operate as opto-electrical transfer functions and are used to carry the full range of the sensor-captured HDR signal through to subsequent processes. Compared to the ITU 709 curve, the S-log transfer functions cover a much greater range of brightness, for brighter signals within the 512 to 1024 data values as depicted. Other transfer functions are within the scope of the invention. The depicted S-Log functions illustrate the principle. For example, the transfer function may include or be based on Filestream from Thomson, S-Log from Sony, Panalog from Panavision, Log C from Arri, or Canon Log from Canon.

The inverse function may be used and provided for use by the receiver to convert the SDR video stream into an HDR video stream, e.g., where the receiver comprises a high-dynamic range display device. The ITU-R BT 709 curve was designed to produce a uniform perception of video noise in an analog signal. However, in quantizing a video signal, it may be more important to avoid contouring and match the human visual systems brightness perception than to avoid noise. To avoid contouring, detecting the difference between adjacent levels is important and is governed by Weber's law, which states that detectable difference in brightness is proportional to the brightness. Weber's law suggests that a logarithmic transfer function optimizes dynamic range while rendering quantization steps imperceptible. Transfer functions according to the disclosure extend dynamic range with a smooth curve. Those functions at low brightness values are (by design) very similar to the Rec 709 curve. Over relevant ranges, preferred transfer functions approximate logarithmic curves. Transfer functions of the disclosure preferably do not result in visible contouring. Furthermore, if the peak brightness of a display using the proposed OETF is approximately a few hundred cd/rn2, then the OETF approximately corresponds to the sensitivity of the eye. Research suggests that an 8 bit version of such a transfer function should be able to produce a higher dynamic range image without visible artifacts. It should be noted that an 8 bit HDR signal would include some exposure latitude to support post processing such as grading. Application of an 8 bit transfer function via methods and devices of the disclosure will allow a higher dynamic range image to be transferred to a display via an 8 bit interface. For background, see UK Patent Publication No. GB 252047 A and Borer, 2014, Non-linear opto-electrical transfer functions for high dynamic range television, Research and Development White Paper, British Broadcasting Corporation (24 pages), both incorporated by reference. According to research and theory, using a logarithmic or approximately logarithmic transfer function, as compared to ITU-R BT 709 as shown in FIG. 5, an HDR stream is convertible (reversibly) into an 8 bit video signal that can be transmitted over 8 bit channels and converted back into HDR by the inverse function within a display/receiver device, which can then display the HDR video.

By using the pipeline processing described above, in which incoming light is passed through a single lens and split onto multiple sensors to begin streaming parallel streams of HE and ME (and optionally LE) pixels into an HDR function (e.g., one or more blocks on an FPGA) and a transfer function, 8 bit HDR video can be captured, broadcast, and displayed live, meaning that live broadcasts can be performed according to existing understandings of the meanings of live broadcasting or in real-time, and in HDR. In preferred embodiments, the HDR video stream comprises HDR pixel values with light levels encoded at greater than 8 bits per color, per pixel, and the SDR video stream comprises SDR pixel values with light levels encoded at no greater than 8 bits per color, per pixel. The HDR function and the transfer function are performed in real-time on the streaming pixel values such that the SDR video stream can be received and displayed by a receiver as a live broadcast.

The output that gets broadcast is an HDR video signal because the method 101 and the apparatus 201 use multiple sensors at different exposure levels to capture multiple isomorphic images (i.e., identical but for light level) and merge them. Data from a high exposure (HE) sensor are used where portions of an image are dim and data from a mid-exposure (ME) (or lower) sensor are used where portions of an image are more brightly illuminated. The method 101 and apparatus 201 merge the HE and ME (and optionally LE) images to produce an HDR video signal. Specifically, the method 101 and the apparatus 201 identify saturated pixels in the images and replace those saturated pixels with values derived from sensors of a lower exposure. In preferred embodiments, a first pixel value from a first pixel on one of the image sensors is identified as saturated if it is at least 90% of a maximum possible pixel value. The HDR function and the transfer function are done by pipeline processing on a pixel-by-pixel basis while the streaming and transmitting steps are performed simultaneously so that the camera captures the video of a live event for display by the receiver as live playback.

In the HDR stream, HDR pixels have more than 8 bits per color and after application of the transfer function the SDR video stream comprises SDR pixels that use no more than 8 bits per pixel value. The transfer function may be a block on the FPGA 219 that applies an optical-electrical transfer function (OETF) (modifying the gamma function) to a stream of pixels, to convert a video stream with >8 bits per color to a video stream with 8 bits per color. Methods may include applying the inverse EOTF at the other end of the transmission to convert the 8 bits per color video signal into a full >8 bits signal.

For example, a video stream may have 14 bits per color, and be subject to a specific S-log transfer function (in a pipeline process, pixel-by-pixel) to convert each pixel to 8-bits per color. Then this 8-bit video signal is sent via a standard 8-bit television, cable, or satellite channel to a receiver, where the receiver would apply an inverse EOTF to produce a 14-bit signal which is displayed on a special 14-bit display. In some embodiments, the transfer function applies s-log compression, in order to compress HDR video data in the camera (which may capture at 14- or even 16-bits per color per pixel) down to 8 bits per color per pixel. The resultant 8 bit signal is transmit over a typical 8-bit broadcast TV channel (cable, satellite, over-the-air). The inverse process (s-log expansion) restores the video to its original 14- or 16-bits per color per pixel for display on an HDR TV monitor.

Various components of the apparatus 201 may be connected via a printed circuit board 205. The apparatus 201 may also include memory 221 and optionally a processor 227 (such as a general-purpose processor like an Intel chip). Apparatus 201 may further include one or more of an input-output device 239 or a display 267. Memory can include RAM or ROM and preferably includes at least one tangible, non-transitory medium. A processor may be any suitable processor known in the art, such as the processor sold under the trademark XEON E7 by Intel (Santa Clara, CA) or the processor sold under the trademark OPTERON 6200 by AMD (Sunnyvale, CA). Input/output devices according to the invention may include a video display unit (e.g., a liquid crystal display or LED display), keys, buttons, a signal generation device (e.g., a speaker, chime, or light), a touchscreen, an accelerometer, a microphone, a cellular radio frequency antenna, port for a memory card, and a network interface device, which can be, for example, a network interface card (NIC), Wi-Fi card, or cellular modem. The apparatus 201 may include or be connected to a storage device 241. The plurality of sensors are preferably provided in an arrangement that allows multiple sensors 265 to simultaneously receive images that are identical except for light level.

In some embodiments, the optical splitting system 300 uses two uncoated, 2-micron thick plastic beamsplitters that rely on Fresnel reflections at air/plastic interfaces so their actual transmittance/reflectance (T/R) values are a function of angle. Glass is also a suitable option. In one embodiment, the first beamsplitter 301 is at a 45° angle and has an approximate T/R ratio of 92/8, which means that 92% of the light from the camera lens 311 is transmitted through the first beamsplitter 301 and focused directly onto the high-exposure (HE) sensor 213. The beamsplitter 301 reflects 8% of the light from the lens 311 upwards, toward the second uncoated beamsplitter 319, which has the same optical properties as the first but is positioned at a 90° angle to the light path and has an approximate T/R ratio of 94/6.

Of the 8% of the total light that is reflected upwards, 94% (or 7.52% of the total light) is transmitted through the second beamsplitter 319 and focused onto the medium-exposure (ME) sensor 211. The other 6% of this upward-reflected light (or 0.48% of the total light) is reflected back down by the second beamsplitter 319 toward the first beamsplitter 301 (which is again at 45°), through which 92% (or 0.44% of the total light) is transmitted and focused onto the low-exposure (LE) sensor 261. With this arrangement, the HE, ME and LE sensors capture images with 92%, 7.52%, and 0.44% of the total light gathered by the camera lens 311, respectively. Therefore, the HE and ME exposures are separated by 12.2×(3.61 stops) and the ME and LE are separated by 17.0×(4.09 stops), which means that this configuration is designed to extend the dynamic range of the sensor by 7.7 stops.

This beamsplitter arrangement makes the apparatus 201 light efficient: a negligible 0.04% of the total light gathered by the lens 311 is wasted. It also allows all three sensors to “see” the same scene, so all three images are optically identical except for their light levels. Of course, the ME image has undergone an odd number of reflections and so it is flipped left-right compared to the other images, but this is fixed easily in software. In preferred embodiments, the three sensors are not gen-locked and instead independently stream incoming pixel values directly into a pipeline that includes a synchronization module. This avoids the requirement for a clock or similar triggering apparatus.

Thus it can be seen that the beamsplitter 301 directs a majority of the light to the first path and a lesser amount of the light to the second path. Preferably, the first path and the second path impinge on the HE and the ME sensor 211, respectively, to generate images that are optically identical but for light level. In the depicted embodiment, the apparatus 201 includes a low exposure (LE) sensor.

In preferred embodiments, the HE sensor 213, the ME sensor 211, and the LE sensor 261 are not gen-locked. Pixel values stream from the sensors in sequences directly to the processing device 219. Those sequences may be not synchronized as they arrive onto the processing device 219.

The method 101 may include receiving 107 incoming light through the lens 311 and splitting 113 the light via at least one beamsplitter 301 onto the multiple image sensors, wherein at least 99% of the incoming beam of light 305 is captured by the multiple image sensors.

The apparatus 201 (1) captures optically-aligned, multiple-exposure images simultaneously that do not need image manipulation to account for motion, (2) extends the dynamic range of available image sensors (by over 7 photographic stops in our current prototype), (3) is inexpensive to implement, (4) utilizes a single, standard camera lens 311, (5) efficiently uses the light from the lens 311, and (6) applies an 8-bit EOTF to provide a digital video in real time (by pipeline processing) with no visible contouring and high dynamic range.

The method 101 preferably (1) combines images separated by more than 3 stops in exposure, (2) spatially blends pre-demosaiced pixel data to reduce unwanted artifacts, (3) produces an HDR stream that is radiometrically correct, (4) uses the highest-fidelity (lowest quantized-noise) pixel data available, and (5) applies an 8-bit S-log or similar transfer function. The apparatus 201 can work with a variety of different sensor types and uses an optical architecture based on beamsplitters located between the camera lens and the sensors.

FIG. 6 shows operation of the sync module 405 to synchronize the pixel values 501 as the pixel values 501 stream onto the processing device 219 from the plurality of image sensors 265. The HE_1 pixel value and ME_1 pixel value are arriving at the sync module 405 approximately simultaneously. However, HE_2 pixel value will arrive late compared to ME_2, and the entire sequence of LE pixel values will arrive late. The sync module 405 can contain small line buffers that circulate the early-arriving pixel values and release them simultaneous with the corresponding later-arriving pixel values. The synchronized pixel values then stream through the pipeline 231 to the kernel operation 413.

FIG. 7 illustrates how the pixel values are presented to the kernel operation 413. The top part of the figures shows the HE sensor 213. Each square depicts one pixel of the sensor 213. A heavy black box with a white center is drawn to illustrate a given pixel 615 for consideration and a neighborhood 601 of pixels surrounding the given pixel 615. The heavy black box would not actually appear on a sensor 213 (such as a CMOS cinematic camera sensor)—it is merely drawn to illustrate what the neighborhood 601 includes and to aid understanding how the neighborhood 601 appears when the sequences 621 of pixel values 501 are presented to the kernel operation 413. The sequences 621 of pixel values stream into the kernel operation 413 after the sync module 405. Pixel values 501 from the neighborhood 601 of pixels on the sensor 213 are still “blacked out” to aid illustration. The given pixel 615 under consideration can be spotted easily because it is surrounded on each side by two black pixels from the row of pixels on the sensor. There are two sequences 621, one of which comes from the depicted HE sensor 213 and one of which originates at the ME sensor 211.

Streaming the pixel values 501 through the kernel operation 413 includes examining values from a neighborhood 601 of pixels surrounding a first pixel 615 on the HE sensor 213, finding saturated values in the neighborhood 601 of pixels, and using information from a corresponding neighborhood 613 from the ME sensor 211 to estimate a value for the first pixel 615. This will be described in greater detail below. To accomplish this, the processing device must make comparisons between corresponding pixel values from different sensors. It may be useful to stream the pixel values through the kernel operation in a fashion that places the pixel under consideration 615 adjacent to each pixel from the neighborhood 601 as well as adjacent to each pixel from the corresponding neighborhood on another sensor. For background, see Bravo, 2011, Efficient smart CMOS camera based on FPGAs oriented to embedded image processing, Sensors 11:2282-2303; Lyu, 2014, A 12-bit high-speed column parallel two-step single-slope analog-to-digital converter (ADC) for CMOS image sensors, Sensors 14:21603-21625; Ab Rahman, 2011, Pipeline synthesis and optimization of FPGA-based video processing applications with CAL, EURASIP J Image Vid Processing 19:1-28; Schulte, 2016, HDR Demystified: Emerging UHDTV systems, SpectraCal 1-22; U.S. Pub. 2017/0237890; U.S. Pub. 2017/0238029; and U.S. Pat. No. 8,982,962 to Gish, the contents of each of which are incorporated by reference.

INCORPORATION BY REFERENCE

References and citations to other documents, such as patents, patent applications, patent publications, journals, books, papers, web contents, have been made throughout this disclosure. All such documents are hereby incorporated herein by reference in their entirety for all purposes.

EQUIVALENTS

Various modifications of the invention and many further embodiments thereof, in addition to those shown and described herein, will become apparent to those skilled in the art from the full contents of this document, including references to the scientific and patent literature cited herein. The subject matter herein contains important information, exemplification and guidance that can be adapted to the practice of this invention in its various embodiments and equivalents thereof.

Claims

1. A video processing method comprising:

streaming pixel values from at least one image sensor through a pipeline on a processor wherein the pipeline includes an HDR function that combines the streaming pixel values mosaiced pixel data in real-time into an HDR stream, and a reversible logarithmic transfer function that reversibly converts the HDR stream to a mosaiced SDR video stream in a pipeline process whereby the transfer function operates on a pixel-by-pixel basis as the pixel values stream through the processor to convert each pixel to 8-bits per color, wherein the transfer function provides brightness values that exceed Rec 709;
demosaicing the mosaiced SDR video stream to provide an SDR video stream; and
transmitting the SDR video stream to a receiver in real time for display as a video.

2. The method of claim 1, wherein the transfer function outputs data values between about 512 and about 1024 for a range of input brightness values that exceed Rec 709 and wherein the HDR function and the transfer function are done while the streaming and transmitting steps are performed simultaneously to thereby capture the video of a live event for display by the receiver as live playback.

3. The method of claim 1, wherein the HDR stream comprises HDR pixels that use greater than 8 bits per color and the SDR video stream comprises SDR pixels that use no more than 8 bits per pixel value.

4. The method of claim 1, wherein the transfer function comprises an opto-electrical transfer function, and the method further comprises providing the receiver with an electro-optical transfer function for use by the receiver to convert the SDR video stream into an HDR video stream.

5. The method of claim 4, wherein the receiver comprises a high-dynamic range display device.

6. The method of claim 1, wherein the transfer function comprises an S-Log gamma curve.

7. The method of claim 1, wherein the transfer function compresses the HDR stream down to 8 bits per color per pixel in the SDR video stream.

8. The method of claim 7, further comprising receiving incoming light through a lens and at least one beamsplitter that splits the incoming light onto multiple image sensors such that the multiple image sensors each receive images that are identical but for light level.

9. The method of claim 8, wherein the pipeline includes a kernel operation that identifies saturated pixel values and a merge operation that merges the pixel values to produce the HDR stream, wherein the transfer function applies an opto-electrical transfer function (OETF) to pixel values in the HDR stream.

10. The method of claim 9, wherein the multiple image sensors include at least a high exposure (HE) sensor and a middle exposure (ME) sensor, and wherein merging the sequences includes using HE pixel values that are not saturated and ME pixel values corresponding to the saturated pixel values.

11. The method of claim 9, wherein streaming the pixel values through the kernel operation includes examining values from a neighborhood of pixels surrounding a first pixel on the HE sensor, finding saturated values in the neighborhood of pixels, and using information from a corresponding neighborhood on the ME sensor to estimate a value for the first pixel to thereby spatially blend the pre-demosaiced pixel data.

12. The method of claim 9, wherein at least some pixel values are converted to SDR by the OETF while the image sensors are still converting the incoming light into the pixel values for the same image.

13. The method of claim 9, wherein sequences of the pixel values are streamed through the processing device and merged without waiting to receive pixel values for an entire image from all pixel sites on the image sensors.

14. A high dynamic range video camera comprising:

at least one image sensor coupled to a processor; and
a pipeline on the processor that processes streams of pixel values from the sensor, wherein the pipeline comprises: an HDR function that combines the streaming pixel values as pre-demosaiced data in real-time into an HDR video stream, a reversible logarithmic transfer function that reversibly converts the HDR video stream to a pre-demosaiced SDR video stream, wherein the transfer function operates pixel-by-pixel as the pixel values stream through the processor to convert each pixel to 8-bits per color, wherein the transfer function provides brightness values that exceed Rec 709, and a demosaicing module that demosaics the pre-demosaiced SDR video stream to provide an SDR video stream,
wherein the HDR function and the transfer function are performed in real-time on the streaming pixel values such that the SDR video stream can be received and displayed by a receiver as a live broadcast.

15. The camera of claim 14, wherein the HDR video stream comprises HDR pixel values with light levels encoded at greater than 8 bits per color, per pixel, and the SDR video stream comprises SDR pixel values with light levels encoded at no greater than 8 bits per color, per pixel.

16. The camera of claim 14, wherein the transfer function outputs data values between about 512 and about 1024 for a range of input brightness values that exceed Rec 709.

17. The camera of claim 14, wherein the image sensor is provided in an optical splitting system that comprises a lens, at least one beamsplitter, and a plurality of image sensors.

18. The camera of claim 17, wherein the plurality of image sensors include at least a high exposure (HE) sensor and a middle exposure (ME) sensor, wherein the HE sensor, the ME sensor, the lens and the at least one beamsplitter are arranged to receive an incoming beam of light and split the beam of light into at least a first path that impinges and HE sensor and a second path that impinges on the ME sensor.

19. The camera of claim 18, wherein the beamsplitter directs a majority of the light to the first path and a lesser amount of the light to the second path.

20. The camera of claim 19, wherein the HDR function in the pipeline includes a kernel operation that identifies saturated pixel values and a merge module to merge the pixel values to form the HDR video stream.

Referenced Cited
U.S. Patent Documents
2560351 July 1951 Kell
2642487 June 1953 Schroeder
2971051 February 1961 Back
3202039 August 1965 DeLang
3381084 April 1968 Wheeler
3474451 October 1969 Abel
3601480 August 1971 Randall
3653748 April 1972 Athey
3659918 May 1972 Tan
3668304 June 1972 Eilenberger
3720146 March 1973 Yost, Jr.
3802763 April 1974 Cook et al.
3945034 March 16, 1976 Suzuki
4009941 March 1, 1977 Verdijk et al.
4072405 February 7, 1978 Ozeki
4084180 April 11, 1978 Stoffels et al.
4134683 January 16, 1979 Goetz et al.
4268119 May 19, 1981 Hartmann
4395234 July 26, 1983 Shenker
4396188 August 2, 1983 Dreissigacker et al.
4486069 December 4, 1984 Neil et al.
4555163 November 26, 1985 Wagner
4584606 April 22, 1986 Nagasaki
4743011 May 10, 1988 Coffey
4786813 November 22, 1988 Svanberg et al.
4805037 February 14, 1989 Noble et al.
4916529 April 10, 1990 Yamamoto et al.
4933751 June 12, 1990 Shinonaga et al.
5024530 June 18, 1991 Mende
5092581 March 3, 1992 Koz
5093563 March 3, 1992 Small et al.
5134468 July 28, 1992 Ohmuro
5153621 October 6, 1992 Vogeley
5155623 October 13, 1992 Miller et al.
5194959 March 16, 1993 Kaneko et al.
5272518 December 21, 1993 Vincent
5275518 January 4, 1994 Guenther
5355165 October 11, 1994 Kosonocky et al.
5386316 January 31, 1995 Cook
5642191 June 24, 1997 Mende
5644432 July 1, 1997 Doany
5707322 January 13, 1998 Dreissigacker et al.
5729011 March 17, 1998 Sekiguchi
5734507 March 31, 1998 Harvey
5801773 September 1, 1998 Ikeda
5835278 November 10, 1998 Rubin et al.
5856466 January 5, 1999 Cook et al.
5881043 March 9, 1999 Hasegawa et al.
5881180 March 9, 1999 Chang et al.
5900942 May 4, 1999 Spiering
5905490 May 18, 1999 Shu et al.
5926283 July 20, 1999 Hopkins
5929908 July 27, 1999 Takahashi et al.
6011876 January 4, 2000 Kishner
6215597 April 10, 2001 Duncan et al.
6392687 May 21, 2002 Driscoll, Jr. et al.
6429016 August 6, 2002 McNeil
6614478 September 2, 2003 Mead
6633683 October 14, 2003 Dinh et al.
6646716 November 11, 2003 Ramanujan et al.
6674487 January 6, 2004 Smith
6747694 June 8, 2004 Nishikawa et al.
6801719 October 5, 2004 Szajewski et al.
6856466 February 15, 2005 Tocci
6937770 August 30, 2005 Oguz et al.
7068890 June 27, 2006 Soskind et al.
7084905 August 1, 2006 Nayar et al.
7138619 November 21, 2006 Ferrante et al.
7177085 February 13, 2007 Tocci et al.
7283307 October 16, 2007 Couture et al.
7336299 February 26, 2008 Kostrzewski et al.
7397509 July 8, 2008 Krymski
7405882 July 29, 2008 Uchiyama et al.
7714998 May 11, 2010 Furman et al.
7719674 May 18, 2010 Furman et al.
7731637 June 8, 2010 D'Eredita
7961398 June 14, 2011 Tocci
8035711 October 11, 2011 Liu et al.
8320047 November 27, 2012 Tocci
8340442 December 25, 2012 Rasche
8441732 May 14, 2013 Tocci
8606009 December 10, 2013 Sun
8610789 December 17, 2013 Nayar et al.
8619368 December 31, 2013 Tocci
8622876 January 7, 2014 Kelliher
8659683 February 25, 2014 Linzer
8843938 September 23, 2014 MacFarlane et al.
8982962 March 17, 2015 Alshin et al.
9087229 July 21, 2015 Nguyen et al.
9129445 September 8, 2015 Mai et al.
9131150 September 8, 2015 Mangiat et al.
9258468 February 9, 2016 Cotton et al.
9264659 February 16, 2016 Abuan et al.
9277122 March 1, 2016 Imura et al.
9459692 October 4, 2016 Li
9488984 November 8, 2016 Williams et al.
9560339 January 31, 2017 Borowski
9633417 April 25, 2017 Sugimoto et al.
9654738 May 16, 2017 Ferguson et al.
9661245 May 23, 2017 Kawano
9675236 June 13, 2017 McDowall
9677840 June 13, 2017 Rublowsky et al.
9720231 August 1, 2017 Erinjippurath et al.
9779490 October 3, 2017 Bishop
9800856 October 24, 2017 Venkataraman et al.
9904981 February 27, 2018 Jung et al.
9948829 April 17, 2018 Kiser et al.
9955084 April 24, 2018 Haynold
9974996 May 22, 2018 Kiser
9998692 June 12, 2018 Griffiths
10038855 July 31, 2018 Cote et al.
10165182 December 25, 2018 Chen
10200569 February 5, 2019 Kiser et al.
10257393 April 9, 2019 Kiser et al.
10257394 April 9, 2019 Kiser et al.
10264196 April 16, 2019 Kiser et al.
10536612 January 14, 2020 Kiser et al.
10554901 February 4, 2020 Kiser et al.
10616512 April 7, 2020 Ingle et al.
10679320 June 9, 2020 Kunz
10742847 August 11, 2020 Kiser et al.
10805505 October 13, 2020 Kiser et al.
10819925 October 27, 2020 Kiser et al.
10951888 March 16, 2021 Kiser et al.
20020014577 February 7, 2002 Ulrich et al.
20020089765 July 11, 2002 Nalwa
20030007254 January 9, 2003 Tocci
20030016334 January 23, 2003 Weber et al.
20030048493 March 13, 2003 Pontifex et al.
20030072011 April 17, 2003 Shirley
20030081674 May 1, 2003 Malvar
20030122930 July 3, 2003 Schofield et al.
20030138154 July 24, 2003 Suino
20040119020 June 24, 2004 Bodkin
20040125228 July 1, 2004 Dougherty
20040143380 July 22, 2004 Stam et al.
20040179834 September 16, 2004 Szajewski et al.
20040202376 October 14, 2004 Schwartz et al.
20050001983 January 6, 2005 Weber et al.
20050041113 February 24, 2005 Nayar et al.
20050099504 May 12, 2005 Nayar et al.
20050117799 June 2, 2005 Fuh et al.
20050151860 July 14, 2005 Silverstein et al.
20050157943 July 21, 2005 Ruggiero
20050168578 August 4, 2005 Gobush
20050198482 September 8, 2005 Cheung et al.
20050212827 September 29, 2005 Goertzen
20050219659 October 6, 2005 Quan
20060001761 January 5, 2006 Haba et al.
20060002611 January 5, 2006 Mantiuk et al.
20060061680 March 23, 2006 Madhavan et al.
20060104508 May 18, 2006 Daly et al.
20060184040 August 17, 2006 Keller et al.
20060209204 September 21, 2006 Ward
20060215882 September 28, 2006 Ando et al.
20060221209 October 5, 2006 McGuire et al.
20060249652 November 9, 2006 Schleifer
20060262275 November 23, 2006 Domroese et al.
20070025717 February 1, 2007 Raskar et al.
20070073484 March 29, 2007 Horibe
20070086087 April 19, 2007 Dent et al.
20070133889 June 14, 2007 Horie et al.
20070152804 July 5, 2007 Breed et al.
20070189750 August 16, 2007 Wong et al.
20070189758 August 16, 2007 Iwasaki
20070201560 August 30, 2007 Segall et al.
20070258641 November 8, 2007 Srinivasan et al.
20080013051 January 17, 2008 Glinski et al.
20080030611 February 7, 2008 Jenkins
20080037883 February 14, 2008 Tsutsumi et al.
20080055683 March 6, 2008 Choe et al.
20080094486 April 24, 2008 Fuh et al.
20080100910 May 1, 2008 Kim et al.
20080112651 May 15, 2008 Cho et al.
20080175496 July 24, 2008 Segall
20080198235 August 21, 2008 Chen et al.
20080198266 August 21, 2008 Kurane
20080239155 October 2, 2008 Wong et al.
20080297460 December 4, 2008 Peng et al.
20080304562 December 11, 2008 Chang et al.
20090015683 January 15, 2009 Ando
20090059048 March 5, 2009 Luo et al.
20090161019 June 25, 2009 Jang
20090213225 August 27, 2009 Jin et al.
20090225433 September 10, 2009 Tocci
20090244717 October 1, 2009 Tocci
20090290043 November 26, 2009 Liu et al.
20100098333 April 22, 2010 Aoyagi
20100100268 April 22, 2010 Zhang et al.
20100149546 June 17, 2010 Kobayashi et al.
20100172409 July 8, 2010 Reznik et al.
20100201799 August 12, 2010 Mohrholz et al.
20100225783 September 9, 2010 Wagner
20100266008 October 21, 2010 Reznik
20100271512 October 28, 2010 Garten
20100328780 December 30, 2010 Tocci
20110028278 February 3, 2011 Roach
20110058050 March 10, 2011 Lasang et al.
20110188744 August 4, 2011 Sun
20110194618 August 11, 2011 Gish et al.
20110221793 September 15, 2011 King, III et al.
20120025080 February 2, 2012 Liu et al.
20120134551 May 31, 2012 Wallace
20120147953 June 14, 2012 El-Mahdy et al.
20120154370 June 21, 2012 Russell et al.
20120179833 July 12, 2012 Kenrick et al.
20120212964 August 23, 2012 Chang et al.
20120241867 September 27, 2012 Ono et al.
20120242867 September 27, 2012 Shuster
20120260174 October 11, 2012 Imaida et al.
20120299940 November 29, 2012 Dietrich, Jr. et al.
20120307893 December 6, 2012 Reznik et al.
20130002908 January 3, 2013 Ben-Eliezer
20130021447 January 24, 2013 Brisedoux et al.
20130021505 January 24, 2013 Plowman et al.
20130027565 January 31, 2013 Solhusvik et al.
20130038689 February 14, 2013 McDowall
20130057971 March 7, 2013 Zhao et al.
20130063300 March 14, 2013 O'Regan et al.
20130064448 March 14, 2013 Tomaselli et al.
20130093805 April 18, 2013 Iversen
20130094705 April 18, 2013 Tyagi et al.
20130148139 June 13, 2013 Matsuhira
20130190965 July 25, 2013 Einecke et al.
20130194675 August 1, 2013 Tocci
20130215290 August 22, 2013 Solhusvik et al.
20130250113 September 26, 2013 Bechtel et al.
20130286451 October 31, 2013 Verhaegh
20130329053 December 12, 2013 Jones et al.
20130329087 December 12, 2013 Tico et al.
20140002694 January 2, 2014 Levy et al.
20140063300 March 6, 2014 Lin et al.
20140085422 March 27, 2014 Aronsson et al.
20140104051 April 17, 2014 Breed
20140132946 May 15, 2014 Sebastian et al.
20140152694 June 5, 2014 Narasimha et al.
20140168486 June 19, 2014 Geiss
20140184894 July 3, 2014 Motta
20140192214 July 10, 2014 Laroia
20140198187 July 17, 2014 Lukk
20140204195 July 24, 2014 Katashiba et al.
20140210847 July 31, 2014 Knibbeler et al.
20140263950 September 18, 2014 Fenigstein et al.
20140297671 October 2, 2014 Richard
20140313369 October 23, 2014 Kageyama et al.
20140321766 October 30, 2014 Jo
20140376608 December 25, 2014 Tourapis
20150077281 March 19, 2015 Taniguchi et al.
20150078661 March 19, 2015 Granados et al.
20150138339 May 21, 2015 Einecke et al.
20150151725 June 4, 2015 Clarke et al.
20150172608 June 18, 2015 Routhier et al.
20150175161 June 25, 2015 Breed
20150201222 July 16, 2015 Mertens
20150208024 July 23, 2015 Takahashi et al.
20150215595 July 30, 2015 Yoshida
20150245043 August 27, 2015 Greenebaum et al.
20150245044 August 27, 2015 Guo et al.
20150296140 October 15, 2015 Kim
20150302562 October 22, 2015 Zhai et al.
20150312498 October 29, 2015 Kawano
20150312536 October 29, 2015 Butler et al.
20160007052 January 7, 2016 Haitsuka et al.
20160007910 January 14, 2016 Boss et al.
20160026253 January 28, 2016 Bradski et al.
20160050354 February 18, 2016 Musatenko et al.
20160057333 February 25, 2016 Liu et al.
20160093029 March 31, 2016 Micovic et al.
20160163356 June 9, 2016 De Haan et al.
20160164120 June 9, 2016 Swiegers et al.
20160165120 June 9, 2016 Lim
20160173811 June 16, 2016 Oh et al.
20160191795 June 30, 2016 Han et al.
20160195877 July 7, 2016 Franzius et al.
20160205341 July 14, 2016 Hollander et al.
20160205368 July 14, 2016 Wallace
20160227193 August 4, 2016 Osterwood et al.
20160252727 September 1, 2016 Mack et al.
20160301959 October 13, 2016 Oh et al.
20160307602 October 20, 2016 Mertens
20160323518 November 3, 2016 Rivard et al.
20160344977 November 24, 2016 Murao
20160345032 November 24, 2016 Tsukagoshi
20160353123 December 1, 2016 Ninan
20160360212 December 8, 2016 Dai et al.
20160360213 December 8, 2016 Lee et al.
20160375297 December 29, 2016 Kiser
20170006273 January 5, 2017 Borer et al.
20170026594 January 26, 2017 Shida et al.
20170039716 February 9, 2017 Morris et al.
20170070719 March 9, 2017 Smolic et al.
20170084006 March 23, 2017 Stewart
20170111643 April 20, 2017 Bugdayci Sansli et al.
20170111661 April 20, 2017 Boyce
20170126987 May 4, 2017 Tan et al.
20170155818 June 1, 2017 Bonnet
20170155873 June 1, 2017 Nguyen
20170186141 June 29, 2017 Ha et al.
20170237879 August 17, 2017 Kiser et al.
20170237890 August 17, 2017 Kiser et al.
20170237913 August 17, 2017 Kiser et al.
20170238029 August 17, 2017 Kiser et al.
20170270702 September 21, 2017 Gauthier et al.
20170279530 September 28, 2017 Tsukagoshi
20170302858 October 19, 2017 Porter et al.
20170352131 December 7, 2017 Berlin et al.
20170374390 December 28, 2017 Leleannec et al.
20180005356 January 4, 2018 Van Der Vleuten et al.
20180048801 February 15, 2018 Kiser et al.
20180054566 February 22, 2018 Yaguchi
20180063537 March 1, 2018 Sasai et al.
20180152721 May 31, 2018 Rusanovskyy et al.
20180191997 July 5, 2018 Oh
20180198957 July 12, 2018 Kiser et al.
20190014308 January 10, 2019 Kiser et al.
20190166283 May 30, 2019 Kiser et al.
20190238725 August 1, 2019 Kiser et al.
20190238726 August 1, 2019 Kiser et al.
20190238766 August 1, 2019 Kiser et al.
20190349581 November 14, 2019 Fuchie
20190373260 December 5, 2019 Kiser et al.
20200036918 January 30, 2020 Ingle et al.
20200058104 February 20, 2020 Kiser et al.
20200059670 February 20, 2020 Kiser et al.
20200154030 May 14, 2020 Kiser et al.
20200219236 July 9, 2020 Olivier
20200320955 October 8, 2020 Kiser et al.
20200368616 November 26, 2020 Delamont
20210029271 January 28, 2021 Kiser et al.
20210034342 February 4, 2021 Hoy
20210044765 February 11, 2021 Kiser et al.
20210099616 April 1, 2021 Kiser et al.
20210227220 July 22, 2021 Kiser et al.
Foreign Patent Documents
101344706 September 2010 CN
105472265 April 2016 CN
0484802 May 1992 EP
1225574 July 2002 EP
1395062 March 2004 EP
3051821 August 2016 EP
3070934 September 2016 EP
2526047 November 2015 GB
2539917 January 2017 GB
853093026 August 1978 JP
853124028 October 1978 JP
S60213178 October 1985 JP
863160489 July 1988 JP
H0468876 March 1992 JP
0564070 March 1993 JP
106335006 December 1994 JP
H07107346 April 1995 JP
H08220585 August 1996 JP
H11127441 May 1999 JP
2000019407 January 2000 JP
2000338313 December 2000 JP
2001136434 May 2001 JP
2002165108 June 2002 JP
2002-369210 December 2002 JP
2003035881 February 2003 JP
2005-117524 April 2005 JP
2007-96510 April 2007 JP
2007-243942 September 2007 JP
2007-281816 October 2007 JP
2007295326 November 2007 JP
2009-17157 January 2009 JP
2013-27021 February 2013 JP
2014-524290 September 2014 JP
101310140 September 2013 KR
2005025685 March 2005 WO
2009/043494 April 2009 WO
2009/111642 September 2009 WO
2009/121068 October 2009 WO
2011/032028 March 2011 WO
2012/076646 June 2012 WO
2013/025530 February 2013 WO
2015/072754 May 2015 WO
2015/173570 November 2015 WO
2017/139363 August 2017 WO
2017/139596 August 2017 WO
2017/139600 August 2017 WO
2017/157845 September 2017 WO
Other references
  • Machine translation of JP S60213178 A generated on May 30, 2017, by EPO website (6 pages).
  • Machine translation of JPH08220585 generated by European Patent Office on Oct. 15, 2019 (11 pages).
  • Myszkowki, 2008, High Dynamic Range Video, Morgan & Claypool Publishers, San Rafael, CA (158 pages).
  • Nayar, 2000, High dynamic range imaging: spatially varying pixel exposures, 2000 Proc IEEE Conf on Comp Vision and Pattern Rec, ISSN: 1063-6919 (8 pages).
  • Nosratinia, 2002, Enhancement of JPEG-compressed images by re-application of JPEG, Journal of VLSI signal processing systems for signal, image and video technology (20 pages).
  • Oliveira, 2012, Functional programming with structured graphs, ICFP'12 (12 pages).
  • Rahman, 2011, Pipeline synthesis and optimization of FPGA-based video processing applications with CAL, EURASIP J Image Vid Processing 19:1-28.
  • Roberts, 2017, Lossy Data Compression: JPEG, Stanford faculty page (5 pages) Retrieved from the Internet on Feb. 3, 2017, from <https://cs.stanford.edu/people/eroberts/courses/soco/projects/data-compression/lossy/jpeg/dct.htm>( 5 pages).
  • Schulte, 2016, HDR Demystified: Emerging UHDTV systems, SpectraCal 1-22.
  • Sedigh, 1998, Evaluation of filtering mechanisms for MPEG video communications, IEES Symp Rel Dist Sys (6 pages).
  • Sony, 2017, HDR (High Dynamic Range), Sony Corporation (15 pages).
  • Stumpfel, 2004, Direct HDR Capture of the Sun and Sky, Computer graphics, virtual reality, visualisation and interaction in Africa (9 pages).
  • Tiwari, 2015, A review on high-dynamic range imaging with its technique, Int J Sig Proc, IPPR 8(9):93-100.
  • Tocci, 2011, A versatile HDR video production system, ACM Transactions on Graphics (TOG)—Proceedings of ACM SIGGRAPH 2011, 30(4):article 41 (9 pages).
  • Touze, 2014, HDR video coding based on local LDR quantization, Second International Conference and SME Workshop on HDR imaging (6 pages).
  • Unattributed, 2018, JPEG YCbCr Support, Microsoft, Retrieved from the Internet on Nov. 20, 2019 from <https://docs.microsoft.com/en-us/windows/win32/wic/jpeg-ycbcr-support> (14 pages).
  • Wong, 2017, Ultra-low latency contiguous block-parallel stream windowing using FPGA on-chip memory, FPT 56-63.
  • Abstract of JP 2000019407 A (2 pages).
  • Abstract of JP 2003035881 A (2 pages).
  • Abstract of JP S60213178 A (2 pages).
  • Aggarwal, 2004, Split Aperture Imaging for High Dynamic Range, Int J Comp Vis 58(1):7-17.
  • Alleysson, 2006, HDR CFA Image Rendering, Proc EURASIP 14th European Signal Processing Conf. (5 pages).
  • Altera, 2010, Memory System Design, Chapter 7 in Embedded Design Handbook, Altera Corporation (18 pages).
  • Banterle, 2009, High dynamic range imaging and low dynamic range expansion for generating HDR content, Eurographics State of the The Art Report (18 pages).
  • Borer, 2014, Non-linear opto-electrical transfer functions for high dynamic range television, Research and Development White Paper, British Broadcasting Corporation (24 pages).
  • Bravo, 2011, Efficient smart CMOS camera based on FPGAs oriented to embedded image processing, Sensors 11:2282-2303.
  • Cao, 2003, Dynamic configuration management in a graph-oriented distributed programming environment, Sci Comp Prog 48:43-65.
  • Cao, 2005, GOP: A graph-oriented programming model for parallel and distributed systems, Chapter 2 in New Horizons of Parallel and Distributed Computing, Guo & Yang, Eds., Springer (Boston, MA) (17 pages).
  • Chan, 2005, Visual programming support for graph-oriented parallel/ distributed processing, Softw Pract Exper 35:1409-1439.
  • Damazio, 2006, A codec architecture for real-time High Dynamic Range video, VIII Symposium on Virtual and Augmented Reality (Belém, PA, Brazil) (9 pages).
  • Debevec, 1997, Recovering High Dynamic Range Radiance Maps from Photographs, Int Conf Comp Graphics and Interactive Techniques, proceedings (10 pages).
  • Dhanani, 2008, HD video line buffering in FPGA, EE Times (5 pages).
  • Geronimo, 2010, Survey of pedestrian detection for advanced driver assistance systems, IEEE Trans Pat Anal Mach Int 32(7):1239-58.
  • Gurel, 2016, A comparative study between RTL and HLS for image processing applications with FPGAs, Thesis, UC San Diego (78 pages).
  • Hegarty, 2014, Darkroom: compiling high-level image processing code into hardware pipelines, ACM Trans Graph 33(4):144.
  • Int Search Report and Written Opinion dated Apr. 14, 2017, for PCT/US2017/17396 (9 pages).
  • Int Search Report and Written Opinion dated Apr. 28, 2017, for PCT/US2017/17405 (9 pages).
  • Int Search Report and Written Opinion dated Dec. 10, 2019, for PCT/US19/046350 (10 pages).
  • Int Search Report and Written Opinion dated Dec. 6, 2019, for PCT/US19/46348 (9 pages).
  • Int Search Report and Written Opinion dated May 2, 2017, for PCT/US17/16991 (7 pages).
  • Int Search Report and Written Opinion dated May 8, 2017, for PCT/US17/17400 (8 pages).
  • Int Search Report and Written Opinion dated Oct. 23, 2017, for PCT/US17/45683 (5 pages).
  • Int Search Report and Written Opinion dated Oct. 23, 2020, for PCT/US2020/044724 (8 pages).
  • Int Search Report and Written Opinion dated Sep. 11, 2019, for PCT/US19/35109 (10 pages).
  • Int Search Report and Written Opinion dated Sep. 11, 2019, for PCT/US2019/35109 (10 pages).
  • Int Search Report and Written Opinion dated Sep. 20, 2018, for PCT/US2018/041034 (10 pages).
  • Jack, 2005, Color spaces, Chapter 3 in Video Demystified: A Handbook for the Digital Engineer, 4Ed, Newnes (20 pages).
  • Kao, 2008, High Dynamic Range Imaging by Fusing Multiple Raw Images and Tone Reproduction, IEEE Transactions on Consumer Electronics 54(1):10-15.
  • Kresch, 1999, Fast DCT domain filtering using the DCT and the DST, IEEE Trans Imag Proc (29 pages).
  • Lawal, 2007, C++ based system synthesis of real-time video processing systems targeting FPGA implementation, IEEE Int Par Dist Proc Symposium, Rome, pp. 1-7.
  • Lawal, 2008, Memory synthesis for FPGA implementations of real-time video processing systems, Thesis, Mid Sweden U (102 pages).
  • Lukac, 2004, Demosaicked Image Postprocessing Using Local Color Ratios, IEEE Transactions on Circuits and Systems for Video Technology 14(6):914-920.
  • Lyu, 2014, A 12-bit high-speed col. parallel two-step single-slope analog-to-digital converter (ADC) for CMOS image sensors, Sensors 14:21603-21625.
  • Machine translation of CN 101344706 B, generated on May 19, 2017, by espacenet (11 pages).
  • Machine translation of JP 2000019407 A generated on May 30, 2017, by EPO website (52 pages).
  • Machine translation of JP 2000338313 A generated on Dec. 21, 2016, by Espacenet (9 pages).
  • Machine translation of JP 2001136434 A generated on Dec. 21, 2016, by Espacent (25 pages).
  • Machine translation of JP 2002165108 A generated on Dec. 21, 2016, by Espacenet (27 pages).
  • Machine translation of JP 2003035881 A genertaed on May 30, 2017, by EPO website (19 pages).
  • Machine translation of JP 2007295326 A generated on Dec. 21, 2016, by the European Patent Office website Espace.net (12 pages).
  • Machine translation of JP 2007295326 A generated on Dec. 21, 2016, by the European Patent Office website Espacent (12 pages).
  • Machine translation of JP H04068876 A generated on Dec. 21, 2016, by Espacent (8 pages).
  • Machine translation of JP H0564070 A generated on Dec. 21, 2016, by Espacenet (19 pages).
  • Machine translation of JP H06335006 A generated on Dec. 21, 2016, by Espacenet (9 pages).
  • Machine translation of JP H07107346 generated on Dec. 21, 2016, by Espacent (21 pages).
  • Machine translation of JP H08 220585 A obtained Feb. 3, 2020, from Espacenet (14 pages).
  • Machine translation of JP S53093026 A, issued as JP S599888, generated on Dec. 21, 2016 (5 pages).
  • De Haan, 2016, HDR CD6: Core experiments 4.3 adn 4.6a: description of the Philips system in 4:2:0 and with automatic reshaper parameter derivation, JCTVC-W0063 (13 pages).
Patent History
Patent number: 11985316
Type: Grant
Filed: Mar 16, 2021
Date of Patent: May 14, 2024
Patent Publication Number: 20210227220
Assignee: Contrast, Inc. (Albuquerque, NM)
Inventors: Willie C. Kiser (Albuquerque, NM), Michael D. Tocci (Albuquerque, NM), Nora Tocci (Albuquerque, NM)
Primary Examiner: Jonathan R Messmore
Application Number: 17/202,529
Classifications
Current U.S. Class: Color Tv (348/242)
International Classification: H04N 5/235 (20060101); H04N 19/12 (20140101); H04N 19/136 (20140101); H04N 19/184 (20140101);