HARDWARE FRIENDLY FIXED-POINT APPROXIMATIONS OF VIDEO QUALITY METRICS

Abstract

A scalable hardware accelerator configured to compute video quality metrics is disclosed. In some embodiments, an accelerator for video quality metrics comprises an application-specific integrated circuit that includes an interface configured to receive pixel data of a frame of a video being analyzed for quality metric determination and a kernel configured to compute a video quality metric for the received pixel data using a fixed-point hardware approximation of a floating-point based algorithm associated with the video quality metric.

Description

CROSS REFERENCE TO OTHER APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 63/061,692 entitled HARDWARE ACCELERATION OF VIDEO QUALITY METRICS filed Aug. 5, 2020 which is incorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

Video transcoding systems rely on video quality metrics for determining optimal video resolutions to serve to end user devices. Video quality metrics in existing video transcoding systems have mostly been implemented in software and have been limited to less computationally complex algorithms so that system resources are not overburdened. Thus, there exists a need for techniques to energy efficiently compute complex video quality metrics in video transcoding systems that provide better measures of transcoded video quality at low power.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

FIG. 1 is a high level block diagram illustrating an embodiment of an accelerator architecture for accelerating computations of objective video quality metrics.

FIG. 2 is a high level flow chart illustrating an embodiment of a process for computing a video quality metric.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims, and the invention encompasses numerous alternatives, modifications, and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example, and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.

With the advancement of digital media and growing demand for video content, video transcoding has become a common operation in data centers. Generally, video transcoding is the process of generating multiple versions of the same video in different resolutions or sizes. More specifically, a video transcoder typically comprises processing steps including receiving an input video, decoding the input video, and re-encoding the decoded input video into a plurality of qualities or resolutions (e.g., 360p, 480p, 720p, 1080p, 4K, etc.) that are persisted server-side so that optimal versions of the video may be selected and provided to different devices based on corresponding viewport sizes and/or available communication bandwidths. Transcoding an input video into a prescribed resolution may result in some quality loss in the resulting encoded video having the prescribed resolution. Moreover, scaling the encoded video having the prescribed resolution to different viewport sizes may result in further quality loss.

Quality metrics comprise a manner for measuring or quantifying quality losses resulting from transcoding an input video into an encoded video having a prescribed resolution and/or from scaling the encoded video having the prescribed resolution to a prescribed viewport size. Video transcoding applications rely on quality metrics to select an optimal version of a video for an end user device based on current capabilities of the device for receiving and displaying the video. Thus, quality metrics are determined for each of a plurality of encoded video resolutions for each of a plurality of viewport resolutions so that corresponding quality scores may be employed to select and provide an appropriate version of a video to an end user device.

Quality metrics may generally be divided into two categories—subjective quality metrics and objective quality metrics. Subjective quality metrics are determined via human test subjects, e.g., by asking users for their ratings or scores. Objective quality metrics are determined via mathematical models that facilitate computation of corresponding quality scores or values. For example, Peak Signal-to-Noise Ratio (PSNR) comprises a simple computation based on summing squared errors that has widely been used as an objective pixel quality metric. While subjective quality metrics provide better measures of true perceptual quality, determining such metrics is not scalable or even feasible for most applications. As such, several perception-based objective quality metrics have been proposed in recent years that have been correlated to human perception during testing and have evolved to closely represent subjective video quality. Examples of such perception-based objective quality metrics include Structural Similarity Index Measure (SSIM), Multi-Scale SSIM (MS-SSIM), Visual Information Fidelity (VIF), Video Multimethod Assessment Fusion (VMAF), Detail Loss Metric (DLM), etc.

Objective quality metrics are very resource intensive since computations are performed for each pixel for each frame of a video. Moreover, computational complexity increases with increasing resolutions since computations have to be performed for more pixels. Furthermore, live applications require dynamic computations of quality metrics in real time that need to be performed without introducing significant latency. Objective quality metrics have traditionally been implemented in software, and typically only less computationally complex metrics (such as PSNR and single scale SSIM) have been employed to minimize resource consumption in an associated system. However, more computationally complex objective quality metrics offer opportunities for better quality measurements that provide more accurate indications of perceived video quality. A hardware accelerator dedicated to efficiently computing objective quality metrics is disclosed herein that provides support for not only simpler objective quality metrics that have traditionally been implemented in software but also for more computationally complex emerging objective quality metrics that have been proposed in literature and limited to use cases that do not have resource and/or time constraints but that have yet to receive adoption in video transcoding systems due to heretofore introducing unacceptable resource overheads.

FIG. 1 is a high level block diagram illustrating an embodiment of an accelerator 100 for accelerating computations of objective video quality metrics. Accelerator 100 comprises an application-specific integrated circuit for computing one or more video quality metrics and supports simultaneously computing multiple video quality metrics in parallel. Various components comprising the architecture of accelerator 100 are scalable for parallel processing based on area and power budgets available with respect to an associated system. Accelerator 100 may comprise a stand-alone component or a component of a system or device. For example, accelerator 100 may comprise an independent component of a video transcoding system that offloads resource intensive video quality metrics computations from a central processing unit of the transcoding system.

A simplified block diagram of components comprising an embodiment of accelerator 100 is illustrated in FIG. 1 for the purpose of explanation. However, generally, accelerator 100 may comprise any other appropriate combination and configuration of components to achieve the described functionality. Although many of the examples described herein are with respect to computing quality metrics for frames of a transcoded video, the disclosed techniques may be employed to compute quality metrics for any type of image data comprising pixels.

Video quality measurements may be categorized into full reference metrics, partial reference metrics, and no reference metrics. For a full reference metric, a complete reference image is available to compute distorted image quality. For a partial reference metric, partial information of a reference image such as a set of associated parameters is available to compute distorted image quality. For a no reference metric, no reference image is available, and the metric is used to establish source or upload quality. An accelerator for computing video quality metrics may generally be configured to support any combination of one or more full, partial, and/or no reference metrics. In the example of FIG. 1, accelerator 100 is specifically configured to support a plurality of full reference metrics as well as a no reference metric. A full reference metric comprises comparing a reference frame and a distorted frame on a per pixel or pixel block basis to predict perceptual quality of the distorted frame with respect to the reference frame. A reference frame comprises an original source frame prior to transcoding while a distorted frame comprises an encoded version of the reference frame after transcoding. A quality score is computed per pixel for a frame. Quality scores of pixels comprising a frame are accumulated or combined to generate a frame level score and/or a block level score for a portion of the frame. Block level scores may be useful in identifying regions of a frame that have higher impact on quality. Computed frame level and/or block level scores may be written to memory so that they are available to other components of an associated transcoding system. In some cases, frame level scores of frames comprising a prescribed video are later combined in an appropriate manner to generate a quality score for the video.

In FIG. 1, controller 102 of accelerator 100 facilitates obtaining input video frame data 101. More specifically, controller 102 of accelerator 100 facilitates obtaining frame data 101 from memory or from one or more intermediary components thereof. For example, in one embodiment, controller 102 communicates, e.g., via a double data rate (DDR) channel, with a direct memory access (DMA) interface that interfaces with physical memory. Controller 102 facilitates reading both reference frame data and distorted frame data from memory. Controller 102 furthermore coordinates or synchronizes reads of reference and distorted frame pairs to ensure that read portions of both frames are spatially aligned when input into accelerator 100 so that reference and distorted frame pairs can later be processed on a pixel by pixel basis when computing video quality metrics. Generally, optimally reading data from memory is desirable since memory bandwidth in an associated system is both a limited and expensive resource in terms of power. By supporting computations of multiple metrics simultaneously, accelerator 100 avoids the need to read the same frame multiple times from memory for different metrics, thus more optimally utilizing both bandwidth and power in an associated system.

Read input frame data 101 is loaded into one or more local input buffers 104. In some embodiments, input buffer 104 is configured in a ping pong buffer configuration in which one buffer partition is populated with data read from memory while data comprising another buffer partition is read for processing so that memory read latency can be hidden. Frame data 101 is read from memory by controller 102 and written into input buffer 104 in units of a prescribed input block size. The block size may be based on the size of input buffer 104 and/or a bandwidth supported by a corresponding on-chip network. In some embodiments, pixel blocks comprising a frame are read from memory in a raster scan order, i.e., from left to right and from top to bottom of the frame. Moreover, pixel data comprising a frame may furthermore be decoupled into luminance (luma) and interleaved chrominance (chroma) components. Accelerator 100 may generally be configured to operate on either or both the luminance and chrominance planes, which may be segregated and processed by accelerator 100 in a prescribed order and/or which may be processed in multiple passes by accelerator 100 with each plane read and processed separately.

Reference frame data 106 and distorted frame data 108 stored in buffer memory 104 are read by and input into processing unit 110. That is, a portion of reference frame 106 and a corresponding portion of distorted frame 108 that each comprise a prescribed processing block size are input into processing unit 110 for processing. Processing unit 110 comprises the core processing kernel of accelerator 100. Processing unit 110 is configured to compute a plurality of video quality metrics or scores based on input frame pixel data. More specifically, processing unit 110 is configured to compute a plurality of different perception-based video quality metrics for distorted frame 108 with respect to reference frame 106. Furthermore, processing unit 110 may be configured to compute one or more other types of video quality metrics such as a PSNR metric for distorted frame 108 with respect to reference frame 106 and/or a no reference metric for reference frame 106 that indicates source or upload quality prior to transcoding. In some embodiments, processing unit 110 is configured to simultaneously compute a plurality of video quality metrics in parallel. For example, in one embodiment, processing unit 110 is configured to simultaneously compute up to three video quality metrics including a no reference quality metric, a PSNR metric, and one of a plurality of supported perception-based video quality metrics. In such cases, a selected one of a plurality of supported perception-based video quality metrics that processing unit 110 is currently configured to compute may be specified via a programming interface associated with accelerator 100. Generally, accelerator 100 may be dynamically programmed to compute any one or more supported video quality metrics and may be programmed differently for different input frames.

Video quality metrics are typically determined for a plurality of different viewport resolutions for each encoded resolution. Thus, in many cases, frame data is first scaled to a desired viewport resolution, and then video quality metrics are computed on the scaled output. Processing unit 110 comprises a plurality of programmable inline scaling units for scaling reference and distorted frame data to desired resolutions prior to computing one or more video quality metrics. More specifically, processing unit 110 comprise scaling unit 112 for scaling reference frame data 106 and scaling unit 114 for scaling distorted frame data 108. Each scaling unit may be dynamically programmed to a prescribed scaling mode (e.g., upscale, downscale, bypass) and scaling ratio or factor via an associated programming interface. Scaled outputs are not stored in memory but rather directly input into one or more processing kernels for on the fly inline computations of corresponding video quality metrics. By providing inline scaling, the architecture of accelerator 100 facilitates more efficient memory bandwidth usage in an associated system by eliminating the need to write and read scaled outputs to and from memory. Scaling units 112 and 114 may comprise any appropriate programmable scaler configurations that, for example, do not introduce any further or at least any significant quality loss during the scaling process.

Scaled frame data is processed by one or more compute kernels that are each configured to compute one or more video quality metrics. In the embodiment of FIG. 1, processing unit 110 comprises three separate hardware partitions, each with one or more kernels. The various kernels comprising processing unit 110 implement fixed point versions of the quality metrics algorithms they are configured to implement. That is, floating point operations specified with respect to the original algorithms are appropriately modified to fixed point equivalents for efficient hardware implementation. The various components comprising processing unit 110 are scalable. That is, processing unit may be extended to include any number of threads of scaling unit pairs and compute kernels so that a plurality of viewport resolutions may be computed in parallel, which effectively facilitates reducing the number of passes needed to compute scores for all viewport resolutions and, in turn, the number of times input frames need to be read from memory. Various details of the specific embodiment of processing unit 110 illustrated in FIG. 1 are next described to provide an example of a manner in which processing unit 110 may be configured. However, generally, processing unit 110 may comprise any other appropriate combination and configuration of components to achieve the described functionalities.

In the embodiment of FIG. 1, partition 116 of processing unit 110 is configured to compute a no reference metric that is used to establish source quality of an input reference frame. In the given example, kernel 118 comprising partition 116 is configured to compute a blurriness metric. Kernel 118 may comprise a plurality of stages. For example, in one embodiment, input reference frame data 106 is smoothened using a Gaussian blur filter in a first stage, the smoothened output from the first stage is input into a Sobel filter to compute pixel gradients in a second stage, and the output of the second stage is input into a third stage that determines edge width values associated with the spread of the edge of each pixel, which are then used to compute final blur scores.

Partition 120 of processing unit 110 comprises kernel 122 and kernel 124. Kernel 122 is configured to compute a PSNR (sum of squared errors) metric with respect to input reference and distorted frame data. Kernel 124 is configured to compute SSIM, for example, using an FFMPEG based algorithm, which comprises an overlapped 8×8 approximation algorithm. In SSIM, three components—luminance (L), contrast (C), and structure (S)—based on local means, standard deviations, and cross-covariance of reference and distorted frame data are computed and combined to obtain an overall similarity measure, i.e., SSIM index.

Partition 126 of processing unit 110 comprises kernel 128. Kernel 128 comprises a unified kernel configured to compute single-scale (LIBVMAF) SSIM, multi-scale SSIM, as well as VIF and may be programmed to compute any one of the aforementioned metrics for a given input frame. In partition 126, distorted and reference frame data is first filtered via filter 130 and filter 132, respectively, which in some cases comprise smoothening Gaussian blur filters. The smoothened frame data output by filters 130 and 132 is then input into kernel 128 which is configured to compute LCS values of SSIM. For single scale SSIM, e.g., that is computed using an LIBVMAF based algorithm, input pixels are sent once through kernel 128. For MS-SSIM, the smoothened outputs of both frames are sent through corresponding dyadic down samplers 134 and 136 and looped back to kernel 128 to process higher scales. This process may be iterated up to a prescribed number of times corresponding to a maximum number of scales or levels supported. The feedback paths of partition 126 facilitate reuse of the same hardware to compute all scales or levels. Kernel 128 is furthermore configured to compute VIF and supports logarithmic operations needed to compute VIF scores.

Software algorithms for computing video quality metrics typically comprise floating-point operations. However, such algorithms need to be converted to fixed-point representations when implemented in hardware, such as in the described accelerator architecture. Various challenges exist in converting a floating point representation to a fixed point representation since at least some error is introduced due to the reduced precision of the fixed-point representation. Moreover, many video quality metrics comprise complex mathematical operations that need to be simplified for viable hardware implementation.

FIG. 2 is a high level flow chart illustrating an embodiment of a process 200 for computing a video quality metric. For example, process 200 of FIG. 2 may be employed by accelerator 100 of FIG. 1 and/or one or more components thereof. At step 202, pixel data of a frame of a video being analyzed for quality metric determination is received, e.g., at an interface of a quality metrics application-specific integrated circuit or component thereof. At step 204, a video quality metric is computed for the received pixel data using a fixed-point hardware approximation of a floating-point based algorithm associated with the video quality metric, e.g., by a compute kernel of the application-specific integrated circuit.

Fixed-point approximations may give rise to numerical instabilities, such as dividing by zero or values that should be positive becoming negative due to limited precision. In some embodiments, one or more programmable numerical stability constants are introduced into a fixed-point approximation of a video quality metric algorithm to prevent numerical instability. In some embodiments, thresholds are employed for values of one or more parameters of a video quality metric algorithm to preserve numerical stability. As one example, a parameter of a video quality metric algorithm may be identified to have the mathematical property of always being positive or non-negative, and a negative value of the parameter may be clipped to zero or replaced by a (small) positive value greater than zero.

Many video quality metrics comprise complex mathematical operations such as non-linear functions. To reduce complexity, an approximation of a non-linear function may be employed to realize a fixed-point hardware implementation. As an example, consider a logarithmic function, which is employed in the computation of VIF. For a more hardware friendly implementation, a non-linear function, such as a logarithmic function, may be approximated, for example, using a k-th order polynomial or a piecewise linear function comprising a plurality of segments. In a piecewise linear function, slope and intercept values of segments comprising the piecewise linear function may be determined, for example, using a least squares method or using a Taylor expansion around a center of each segment. Segments comprising a piecewise linear function may be uniformly sampled or may be sampled in a non-uniform manner. In a non-uniform sampling of a logarithmic function, for example, more segments may be used in a region closer to an input argument having a value of one than a region closer to an input argument having a value of two. Hardware implementation may furthermore be simplified by constraining (e.g., via scaling) an input argument of a logarithmic function approximation to a value comprising a prescribed range (e.g., greater than or equal to one but less than two). In such cases, the logarithm of any input argument value may be represented by a scaled logarithm of a value comprising the constrained input argument range. Furthermore, in some embodiments, a sum of lifting values (i.e., scaling values to maintain intermediate precision) of the input argument, coefficients, and output is bounded by a prescribed value comprising a systemic constraint on a maximum number of bits supported in a corresponding system.

Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A system, comprising:

an interface of an application-specific integrated circuit configured to receive pixel data of a frame of a video being analyzed for quality metric determination; and

a kernel of the application-specific integrated circuit configured to compute a video quality metric for the received pixel data using a fixed-point hardware approximation of a floating-point based algorithm associated with the video quality metric.

2. The system of claim 1, wherein the fixed-point hardware approximation comprises an approximation of a non-linear function.

3. The system of claim 2, wherein the approximation of the non-linear function comprises a piecewise linear function.

4. The system of claim 3, wherein the piecewise linear function is uniformly or non-uniformly sampled.

5. The system of claim 2, wherein the approximation of the non-linear function comprises a k-th order polynomial.

6. The system of claim 1, wherein the fixed-point hardware approximation comprises an approximation of a logarithmic function.

7. The system of claim 6, wherein an input argument of the approximation of the logarithmic function is scaled to a value comprising a prescribed range.

8. The system of claim 6, wherein the approximation of the logarithmic function comprises a piecewise linear function with more segments in a region closer to an input argument having a value of one than a region closer to a value of two.

9. The system of claim 1, wherein the fixed-point hardware approximation preserves numerical stability of the algorithm by using corresponding thresholds for one or more associated parameters of the algorithm.

10. The system of claim 1, wherein the fixed-point hardware approximation replaces a negative value of a parameter of the algorithm with a non-negative value, wherein the parameter has a mathematical property of being non-negative.

11. The system of claim 10, wherein the non-negative value is zero.

12. The system of claim 1, wherein the video quality metric comprises a no reference metric or a full reference metric.

13. The system of claim 1, wherein the video quality metric comprises a perception-based video quality metric.

14. The system of claim 13, wherein the perception-based video quality metric comprises one or more of: Structural Similarity Index Measure (SSIM), Multi-Scale SSIM (MS-SSIM), Visual Information Fidelity (VIF), Video Multimethod Assessment Fusion (VMAF), and Detail Loss Metric (DLM).

15. A method, comprising:

configuring an interface of an application-specific integrated circuit to receive pixel data of a frame of a video being analyzed for quality metric determination; and

configuring a kernel of the application-specific integrated circuit to compute a video quality metric for the received pixel data using a fixed-point hardware approximation of a floating-point based algorithm associated with the video quality metric.

16. The method of claim 15, wherein the fixed-point hardware approximation comprises an approximation of a non-linear function.

17. The method of claim 15, wherein the fixed-point hardware approximation preserves numerical stability of the algorithm by using corresponding thresholds for one or more associated parameters of the algorithm.

18. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

configuring an interface of an application-specific integrated circuit to receive pixel data of a frame of a video being analyzed for quality metric determination; and

configuring a kernel of the application-specific integrated circuit to compute a video quality metric for the received pixel data using a fixed-point hardware approximation of a floating-point based algorithm associated with the video quality metric.

19. The computer program product of claim 18, wherein the fixed-point hardware approximation comprises an approximation of a non-linear function.

20. The computer program product of claim 18, wherein the fixed-point hardware approximation preserves numerical stability of the algorithm by using corresponding thresholds for one or more associated parameters of the algorithm.