Method and apparatus for contorlling coefficient transmission by early determining whether to decode information that inverse transform needs
A video decoding method includes: before residual decoding of a coding unit is completed, referring to available information to determine whether to decode information that an inverse transform (IT) circuit needs for applying inverse transform to transform blocks of the coding unit, and generating a determination result; and controlling coefficient transmission of the coding unit to the IT circuit according to the determination result.
Latest MEDIATEK INC. Patents:
- METHOD FOR PERFORMING MEDIUM ACCESS CONTROL PROTOCOL DATA UNIT DISPATCH CONTROL IN MULTI-LINK OPERATION ARCHITECTURE, AND ASSOCIATED APPARATUS
- DIGITAL ENVELOPE TRACKING SUPPLY MODULATOR
- MODEL SPECIALIZATION FOR MACHINE LEARNING BASED COMPILER OPTIMIZATION
- CONTROL METHOD OF ELECTRONIC DEVICE FOR UNIDIRECTIONAL DATA TRANSMISSION
- Systems for controlling a slew rate of a switch
This application claims the benefit of U.S. Provisional Application No. 63/257,627, filed on Oct. 20, 2021. The content of the application is incorporated herein by reference.
BACKGROUNDThe present invention relates to video processing, and more particularly, to a method and apparatus for controlling coefficient transmission by early determining whether to decode information that inverse transform needs.
The conventional video coding standards generally adopt a block based coding technique to exploit spatial and temporal redundancy. For example, the basic approach is to divide the whole source picture into a plurality of blocks, perform intra/inter prediction on each block, transform residuals of each block, and perform quantization and entropy encoding. Besides, a reconstructed picture is generated in a coding loop to provide reference pixel data used for coding following blocks. For certain video coding standards, in-loop filter(s) may be used for enhancing the image quality of the reconstructed frame. The video decoder is used to perform an inverse operation of a video encoding operation performed by a video encoder. For example, the video decoder may include an entropy decoding circuit, an inverse scan circuit, an inverse transform circuit, etc. The operation of the inverse transform circuit may be configured by values of certain syntax elements. In accordance with a syntax decoding flow specified by a versatile video coding (VVC) standard (also known as H.266 standard), the information that the inverse transform circuit needs for applying inverse transform to transform blocks of a coding unit may be decoded by the entropy decoding circuit after residual decoding of the coding unit is fully completed by the entropy decoding circuit. As a result, the inverse transform circuit does not start to apply inverse transform to transform blocks of the coding unit until information that the inverse transform circuit needs is determined. Furthermore, the inverse scan circuit needs a large buffer to keep the residual decoding result of the coding unit before information that the inverse transform circuit needs is determined.
Thus, there is a need for an innovative video decoder design which is capable of enhancing the performance of the inverse transform circuit and/or reducing the buffer requirement of the inverse scan circuit.
SUMMARYOne of the objectives of the claimed invention is to provide a method and apparatus for controlling coefficient transmission by early determining whether to decode information (e.g. syntax elements lfnst_idx and mts_idx) that inverse transform needs.
According to a first aspect of the present invention, an exemplary video decoding method is disclosed. The exemplary video decoding method includes: before residual decoding of a coding unit is completed, referring to available information to determine whether to decode information that an inverse transform (IT) circuit needs for applying inverse transform to transform blocks of the coding unit, and generating a determination result; and controlling coefficient transmission of the coding unit to the IT circuit according to the determination result.
According to a second aspect of the present invention, an exemplary video decoding apparatus is disclosed. The exemplary video decoding apparatus includes an entropy decoding circuit, an inverse scan (IS) circuit, and an inverse transform (IT) circuit. The entropy decoding circuit is arranged to perform residual decoding of a coding unit. The IT circuit is arranged to apply inverse transform to transform blocks of the coding unit. The IS circuit is coupled between the entropy decoding circuit and the IT circuit, and is arranged to re-order coefficients of the coding unit. Before the residual decoding of the coding unit is completed, at least one of the entropy decoding circuit and the IS circuit is arranged to determine whether the entropy decoding circuit needs to decode information that the IT circuit needs for applying the inverse transform to the transform blocks of the coding unit, to generate a determination result, and is further arranged to control coefficient transmission of the coding unit to the IT circuit according to the determination result.
These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.
Certain terms are used throughout the following description and claims, which refer to particular components. As one skilled in the art will appreciate, electronic equipment manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not in function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ”. Also, the term “couple” is intended to mean either an indirect or direct electrical connection. Accordingly, if one device is coupled to another device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.
It should be noted that the video decoder architecture shown in
One picture may be divided into a plurality of coding units (CUs), each CU may be divided into a plurality of transform units (TUs), and each TU may consist of multiple transform blocks (TBs). For example, one TU may consist of one luma TB and two chroma TBs. In accordance with an original syntax decoding flow specified by the VVC standard, information that the IT circuit 108 needs for applying inverse transform to transform blocks of a coding unit is decoded by the VLD circuit 102 after residual decoding of the coding unit is fully completed by the VLD circuit 102.
The code segment 202 includes partial code segments 212, 214 and 216. The code segment 204 includes partial code segments 218, 220, 222, and 224. High level information (which can be obtained before residual decoding of the current coding unit is started) is checked by the partial code segments 212, 218 and 220. The other four variables LfnstDcOnly, LfnstZeroOutSigCoeffFlag, MtsZeroOutSigCoeffFlag, MtsDcOnly (which are not available before residual decoding of the current coding unit is started, and can be determined during a process of the residual decoding of the current coding unit) are checked by the partial code segments 214, 216, 222, and 224. Since all of the high level information and the other four variables can be available before residual decoding of the current coding unit is fully completed by the VLD circuit 102, whether to decode the information (e.g. syntax elements lfnst_idx and mts_idx) needed by the IT circuit 108 may be early determined, which allows a start time of coefficient transmission to the IT circuit 108 to be earlier than an end time of residual decoding of the current coding unit.
Based on above observations, the present invention proposes a VLD circuit equipped with an early determination function and an IS circuit equipped with an early determination function. As shown in
Assume that all TBs in one CU are 32×32 TBs as illustrated in
Assume that all the luma TBs in this CU are 32×32 luma TBs as illustrated in
When MtsZeroOutSigCoeffFlag==0 and LfnstZeroOutSigCoeffFlag==0 are early determined by the early determination circuit 128 according to coefficients stored in the IS buffer 126 (step 304), the flow proceeds with step 306.
At step 306, the VLD circuit 102 directly sets the information (lfnst_idx & mts_idx) that the IT circuit 108 needs by default values with no decoding/parsing of the information (lfnst_idx & mts_idx). For example, the syntax element lfnst_idx is inferred to be a default value (e.g. 0), and the syntax element mts_idx is inferred to be a default value (e.g. 0). At step 308, the IS circuit 104 starts coefficient transmission in response to early determination of syntax values of lfnst_idx and mts_idx. Since syntax values of lfnst_idx and mts_idx can be early determined before residual decoding of the current coding unit is fully completed by the VLD circuit 102, the coefficient transmission from the IS circuit 104 (particularly, IS buffer 126 of IS circuit 104) to the IT circuit 108 via the IQ circuit 106 is allowed to be early started before residual decoding of the current coding unit is fully completed by the VLD circuit 102. In this way, the performance of the IT circuit 108 can be improved greatly.
According to the proposed syntax decoding flow with an early determination scheme, the early determination circuit 124 may refer to the high level information to judge that the VLD circuit 102 may still need to decode the IT-related syntax elements lfnst_idx and mts_idx from the encoded bitstream BS. In addition, the VLD circuit 102 performs residual decoding for a luma TB during a period between time points T0 and T1, and stores the residual decoding result (i.e. transform coefficients) of the luma TB into the IS buffer 126. At the time point T1, the early determination circuit 128 may find that the luma TB has residuals (non-zero coefficients) occupying outside the left-top 16×16 region, meaning that MTS and LFNST conditions are both violated. Hence, at the time point T1, it is early determined that there is no need to decode the IT-related syntax elements lfnst_idx and mts_idx from the encoded bitstream BS, and the IT-related syntax elements lfnst_idx and mts_idx are inferred to be default values. Since the IT-related syntax elements lfnst_idx and mts_idx are early determined at the time point T1, the IS buffer 126 outputs the residual decoding result (i.e. transform coefficients) of the luma TB to the IT circuit 108 through the IQ circuit 106 during a period between time points T1 and T2. In addition, the VLD circuit 102 performs residual decoding for one chroma TB (e.g. Cb TB) during the same period between time points T1 and T2, and stores the residual decoding result (i.e. transform coefficients) of the chroma TB (e.g. Cb TB) into the IS buffer 126. During a period between time points T2 and T3, the IS buffer 126 outputs the residual decoding result (i.e. transform coefficients) of the chroma TB (e.g. Cb TB) to the IT circuit 108 through the IQ circuit 106, and the VLD circuit 102 performs residual decoding for the other chroma TB (e.g. Cr TB) and stores the residual decoding result (i.e. transform coefficients) of the chroma TB (e.g. Cr TB) into the IS buffer 126. During a period between time points T3 and T4, the IS buffer 126 outputs the residual decoding result (i.e. transform coefficients) of the chroma TB (e.g. Cr TB) to the IT circuit 108 through the IQ circuit 106. Compared to the original syntax decoding flow as specified by the VVC standard, the proposed syntax decoding flow enables coefficient transmission to the IT circuit earlier, thereby improving the video decoding performance greatly.
According to the proposed syntax decoding flow with an early determination scheme, the early determination circuit 124 may refer to the high level information to judge that the VLD circuit 102 may still need to decode the IT-related syntax element lfnst_idx from the encoded bitstream BS. In addition, the VLD circuit 102 performs residual decoding for one chroma TB (e.g. Cb TB) during a period between time points T0 and T1, and stores the residual decoding result (i.e. transform coefficients) of the chroma TB (e.g. Cb TB) into the IS buffer 126. At the time point T1, the early determination circuit 128 may find that the chroma TB has residuals (non-zero coefficients) occupying outside the left-top 4×4 region, meaning that LFNST condition is violated. Hence, at the time point T1, it is early determined that there is no need to decode the IT-related syntax element lfnst_idx from the encoded bitstream BS, and the IT-related syntax element lfnst_idx is inferred to be a default value. Since the IT-related syntax element lfnst_idx is early determined at the time point T1, the IS buffer 126 outputs the residual decoding result (i.e. transform coefficients) of the chroma TB (e.g. Cb TB) to the IT circuit 108 through the IQ circuit 106 during a period between time points T1 and T2. In addition, the VLD circuit 102 performs residual decoding for the other chroma TB (e.g. Cr TB) during the same period between time points T1 and T2, and stores the residual decoding result (i.e. transform coefficients) of the chroma TB (e.g. Cr TB) into the IS buffer 126. During a period between time points T2 and T3, the IS buffer 126 outputs the residual decoding result (i.e. transform coefficients) of the chroma TB (e.g. Cr TB) to the IT circuit 108 through the IQ circuit 106. Compared to the original syntax decoding flow as specified by the VVC standard, the proposed syntax decoding flow enables coefficient transmission to the IT circuit earlier, thereby improving the video decoding performance greatly.
Regarding the code segment 202 shown in
Since the above-mentioned coding block size constraints imply a smaller IS buffer size for residual decoding of one CU, the present invention can take the advantage of the coding block size constraints for achieving an objective of reducing the buffer requirement of the IS circuit 104. If the transform block size is not larger than 32, the VLD circuit 102 may have the chance to decode the IT-related syntax elements lfnst_idx and mts_idx from the encoded bitstream BS. In a case where the proposed syntax decoding flow with an early determination scheme is employed, the IS buffer may be required to have a buffer size that is large enough to accommodate 16×16+2×(16×16) residuals (coefficients) under a worst case where a CU has a CU size of 32×32 in a shared tree case, the luma residuals (coefficients) only occupy a left-top 16×16 region, and chroma residuals (coefficients) occupy the whole 16×16 region. With regard to a normal case, the IS buffer may be required to have a buffer size that is large enough to accommodate 32×32 residuals (coefficients). Hence, the buffer size of the IS buffer 126 may be set by 32×32, that is, max(16×16+2×(16×16), 32×32). To put it simply, the IS buffer 126 can be designed to have a buffer size being smaller than a maximum data amount of coefficients possessed by one CU with the largest CU size (e.g. 128×128). In this way, the buffer requirement of the IS circuit 104 can be relaxed.
Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.
Claims
1. A video decoding method comprising:
- before residual decoding of a coding unit is completed, referring to available information to determine whether to decode information that an inverse transform (IT) circuit needs for applying inverse transform to transform blocks of the coding unit, and generating a determination result; and
- controlling coefficient transmission of the coding unit to the IT circuit according to the determination result.
2. The video decoding method of claim 1, wherein referring to the available information to determine whether to decode the information that the IT circuit needs comprises:
- before the residual decoding of the coding unit is started, obtaining the available information.
3. The video decoding method of claim 1, wherein referring to the available information to determine whether to decode the information that the IT circuit needs comprises:
- during a process of the residual decoding of the coding unit, obtaining the available information.
4. The video decoding method of claim 1, wherein controlling the coefficient transmission of the coding unit to the IT circuit according to the determination result comprises:
- in response to the determination result indicating that there is no need to decode the information that the IT circuit needs, starting the coefficient transmission of the coding unit before the residual decoding of the coding unit is completed.
5. The video decoding method of claim 4, further comprising:
- in response to the determination result indicating that there is no need to decode the information that the IT circuit needs, setting the information that the IT circuit needs by default values.
6. The video decoding method of claim 4, wherein the information that the IT circuit needs comprises a syntax element lfnst_idx and a syntax element mts_idx; and before an end time of the residual decoding of the coding unit, the syntax element lfnst_idx and the syntax element mts_idx are determined in response to the determination result indicating that there is no need to decode the information that the IT circuit needs.
7. The video decoding method of claim 1, further comprising:
- during a process of the residual decoding of the coding unit, storing each coefficient of the coding unit into a buffer, wherein a buffer size of the buffer is smaller than a maximum data amount of coefficients possessed by one CU with a largest CU size.
8. The video decoding method of claim 1, wherein the information that the IT circuit needs comprises a syntax element lfnst_idx or a syntax element mts_idx.
9. A video decoding apparatus comprising:
- an entropy decoding circuit, arranged to perform residual decoding of a coding unit;
- an inverse transform (IT) circuit, arranged to apply inverse transform to transform blocks of the coding unit; and
- an inverse scan (IS) circuit, coupled between the entropy decoding circuit and the IT circuit, and arranged to re-order coefficients of the coding unit;
- wherein before the residual decoding of the coding unit is completed, at least one of the entropy decoding circuit and the IS circuit is arranged to determine whether the entropy decoding circuit needs to decode information that the IT circuit needs for applying the inverse transform to the transform blocks of the coding unit, to generate a determination result, and is further arranged to control coefficient transmission of the coding unit to the IT circuit according to the determination result.
10. The video decoding apparatus of claim 9, wherein the entropy decoding circuit obtains the available information before the residual decoding of the coding unit is started, and determines whether to decode the information that the IT circuit needs according to the available information.
11. The video decoding apparatus of claim 9, wherein the IS circuit obtains the available information during a process of the residual decoding of the coding unit performed by the entropy decoding circuit, and determines whether the entropy decoding circuit needs to decode the information that the IT circuit needs according to the available information.
12. The video decoding apparatus of claim 9, wherein when the determination result indicates that there is no need to decode the information that the IT circuit needs, the IS circuit starts the coefficient transmission of the coding unit before the residual decoding of the coding unit is completed by the entropy decoding circuit.
13. The video decoding apparatus of claim 12, wherein when the determination result indicates that there is no need to decode the information that the IT circuit needs, the information that the IT circuit needs is set by default values.
14. The video decoding apparatus of claim 12, wherein the information that the IT circuit needs comprises a syntax element lfnst_idx and a syntax element mts_idx; and before an end time of the residual decoding of the coding unit, the syntax element lfnst_idx and the syntax element mts_idx are determined in response to the determination result indicating that there is no need to decode the information that the IT circuit needs.
15. The video decoding apparatus of claim 9, wherein the IS circuit comprises a buffer for storing each coefficient of the coding unit that is generated from the residual decoding of the coding unit; and a buffer size of the buffer is smaller than a maximum data amount of coefficients possessed by one CU with a largest CU size.
16. The video decoding apparatus of claim 9, wherein the information that the IT circuit needs comprises a syntax element lfnst_idx or a syntax element mts_idx.
20170163988 | June 8, 2017 | Lu |
202041022 | November 2020 | TW |
202046719 | December 2020 | TW |
Type: Grant
Filed: Jul 24, 2022
Date of Patent: Apr 2, 2024
Patent Publication Number: 20230122258
Assignee: MEDIATEK INC. (Hsin-Chu)
Inventors: Ming-Hsien Lai (Hsinchu), Min-Hao Chiu (Hsinchu), Chia-Yun Cheng (Hsinchu)
Primary Examiner: Nam D Pham
Application Number: 17/871,976