MOVING PICTURE DECODING DEVICE, MOVING PICTURE DECODING METHOD AND INTEGRATED CIRCUIT
An moving picture decoding device includes a receiver that receives data that has been coded according to multi-view coding of pictures that are included in a moving picture and provided for a plurality of views; a first decoder that decodes coded data that is included in the data coded according to the multi-view coding and provided for a first view; and a second decoder that decodes coded data that is included in the data coded according to the multi-view coding and provided for a second view A detector detects a region of the data that cannot be decoded and is among a plurality of regions to be decoded by the first decoder. A concealing unit conceals the predetermined region detected by the detector using decoded data.
Latest FUJITSU LIMITED Patents:
- Communication device and communication system for selecting resources to be used for data transmission
- OPTICAL TRANSMITTER THAT TRANSMITS MULTI-LEVEL SIGNAL
- COMPUTER-READABLE RECORDING MEDIUM STORING DETERMINATION PROGRAM, DETERMINATION METHOD, AND INFORMATION PROCESSING APPARATUS
- METHOD AND APPARATUS FOR INFORMATION PROCESSING
- STORAGE MEDIUM, INFORMATION PROCESSING APPARATUS, AND MERCHANDISE PURCHASE SUPPORT METHOD
This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2010-212711, filed on Sep. 22, 2010, the entire contents of which are incorporated herein by reference.
FIELDEmbodiments relate to a moving picture decoding device, a moving picture decoding method and an integrated circuit, which allow coded data of a moving picture to be decoded.
BACKGROUNDIn recent years, an environment for viewing and recording a three-dimensional video using a consumer device is being developed. Especially, Blu-ray Disc Association (BDA) has determined a standard for a three-dimensional image format. A market of devices (such as TVs and mobile phones) that support three-dimensional video has started to grow. It is expected that digital broadcasting of three-dimensional (3D) video will be widely available in the near future.
In the broadcasting of three-dimensional video or transmission of three-dimensional video, a broadcast wave may not be appropriately received due to low signal strength in a transmission network. When a transmission error (such as an error indicating that a broadcast wave cannot be appropriately received) occurs, video data or audio data cannot be accurately decoded. In order to encode an image, a temporal-directional prediction is performed. Thus, when a picture in which an error occurs is a picture (such as an I picture or a P picture) that is referenced for prediction of another picture, information of the error is referenced, and the error is transferred to multiple frames. When the error is transferred to the multiple frames, a range that is affected by the error is increased.
To avoid the transfer of the error, there is a technique for concealing the error (occurring in the error picture or an error region of the picture) using an accurately decoded picture or a predetermined region of the accurately decoded picture.
For example, Japanese Laid-open Patent Publication No. 2006-203598 discloses a technique for performing a panning detection and for, when an error occurs, setting a motion vector for a motion compensation and for a block (error block) in which the error occurs on the basis of a motion vector of a macro-block (MB) located near the error block.
In addition, Japanese Laid-open Patent Publication No. 2007-325206 discloses a technique for determining the validity of a macro-block located near a macro-block to be restored, acquiring context information for image restoration from the macro-block determined to be valid, and calculating an estimated motion vector to be used for the restoration.
SUMMARYIn accordance with an aspect of the embodiments, an moving picture decoding device includes, an receiving unit that receives data that has been coded according to multi-view coding and includes data obtained by coding, on a predetermined region basis, pictures that are included in a moving picture and provided for a plurality of views; a first decoder that decodes, on a predetermined region basis, coded data that is included in the data coded according to the multi-view coding and provided for a first view; a second decoder that decodes, on a predetermined region basis, coded data that is included in the data coded according to the multi-view coding and provided for a second view; a detector that detects a predetermined region that cannot be decoded and is among a plurality of predetermined regions to be decoded by the first decoder; and a concealing unit that conceals the predetermined region detected by the detector using decoded data.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed.
These and/or other aspects and advantages will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawing of which:
When multi-view video coding (MVC) such as a stereo profile of ITU-T H.264 (ITU-T: International Telecommunication Union Telecommunication Standardization Sector)/ISO/IEC MPEG-4 AVC (ISO/IEC: International Organization for Standardization/International Electrotechnical Commission) is used, images that are provided for multiple views and for which other images are referenced exist, and the other images that are provided for the multiple views and referenced have a relationship as illustrated in
The present inventor newly found that there was the following problem. It is considered that each of the aforementioned conventional techniques is used to perform an error concealment process for each of views in order to decode a multi-view image. When the error concealment process is performed using only information on the same view, and an error occurs in several pictures, the error cannot be appropriately concealed.
In addition, as illustrated in
Next, embodiments are described with reference to the accompanying drawings.
First, amounts of information generated for each of views in multi-view video coding are considered. In the following description, multiple views are two views, a right view and a left view, as an example. However, the multiple views may be three or more views. Regarding amounts of information generated in multi-view video coding, larger amounts of information are applied to pictures (such as I pictures and P pictures) that may be referenced, compared to amounts of information to be applied to pictures (such as B pictures) that are not referenced, in general. It is considered that when the amount of information is large, the possibility that an error occurs is high. For example, a B picture may be referenced in H.264, depending on the coding method.
As illustrated in
As illustrated in
As illustrated in
A picture that is included in coded moving picture data to be decoded may be a field to be obtained by an interlace scheme or may be a frame to be obtained by a progressive scheme.
The receiving unit 101 receives data (hereinafter referred to as multi-view coded data) coded using a multi-view video coding technique. The multi-view coded data includes coded data of a moving picture. The coded data of the moving picture is provided for multiple views. The coded data of the moving picture is data obtained by coding pictures on a predetermined region basis.
When the receiving unit 101 receives the multi-view coded data, the receiving unit 101 divides the multi-view coded data into coded data for the multiple views, and outputs the coded data for the interested views to the variable length decoders 102 and 122. The receiving unit 101 may reference headers of the pictures of the multi-view coded data and thereby determine the views for which the coded data is provided. The headers of the pictures each include data that indicates the base side or the dependent side.
A scheme that is used to form the stream of the multi-view coded data is not limited to the frame sequential scheme. The stream of the multi-view coded data may be formed by a side-by-side scheme in which a picture for the right view and a picture for the left view are arranged side by side in a horizontal direction. In addition, the stream of the multi-view coded data may be formed by a top-and-bottom scheme in which a picture for the right view and a picture for the left view are arranged in a vertical direction.
Returning to
The variable length decoder 102 outputs identification numbers of the macro-blocks decoded according to the variable length decoding, pictures decoded according to the variable length decoding, information of the macro-blocks decoded according to the variable length decoding and the like to the error detector 103. The variable length decoder 102 outputs the decoded data such as motion vectors of the macro-blocks, the coefficient information of the macro-blocks, the quantization scales of the macro-blocks to the buffer memory 104.
In addition, the variable length decoder 102 performs the variable length decoding on data, which is included in a predetermined number (2 or more) of pictures and has been coded according to the variable length coding, in the order of the data to be decoded before processes to be performed by the image decoder 106, the block extractor 108 and the concealing unit 107.
Thus, the moving picture decoding device 100 may use a motion vector of a macro-block that is included in a picture to be decoded after decoding of a picture in which an error is to be concealed. The predetermined number of pictures that can be reference can be two or more and in a range up to a value obtained by adding 1 to the number of pictures for which a picture in which an error is to be concealed.
The error detector 103 detects a region that is included in a picture included in the coded data and in which an error occurs during the variable length decoding and/or image decoding. The error detector 103 detects whether or not an error occurs in each of macro-blocks or in each of predetermined blocks, while the macro-blocks or the predetermined blocks are unit regions for which motion compensation is performed. For example, the error detector 103 detects whether or not data that is necessary to decode a macro-block or a predetermined block is lost.
In the following description, a macro-block in which an error occurs is called an error macro-block (error MB). In order to simplify the description, predetermined regions that are included in each of pictures and subjected to an error detection are regarded as macro-blocks. Even when the predetermined regions that are included in each of the pictures and subjected to the error detection are regarded as predetermined blocks, the error detector 103 may detect an error by performing the same process as the process to be performed on the macro-blocks.
The error detector 103 may specify a macro-block to be detected whether or not an error occurs in the macro-block on the basis of the value of a macro-block address of the macro-block. Macro-block addresses are provided for the macro-blocks, respectively. The error detector 103 checks if all predetermined information that is defined by a standard for the coding scheme and provided for the specified macro-block exists. When all the information provided for the specified macro-block exists, the error detector 103 determines that there is no error in the specified macro-block. When at least a part of all the information provided for the specified macro-block does not exist, the error detector 103 determines that the macro-block is an error MB.
In addition, the error detector 103 may determine whether or not identification numbers such as macro-block addresses provided for macro-blocks are continuous. In this case, when the identification numbers are not continuous, the error detector 103 determines that a macro-block of which the identification number does not exist is an error MB.
The error detector 103 controls the selector 105 on the basis of whether or not an error exists. When the error detector 103 detects an error MB, the error detector 103 controls the selector 105 so that data that expresses the error MB and is stored in the buffer memory 104 may be read into the block extractor 108 and that the data that expresses the error MB and is stored in the buffer memory 124 may be read into the block extractor 108 when necessary. On the other hand, when the error detector 103 does not detect an error MB, the error detector 103 controls the selector 105 so that data that is stored in the buffer memory 104 may be read into the image decoder 106. The error detector 103 also controls the selector 109. This control is described later.
The buffer memory 104 temporarily stores data that has been decoded by the variable length decoder 102 and expresses the predetermined number of pictures. The data that is temporarily stored in the buffer memory 104 is motion vectors, coefficient information, quantization scales, MB types and the like.
The selector 105 is controlled by the error detector 103. When the error detector 103 notifies the selector 105 that an error MB exists, the selector 105 connects the buffer memory 104 to the buffer memory 124 and the block extractor 108. When the error detector 103 notifies the selector 105 that an error MB does not exist, the selector 105 connects the buffer memory 104 to the image decoder 106.
The dependent side is described below. The receiving unit 101 references the headers of the pictures of the multi-view coded data and outputs the coded data for the dependent view to the variable length decoder 122.
The variable length decoder 122 performs the same process as the variable length decoder 102. For example, the variable length decoder 122 decodes variable length coded data included in pictures of the received data and outputs the decoded data to the buffer memory 124. In addition, the variable length decoder 122 outputs, to the error detector 123, the decoded data that is used to for error detection.
The error detector 123 performs the same process as the error detector 103. For example, the error detector 123 uses the aforementioned predetermined method to detect whether or not an error occurs in each of the macro-blocks included in the pictures. In addition, the error detector 123 controls the selector 125 on the basis of whether or not an error exists. The error detector 123 also controls the selector 129. This control is described later.
The buffer memory 124 temporarily stores the data decoded by the variable length decoder 122. The data that is temporarily stored in the buffer memory 124 is motion vectors, coefficient information, quantization scales, MB types and the like.
When the error detector 123 notifies the selector 125 that an error MB exists, the selector 125 connects the buffer memory 124 to the block extractor 108. When the error detector 123 notifies the selector 125 that an error MB does not exist, the selector 125 connects the buffer memory 124 to the image decoder 126.
The preceding decoding that is performed by the variable length decoders 102 and 122 is described with reference to
A normal displaying order is indicated by a block string 202. The pictures are displayed in order of increasing number or in the order of the pictures B0, B1, I2, B3, B4, P5, B6, B7 and P8.
As indicated by the block strings 211 and 212, the variable length decoder 102 performs the variable length decoding earlier by a time corresponding to three pictures than the process to be performed by the image decoder 106 or the like, for example. Thus, the block extractor 108 and the concealing unit 107 may use a motion vector of a picture that is decoded after decoding of a picture in which an error is to be concealed.
For example, when the picture P5 has a region (error region) in which an error exists, the block extractor 108 and the concealing unit 107 may use information that is the results of the variable length decoding performed on the pictures B3 and B4 from which arrows that are indicated by dotted lines of
The pictures B3 and B4 are pictures that have been coded according to inter-frame coding using the picture P5 as a reference picture. Thus, the pictures B3 and B4 are the pictures coded by referencing the location of the picture P5. In the image decoding process, the pictures B3 and B4 are decoded after decoding of the picture P5. For example, even when information of a certain region of the picture P5 is lost due to an error, the concealing unit 107 may perform the error concealment process using motion vectors that are from regions included in the pictures B3 and B4 and for which the certain region of the picture P5 is referenced.
A block string 221 indicates timings of the variable length decoding to be performed on pictures for the dependent side. A block string 222 indicates timings of the image decoding to be performed on the pictures for the dependent side. A block string 223 indicates timings of display of the pictures for the dependent side.
As indicated by the block strings 221 and 222, the variable length decoder 122 performs the variable length decoding earlier by a time corresponding to three pictures than the process to be performed by the image decoder 126 or the like, for example. Thus, the block extractor 108 and the concealing units 107 and 127 may use a motion vector of a picture that is decoded after decoding of a picture in which an error is to be concealed. It is assumed that a picture P5′ is coded using the picture P5 as a reference picture. Thus, in the image decoding process, the picture P5′ is decoded after decoding of the picture P5.
For example, when the picture 5 has a region in which an error exists, the block extractor 108 and the concealing unit 107 may use information that is the results of the variable length decoding performed on the picture P5′ from which an arrow that is indicated by an alternate long and short dashed line of
The picture P5′ is a picture coded according to inter-frame coding using the picture P5 as a reference picture. Thus, the picture P5′ is the picture coded by referencing a certain region of the picture P5. For example, even when information of the certain region of the picture P5 is lost due to an error, the concealing unit 107 may perform the error concealment process using a motion vector of a region that is included in the pictures P5′ coded by referencing the certain region of the picture P5.
In addition, the block extractor 108 and the concealing unit 127 may uses a motion vector of a picture that is decoded after decoding of a picture in which an error is to be concealed.
For example, when the picture P5′ has a region in which an error exists, the block extractor 108 and the concealing unit 127 may use information that is the results of the variable length decoding performed on the pictures B3′ and B4′ from which arrows that are indicated by broken lines of
The pictures B3′ and B4′ are pictures coded according to inter-frame coding using the picture P5′ as a reference picture. Thus, the pictures B3′ and B4′ are the pictures coded by referencing the position of the picture P5′. The pictures B3′ and B4′ are decoded after decoding of the picture P5′. For example, when information of a certain region of the picture P5′ is lost due to an error, the concealing unit 127 may perform the error concealment process using motion vectors that are from regions included in the pictures B3′ and B4′ and for which the certain region of the picture P5′ is referenced.
Returning to
The image decoder 106 outputs the decoded macro-blocks of the pictures, the macro-block addresses of the decoded macro-blocks, and the identification numbers of the pictures including the decoded macro-blocks to the selector 109.
The image decoder 126 for the dependent side performs the same process as the image decoder 106 for the base side so as to decode the macro-block of the picture. The difference between the image decoder 106 and the image decoder 126 is that the image decoder 126 may use the pictures decoded by the image decoder 106 and provided for the base side and the pictures decoded by the image decoder 126 and provided for the dependent side as reference pictures. On the other hand, the image decoder 106 may use only the pictures decoded by the image decoder 106 as reference pictures.
Next, the block extractor 108 is described. The block extractor 108 receives a motion vector from at least one of the buffer memories 104 and 124, uses the motion vector to estimate a similar macro-block to an error MB, and extracts the estimated macro-block. The block extractor 108 is described in detail with reference to
When an error MB occurs in a picture for the base side, the parallax information extractor 141 extracts a motion vector for which the error MB that is included in the picture for the base side is referenced. Since this motion vector is a motion vector between the picture for the base view and a picture for the dependent side, the motion vector indicates parallax information. The parallax information extractor 141 outputs the extracted parallax information to the motion vector calculator 142. When the motion vector for which the error MB is referenced does not exist in the buffer memory 124, the parallax information extractor 141 notifies the motion vector calculator 142 that the motion vector for which the error MB is referenced does not exist in the buffer memory 124.
The motion vector calculator 142 specifies a macro-block (hereinafter also referred to as a parallax MB) that has the received parallax information. Then, the motion vector calculator 142 extracts, from the buffer memory 124, a motion vector (hereinafter also referred to as a reference source motion vector) for which the parallax MB is referenced.
The motion vector calculator 142 determines whether or not a macro-block (hereinafter also referred to as a reference source MB) that has the reference source motion vector has another motion vector. When a macro-block (hereinafter also referred to as an MB to be referenced) that is included in a decoded picture is referenced for the aforementioned other motion vector, the motion vector calculator 142 uses the two motion vectors of the reference source MB to calculate a motion vector for error concealment. The motion vector for the error concealment is hereinafter also referred to as a concealment motion vector. The motion vector calculator 142 outputs the calculated concealment motion vector to the reference block extractor 143.
The concealment motion vector includes, as vector elements, the amounts of shifting in horizontal and vertical directions between the error MB and the MB to be referenced in the pictures, and the number of pictures arranged between a picture provided for the dependent side and corresponding to the picture including the error MB and the picture including the MB to be referenced.
When the motion vector calculator 142 receives, from the parallax information extractor 141, the notification indicating that the motion vector does not exist, the motion vector calculator 142 extracts, from the buffer memory 104, the motion vector for which the error MB is referenced. The motion vector calculator 142 determines whether or not a block (hereinafter also referred to as a same-view reference source MB) that has the motion vector for which the error MB is referenced has another motion vector. When a decoded picture is referenced for the other motion vector, the motion vector calculator 142 uses the two motion vectors of the single-view reference source MB to calculate a concealment motion vector. The motion vector calculator 142 outputs the calculated concealment motion vector to the reference block extractor 143.
The reference block extractor 143 extracts the macro-block address of a macro-block included in a decoded picture and specified by the received concealment motion vector, and extracts the identification number of the picture that includes the macro-block. The reference block extractor 143 outputs the extracted macro-block address and the extracted identification number to the concealing unit 107. The macro-block that is specified by the reference block extractor 143 is a macro-block estimated as a similar macro-block to the error MB.
Next, details of a process that is performed by the block extractor 108 when an error MB occurs in a picture for the dependent side are described. When an error MB occurs in a picture for the dependent side, the motion vector calculator 142 extracts, from the buffer memory 124, a motion vector for which the error MB is referenced. The motion vector calculator 142 selects, from the buffer memory 124, a block (hereinafter also referred to as a same-view reference source MB) that is a reference source of the extracted motion vector.
The motion vector calculator 142 determines whether or not the selected same-view reference source MB includes another motion vector. When a decoded picture is referenced for the other motion vector, the motion vector calculator 142 uses the two motion vectors of the same-view reference source MB to calculate a concealment motion vector. The motion vector calculator 142 outputs the calculated concealment motion vector to the reference block extractor 143.
The reference block extractor 143 extracts the macro-block address of a macro-block (MB to be referenced) included in a decoded picture and specified by the received concealment motion vector, and extracts the identification number of the picture that includes the macro-block. The reference block extractor 143 outputs the extracted macro-block address and the extracted identification number to the concealing unit 127.
The motion vector calculator 142 has an evaluating unit 150. When a plurality of reference source MBs exist, the evaluating unit 150 determines one of the reference source MBs. The evaluating unit 150 uses information included in the reference source MBs to calculate evaluation values. The evaluating unit 150 selects one of the reference source MBs on the basis of the calculated evaluation values. The evaluating unit 150 is described later in detail. When a plurality of parallax MBs and a plurality of same-view reference source MBs exist, the evaluating unit 150 calculates evaluation values and selects one of the macro-blocks on the basis of the calculated evaluation values.
Next, relationships among an error MB, a parallax MB, a reference source MB and an MB to be referenced are described, and a calculation of a concealment motion vector is described in detail.
For example, when information of an MB 302 included in a picture P5 301 that is provided for the base side and is to be decoded third among pictures for the base side is lost due to an error, a macro-block (parallax MB) 305 that is shifted by an amount corresponding to parallax information 311 from the error MB 302 and for which the error MB 302 is referenced exists in the picture P5′ that is provided for the dependent side and is to be decoded third among pictures for the dependent side.
The shifted amount that corresponds to the parallax information 311 is also indicated by MVD. Since the picture P5′ is to be decoded immediately after decoding of the picture P5 for the base side, a decoded image of the picture P5′ does not exist when the picture P5 is decoded.
Since the pictures B for the dependent side are decoded according to the variable length decoding in advance, information of vectors 312 and 313 of a reference source MB 306 for which the parallax information MB 305 is referenced exists in the buffer memory 124. The motion vector calculator 142 uses the information of the vectors 312 and 313 of the reference source MB 306 to generate a temporary concealment motion vector (MVC′) 314.
In the example illustrated in
MVC′=MVF′−MVB′ Equation (1)
The motion vector calculator 142 uses the temporary concealment motion vector 314 and the parallax information (MVD) 311 to calculate a concealment motion vector (MVC) 315 according to Equation (2).
MVC=MVC′−MVD Equation (2)
A picture to be referenced for the temporary concealment motion vector 314 is the picture I2′. The picture I2′ is already decoded by the image decoder 126 when the picture P5 is decoded. An image that is predicted using an MB 307 (to be referenced) of the picture I2′ and the concealment motion vector 315 is regarded as a block for error concealment.
The reference block extractor 143 may use the MB 307 (to be referenced) of the picture I2′ and the temporary concealment motion vector (MVC′) 314 to decode the parallax information MB 305 of the picture P5′ and may use the parallax information 311 to perform the error concealment process on the error MB 302.
Pictures I3′, rB1′, B0′, B2′, P7′, rB5′, B4′ and B6′ for the dependent side are decoded in this order. The pictures that are indicated with rB are reference B pictures.
It is assumed that when an error MB 402 exists in a picture P7 401, the variable length decoding process is completely performed on the pictures P7′, rB5′, B4′ and B6′ before the image decoding is performed on the picture P7 401.
When the error MB 402 exists in the picture P7 401 for the base side, the picture P7′ for the dependent side has a macro-block (parallax MB) 403 that is shifted by an amount corresponding to parallax information 411 from the error MB 402 and for which the error MB is referenced. Since the picture P7′ is a picture to be decoded immediately after decoding of the picture P7 for the base side, a decoded image of the picture P7′ does not exist when the picture P7 is decoded.
Since the pictures B for the dependent side are decoded according to the variable length decoding in advance, information of vectors 412 and 413 of a reference source MB 404 (included in the picture B6′) for which the parallax MB 403 of the picture P7′ is referenced exists in the buffer memory 124. The motion vector calculator 142 uses the information of the vectors 412 and 413 of the reference source MB 404 to generate a temporary concealment motion vector (MVC′) 414.
In the example illustrated in
Therefore, as illustrated in
A plurality of reference source MBs for which a parallax MB is referenced exist in some cases. A plurality of pictures (such as the pictures B3′ and B4′) for which the single picture P5′ may be referenced exist. In addition, a motion prediction may be performed using a plurality of motion vectors for each of blocks. Thus, there is a possibility that a plurality of reference source MBs for which a parallax MB is referenced exist.
The evaluating unit 150 of the motion vector calculator 142 illustrated in
When a parallax MB is referenced for a plurality of macro-blocks, the evaluating unit 150 uses information of the macro-blocks to determine a macro-block that is most suitable for the motion prediction and is among the macro-blocks. The information of each of the macro-blocks includes the type of a prediction, differential information (coefficient information), and a component of a motion vector.
The determination using the type of the prediction is described below. For example, a block for which the parallax MB of the picture P5′ (illustrated in
Next, the determination using the differential information is described. In the coding process, to obtain the differential information, the difference between frames is calculated using a motion vector. Then, the difference (predicted difference) is subjected to discrete cosine transform (DCT) and a quantization process. Then, the difference is coded according to the variable length coding. As the differential information after the variable length decoding, a quantization coefficient (Coef) and a quantization value (Qscale) are obtained for each of the macro-blocks.
In order to more accurately use the differential information, the difference is subjected to an inverse quantization process and an inverse DCT process so that the differential information is calculated on a pixel basis and the sum of absolute values of components of the differential information or the like is calculated. In order to simplify the process, the following evaluation is performed.
It is assumed that a plurality of candidate blocks MB0, MB1, . . . , MBn exist, quantization values (Qscale values) of the candidate blocks are Q0, Q1, . . . , Qn, and the values of quantization coefficients of the candidate blocks are Coef0_i,j, Coef1_i,j, . . . , Coefn_i,j. The evaluating unit 150 selects a macro-block that causes the difference to be smallest.
As the method for selecting a macro-block, there are at least three examples described below.
FIRST EXAMPLEThe evaluating unit 150 determines that a macro-block of which the quantization scale is smallest is the macro-block that causes the difference to be smallest.
The evaluating unit 150 determines a macro-block that causes the sum of absolute differences among the quantization coefficients to be smallest is the macro-block that causes the difference to be smallest.
The evaluating unit 150 simply determines that a macro-block that causes the number of valid coefficients among the quantization coefficients to be smallest is the macro-block that causes the difference to be smallest.
The evaluating unit 150 may use any of the aforementioned three examples to select one of the macro-blocks.
The evaluating unit 150 determines that a macro-block that has vector components for which the largest number of areas are referenced as reference areas to which motion vectors point is the macro-block that causes the difference to be smallest.
When the macro-blocks each have a size of 16×16, an overlapping region of the macro-block of the original picture and the macro-block of a reference picture is larger as a value of a component of a motion vector is closer to any of multiples of 16. From this standpoint, the motion vector is calculated.
It is assumed that values of components of motion vectors are each close to any of multiples of 16, and the remainders of divisions of the values of the components of the motion vectors by 16 are 0, 1, 2, 3, . . . , 14, and 15. In this assumption, the evaluating unit 150 provides a priority to each of the values 1 to 15 so that the priority is increased in the order of 0, 1, 15, 2, 14, 3, 13, . . . , 7, 9 and 8. The evaluating unit 150 selects a macro-block on the basis of the priorities according to the following calculation formula. The following calculation formula means that in order to select a motion vector that has a component of a value that is largely different from the value of 8, a motion vector that has a component of a large absolute value among values of −8 to 7 obtained by subtracting 8 from the values of 0 to 15 is selected.
The accuracy (such as quarter-pixel accuracy (a value of 2 indicates a single motion vector) or half-pixel accuracy (a value of 4 indicates a single motion vector)) of representing the motion vector varies depending on the coding method. Thus, the evaluating unit 150 changes the calculation formula depending on the accuracy.
For example, when the motion vector is represented by the quarter-pixel accuracy, the following calculation formula is used.
The evaluating unit 150 changes the calculation formula depending on the accuracy as described above. The evaluating unit 150 may not change the calculation formula depending on the accuracy of representing the motion vector. The evaluating unit 150 may change the calculation formula depending on the sizes of the macro-blocks when the macro-blocks do not have a size of 16×16.
Thus, even when a plurality of reference source MBs exist, an appropriate block may be selected by the evaluating unit 150.
The case in which a plurality of parallax MBs for which an error MB is referenced exist may be considered. In this case, the motion vector calculator 142 may select one of the parallax MBs by making the determination using the differential information.
In addition, as illustrated in
In this case, when a plurality of blocks for which the error MB is referenced exist, the evaluating unit 150 determines whether a block for the same view or a block for the other view is prioritized. In the present embodiment, the block for the other view is prioritized and used. However, the block for the same view may be prioritized.
Next, when a plurality of blocks that are provided for the determined view and for which the error MB is referenced exist, one of the blocks is selected by the evaluating unit 150 in the aforementioned manner. When a parallax MB is selected, the evaluating unit 150 selects one of the blocks on the basis of the determination using differential information according to the second example or the determination using motion vectors according to the third example.
Returning to
Thus, since the concealing unit 107 uses parallax information decoded in advance, the concealing unit 107 may conceal an error using not only a decoded picture for the same view but also a decoded picture for the other view.
The selector 109 is controlled by the error detector 103. When an error MB exists, the selector 109 selects a macro-block output from the concealing unit 107. When an error MB does not exist, the selector 109 selects a macro-block output from the image decoder 106. The selector 109 sequentially arranges macro-blocks decoded by the image decoder 106 (or arranges macro-blocks decoded by the image decoder 106 and a macro-block generated by the concealing unit 107) on the basis of the identification number of a picture including the macro-blocks and the macro-block addresses of the macro-blocks and outputs the arranged macro-blocks to the picture memory 110.
The picture memory 110 temporarily stores the macro-blocks output from the selector 109 on a picture basis. Pictures that are stored in the picture memory 110 are output to a display unit or the like in the order of the pictures to be displayed. In addition, there is a possibility that the pictures that are stored in the picture memory 110 are each referenced for a picture for the base side or a picture for the dependent side. Thus, the pictures that are stored in the picture memory 110 are output to the image decoder 106 and the picture memory 130 when necessary. Furthermore, the pictures that are stored in the picture memory 110 are also used for error concealment. Thus, the pictures that are stored in the picture memory 110 are read by the concealing unit 107 when necessary.
The concealing unit 127 for the dependent side basically performs the same process as the concealing unit 107 for the base side. The concealing units 107 and 127 are different from each other in that the concealing unit 127 performs the error concealment process using a decoded picture for the same view.
The selector 129 performs the same process as the selector 109 for the base side. For example, the selector 129 sequentially arranges macro-blocks decoded by the image decoder 126 (or arranges macro-blocks decoded by the image decoder 126 and a macro-block generated by the concealing unit 127) on the basis of the identification number of a picture including the macro-blocks and the macro-block addresses of the macro-blocks and outputs the arranged macro-blocks to the picture memory 130.
The picture memory 130 temporarily stores the macro-blocks output from the selector 129 on a picture basis. The picture memory 130 temporarily stores the pictures that are provided for the base side and have been output from the picture memory 110. This is due to the fact that the pictures for the base side may be referenced for pictures for the dependent side.
In addition, the pictures that are provided for the dependent side and stored in the picture memory 130 are output to the display unit or the like in the order of the pictures to be displayed. Furthermore, the pictures that are provided for the dependent side and stored in the picture memory 130 are read by the concealing unit 127. Furthermore, the pictures that are provided for the base side and the dependent side and stored in the picture memory 130 are output to the image decoder 126 as reference pictures when necessary.
With the aforementioned configuration, each of the concealing units 107 and 127 may use a motion vector decoded according to the variable length decoding before the other processes and perform the error concealment process on a macro-block estimated to be similar to an error MB. In addition, since the concealing unit 107 may use a motion vector (parallax information) for which a decoded picture for the other view is referenced, the concealing unit 107 may perform the error concealment process on a macro-block estimated to be similar to an error MB and included in the decoded picture for the other view.
Next, operations of the moving picture decoding device according to the first embodiment are described.
In operation S101, the variable length decoder 102 performs the variable length decoding on coded data that expresses a single picture for the base side and has been divided by the receiving unit 101, before the image decoding to be performed by the image decoder 106. The variable length decoder 102 causes the decoded data (such as motion vectors and coefficient information) obtained by the variable length decoding to be stored in the buffer memory 104. In addition, the variable length decoder 102 outputs, to the error detector 103, data (such as macro-block addresses) that is necessary for error detection.
In operation S102, the moving picture decoding device 100 determines whether or not decoded data that is accumulated in the buffer memory 104 has an amount corresponding to the predetermined number of pictures. When the decoded data that is accumulated in the buffer memory 104 has the amount corresponding to the predetermined number of pictures (Yes in operation S102), the process proceeds to operation S103. When the decoded data that is accumulated in the buffer memory 104 does not have the amount corresponding to the predetermined number of pictures (No in operation S102), the process returns to operation S101. Thus, the variable length decoder 102 may performs the variable length decoding on the predetermined number of pictures before the image decoding to be performed by the image decoder 106.
In operation S103, the variable length decoder 102 performs the variable length decoding on the predetermined number of the next pictures. The variable length decoder 102 causes decoded data (such as motion vectors and coefficient information) obtained by the variable length decoding to be stored in the buffer memory 104. In addition, the variable length decoder 102 outputs, to the error detector 103, data (such as macro-block addresses) that is necessary for error detection.
In operation S104, the error detector 103 detects, on a macro-block basis on the basis of the data received from the variable length decoder 102, an error region from pictures in order from a picture to be first decoded to a picture to be lastly decoded. The error detector 103 may detect an error when the values of the macro-block addresses are not continuous, for example.
In operation S105, the error detector 103 determines whether or not an error exists in a macro-block to be detected. When an error exists in the macro-block to be detected (Yes in operation S105), the process proceeds to operation S106. When an error does not exist in the macro-block to be detected (No in operation S105), the process proceeds to operation S113.
In operation S106, the block extractor 108 determines whether or not a parallax MB for which the error MB is referenced exists in the buffer memory 124 for the dependent side. The parallax MB is a macro-block that is provided for the dependent side and has a motion vector (parallax information) for which the error MB is referenced. When the parallax MB exits in the buffer memory 124 (Yes in operation S106), the process proceeds to operation S107. When the parallax MB does not exist in the buffer memory 124 (No in operation S106), the process proceeds to operation S109.
In operation S107, the block extractor 108 determines whether or not a reference source MB for which the parallax MB is referenced exists in the buffer memory 124 for the dependent side. The reference source MB is a macro-block that is provided for the dependent side and has a motion vector for which the parallax MB is referenced. When the reference source MB exists in the buffer memory 124 (Yes in operation S107), the process proceeds to operation S108. When the reference source MB does not exist in the buffer memory 124 (No in operation S107), the process proceeds to operation S109.
In operation S108, when a plurality of reference source MBs exist in the buffer memory 124, the block extractor 108 selects an appropriate macro-block (MB) that is estimated to be similar to the error MB. Operation S108 is described later with reference to
In operation S109, the block extractor 108 determines whether or not a same-view reference source MB for which the error MB is referenced exists in the buffer memory 104. The same-view reference source MB is included in a picture for the base side and has a motion vector for which the error MB is referenced. When the same-view reference source MB exists in the buffer memory 104 (Yes in operation S109), the process proceeds to operation S110. When the same-view reference source MB does not exist in the buffer memory 104 (No in operation S109), the process proceeds to operation S112.
In operation S110, when a plurality of same-view reference source MBs exist in the buffer memory 104, the block extractor 108 selects an appropriate macro-block that is estimated to be similar to the error MB. This selection operation is performed in the same manner as operation S108 and described later with reference to
In operation S111, the concealing unit 107 uses, for an error concealment process (error concealment process A), the macro-block included in a decoded picture and estimated to be similar to the error MB using the motion vector decoded according to the variable length decoding before the image decoding. When the process is performed through operation S108, the concealing unit 107 uses, for the error concealment process, a macro-block estimated to be similar to the error MB and included in a decoded picture for the dependent side that is the other view. When the process is performed through operation S110, the concealing unit 107 uses, for the error concealment process, a macro-block estimated to be similar to the error MB and included in a decoded picture for the base side that is the same view.
In operation S112, when a motion vector for which the error MB is referenced is not among motion vectors decoded according to the variable length decoding before the image decoding, the concealing unit 107 performs a known error concealment process (error concealment process B). In the known error concealment process, a macro-block that is referenced for a motion vector of a macro-block located near the error MB is used to conceal the error, or a macro-block that is included in a picture decoded immediately before decoding of a picture including the error MB and is located at the same position as the error MB is used to conceal the error, for example. In operation S112, any error concealment process may be performed as long as the concealing process is a known error concealment process in which the motion vectors obtained by the variable length decoding performed before the image decoding are not used.
In operation S113, the image decoder 106 decodes the picture on a macro-block basis as described above.
In operation S114, the moving picture decoding device 100 determines whether or not macro-blocks for a single picture are subjected to the image decoding process. When any of the macro-blocks for the single picture is not subjected to the image decoding process (No in operation S114), the process returns to operation S104 so that the error detection is performed on the next macro-block. When all the macro-blocks for the single picture are subjected to the image decoding process (Yes in operation S114), the process proceeds to operation S115.
The selector 109 arranges the macro-blocks decoded by the image decoder 106 (or arranges the macro-blocks decoded by the image decoder 106 and the macro-block subjected to the error concealment process by the concealing unit 107) on the basis of the macro-block addresses and the identification number of the picture. Then, the selector 109 causes the macro-blocks to be stored in the picture memory 110 on a picture basis.
In operation S115, the moving picture decoding device 100 determines whether or not all pictures that are included in the coded data are decoded. When all the pictures are decoded (Yes in operation S115), the moving picture decoding device 100 terminates the decoding process. When any of all the pictures is not decoded (No in operation S115), the process returns to operation S103 so that the decoding process is performed on the single picture.
Next, the decoding process that is performed for the dependent side is described.
In operation S201, the variable length decoder 122 performs the variable length decoding on coded data that expresses a single picture for the dependent side and has been divided by the receiving unit 101, before the image decoding to be performed by the image decoder 126. The variable length decoder 122 causes decoded data (such as motion vectors and coefficient information) obtained by the variable length decoding to be stored in the buffer memory 124. In addition, the variable length decoder 122 outputs, to the error detector 123, data (such as macro-block addresses) that is necessary for error detection.
In operation S202, the moving picture decoding device 100 determines whether or not decoded data that is accumulated in the buffer memory 124 has an amount corresponding to the predetermined number of pictures. When the decoded data that is accumulated in the buffer memory 104 has the amount corresponding to the predetermined number of pictures (Yes in operation S202), the process proceeds to operation S203. When the decoded data that is accumulated in the buffer memory 104 does not have the amount corresponding to the predetermined number of pictures (No in operation S202), the process returns to operation S201. Thus, the variable length decoder 122 may perform the variable length decoding on the predetermined number of pictures before the image decoding to be performed by the image decoder 126.
In operation S203, the variable length decoder 122 performs the variable length decoding on the predetermined number of the next pictures. The variable length decoder 122 causes decoded data (such as motion vectors and coefficient information) obtained by the variable length decoding to be stored in the buffer memory 124. In addition, the variable length decoder 122 outputs, to the error detector 123, data (such as macro-block addresses) that is necessary for error detection.
In operation S204, the error detector 123 detects, on a macro-block basis on the basis of the data received from the variable length decoder 122, an error region from pictures in order from a picture to be first decoded to a picture to be lastly decoded. The error decoder 123 may detect an error when the values of the macro-block addresses are not continuous, for example.
In operation S205, the error detector 123 determines whether or not an error exists in a macro-block to be detected. When an error exists in the macro-block to be detected (Yes in operation S205), the process proceeds to operation S206. When an error does not exist in the macro-block to be detected (No in operation S206), the process proceeds to operation S210.
In operation S206, the block extractor 108 determines whether or not a same-view reference source MB for which the error MB is referenced exists in the buffer memory 124 for the dependent side. The same-view reference source MB is a macro-block included in a picture for the dependent side and has a motion vector for which the error MB is referenced. When the same-view reference source MB exists in the buffer memory 124 (Yes in operation S206), the process proceeds to operation S207. When the same-view reference source MB does not exist in the buffer memory 124 (No in operation S206), the process proceeds to operation S209.
In operation S207, when a plurality of same-view reference source MBs exist in the buffer memory 124, the block extractor 108 selects an appropriate MB that is estimated to be similar to the error MB. This selection operation is described with reference to
In operation S208, the concealing unit 127 uses, for an error concealment process (error concealment process A), the macro-block included in the decoded picture and estimated to be similar to the error MB using the motion vector decoded according to the variable length decoding before the image decoding. The concealing unit 127 uses, for the error concealment process, the macro-block included in the decoded picture for the dependent side that is the same view.
In operation S209, when a motion vector for which the error MB is referenced is not among motion vectors decoded according to the variable length decoding before the image decoding, the concealing unit 127 performs a known error concealment process (error concealment process B). In the known error concealment process, a macro-block that is referenced for a motion vector of a macro-block located near the error MB is used to conceal the error, or a macro-block that is included in a picture decoded immediately before decoding of a picture including the error MB and is located at the same position as the error MB is used to conceal the error, for example. In operation S209, any error concealment process may be performed as long as the concealing process is a known error concealment process in which the motion vectors obtained by the variable length decoding performed before the image decoding are not used.
In operation S210, the image decoder 126 decodes the picture on a macro-block basis as described above.
In operation S211, the moving picture decoding device 100 determines whether or not macro-blocks for a single picture are subjected to the image decoding process. When any of the macro-blocks for the single picture is not subjected to the image decoding process (No in operation S211), the process returns to operation S204 so that the error detection is performed on the next macro-block. When all the macro-blocks for the single picture are subjected to the image decoding process (Yes in operation S211), the process proceeds to operation S212.
The selector 129 arranges the macro-blocks decoded by the image decoder 126 or arranges (the macro-blocks decoded by the image decoder 126 and the macro-block subjected to the error concealment process by the concealing unit 127 on the basis of the macro-block addresses and the identification number of the picture. Then, the selector 129 causes the macro-blocks to be stored in the picture memory 130 on a picture basis.
In operation S212, the moving picture decoding device 100 determines whether or not all pictures that are included in the coded data are decoded. When all the pictures are decoded (Yes in operation S212), the moving picture decoding device 100 terminates the decoding process. When any of all the pictures is not decoded (No in operation S212), the process returns to operation S203 so that the decoding process is performed on the single picture.
Next, operations S108 and S110 illustrated in
In operation S301, the block extractor 108 determines whether or not the type of a reference source MB is a bi-direction prediction type. When the type of the reference source MB is the bi-direction prediction type (Yes in operation S301), the process proceeds to operation S302. When the type of the reference source MB is not the bi-direction prediction type (No in operation S301), the process proceeds to operation S305.
In operation S302, the block extractor 108 determines whether or not a picture that is referenced for a forward motion vector (FMV) of the reference source MB is a decoded picture. When the picture is the decoded picture (Yes in operation S302), the process proceeds to operation S303. When the picture is not the decoded picture (No in operation S302), the process proceeds to operation S305. When the picture is to be decoded before decoding of a picture including the parallax MB, the block extractor 108 determines that the picture is the decoded picture, for example.
In operation S303, the block extractor 108 determines whether or not the evaluation value that is calculated on the basis of the difference coefficient (coefficient information) and the quantization scale is smaller than a predetermined value. The predetermined value is initially set to a sufficient large value. When the block extractor 108 determines that the evaluation value for the reference source MB to be evaluated is smaller than the predetermined value, the block extractor 108 updates the evaluation value so that the evaluation value is equal to the predetermined value. The evaluation value may be calculated according to the method according to the aforementioned second or third example. When the evaluation value is smaller than the predetermined value (Yes in operation S303), the process proceeds to operation S304. When the evaluation value is not smaller than the predetermined value (No in operation S303), the process proceeds to operation S305.
In operation S304, the block extractor 108 updates an appropriate reference source MB so that the macro-block evaluated using the evaluation value determined to be smaller than the predetermined value in operation S303 is set as the appropriate reference source MB.
In operation S305, the block extractor 108 determines whether or not another reference source MB for which the error MB is referenced exists. When the other reference source MB exists (Yes in operation S305), the process returns to operation S301. When the other reference source MB does not exist (No in operation S305), the operation of selecting an appropriate reference source MB is terminated. A reference source MB that is lastly updated in this process is the appropriate reference source MB. Thus, when a plurality of reference source MBs exist, an appropriate reference source MB may be selected. The appropriate reference source MB is an MB estimated to be similar to the error MB.
In the first embodiment, when a region (error region) in which an error occurs exists in a picture for the base side, a region that is included in a decoded picture for the dependent side and similar to the error region is estimated, and the region in which the error occurs may be concealed using data of the estimated region. This process may be achieved by performing the variable length decoding on data that has been coded according to the variable length coding and expresses several pictures before the image decoding of the data.
In the first embodiment, a parallax MB that is similar to an error MB for the base side and provided for the dependent side is extracted using parallax information for which the error MB for the base side is referenced. At this time, the parallax MB is yet to be decoded. Thus, the moving picture decoding device 100 specifies a reference source MB for which the parallax MB is referenced. Then, the moving picture decoding device 100 uses a motion vector of the specified reference source MB to extract, from a decoded picture, an MB that is to be referenced and similar to the parallax MB. Since the MB to be referenced is similar to the parallax MB and the parallax MB is similar to the error MB, the MB to be referenced is similar to the error MB. The moving picture decoding device 100 uses the MB (to be referenced) to perform the error concealment process on the error MB. Thus, in the moving picture decoding device 100, error resistance may be improved. From the perspective of the amount of generated coded data, an error occurs in a dependent image at a lower rate than a base image. When the MB to be referenced is extracted from the dependent image, the error resistance may be improved.
Coded data of a moving picture to be decoded is data coded according to the multi-view coding technique based on Moving Picture Expert Group (MPEG)-2, H.264 or the like. The parts that are included in the moving picture decoding device 100 illustrated in
In the example illustrated in
Next, a moving picture decoding device 100 according to a second embodiment is described. The second embodiment applies to the case in which parallax information for which an error MB is referenced cannot be acquired. One of the reasons that the parallax information for which the error MB is referenced cannot be acquired is that an error region is large and the parallax information is lost due to an error, for example. Another one of the reasons that the parallax information for which the error MB is referenced cannot be acquired may be that the parallax information for which the error MB is referenced does not exist. The moving picture decoding device 100 according to the second embodiment has the same configuration as the configuration illustrated in
It is considered that a parallax exist in a stereoscopic image and an image for the base side and an image for the dependent side are basically similar to each other. In the second embodiment, the error concealment process is performed using information of a macro-block located at the same position as an error MB and included in a picture for the dependent side.
The same position block extractor 501 specifies a picture that is provided for the dependent side and has been imaged at the same time as a picture that is provided for the base side and includes an error MB. Then, the same position block extractor 501 extracts a macro-block that is included in the picture for the dependent side and located at the same position as the error MB. The macro-block that is located at the same position as the error MB is hereinafter referred to as a same position MB. The same position block extractor 501 outputs information such as the macro-block address of the extracted same position MB to the motion vector calculator 502.
The motion vector calculator 502 calculates a concealment motion vector using a motion vector of the same position MB or a motion vector for which the same position MB is referenced. The calculation of the concealment motion vector is described later with reference to
The reference block extractor 503 extracts: the macro-block address of a macro-block (MB to be referenced) that is included in a decoded picture and to which the received concealment motion vector points; and the identification number of the picture including the macro-block. Then, the reference block extractor 503 outputs the extracted macro-block address and the extracted identification number to the concealing unit 107.
The concealing unit 107 performs the error concealment process using the macro-block referenced for the concealment motion vector.
Next, relationships among an error MB, a same position MB, a reference source MB and an MB to be referenced are described, and a calculation of a concealment motion vector is described in detail.
For example, when information of an MB 602 included in a picture P5 601 that is provided for the base side and to be decoded third among pictures for the base side is lost due to an error, a macro-block (same position MB) 603 that is located at the same position as the error MB 602 included in the picture P5 601 for the base side exists in a picture P5′ that is provided for the dependent side and to be decoded third among pictures for the dependent side.
Since the picture P5′ is decoded immediately after decoding of the picture P5 for the base side, a decoded image of the picture P5′ does not exist when the picture P5 is decoded.
Pictures B for the dependent side are decoded according to the variable length decoding in advance. Thus, information of vectors 611 and 612 of a reference source MB 604 for which the same position MB 603 of the picture P5′ is referenced exists in the buffer memory 124. The motion vector calculator 502 generates a temporary concealment motion vector (MVC′) 613 using the information of the vectors 611 and 612 of the reference source MB 604.
In the example illustrated in
The motion vector calculator 502 may treat the temporary concealment motion vector 613 as a concealment motion vector. In this case, a block 605 that is included in a picture I2′ is an MB to be referenced. In addition, the temporary concealment motion vector 613 may be treated as a concealment motion vector 614, and the error MB 602 may be referenced for the concealment motion vector 614. In this case, a block 606 that is included in a picture I2 and to which the motion vector 614 points from the error MB is an MB to be referenced.
In the example illustrated in
The picture P5′ is decoded immediately after decoding of the picture P5 for the base side. Thus, a decoded image of the picture P5′ does not exist when the picture P5 is decoded.
Pictures B for the dependent side are decoded according to the variable length decoding before in advance. A motion vector (MVO) 712 of the same position MB 703 included in the picture P5′ exists in the buffer memory 124. It is assumed that a block 704 that is included in a picture I2′ and referenced for the motion vector 712 has parallax information (MVD2) 713. The motion vector calculator 502 may treat the motion vector 712 as a concealment motion vector and treat the block 704 as an MB to be referenced.
In addition, the motion vector calculator 502 applies the parallax information 713 to the same position MB 703 and specifies a macro-block 707 that is shifted by an amount corresponding to the parallax information 713 from the same position MB 703. When the macro-block 707 has a motion vector 714 for which the picture I2′ is referenced, the motion vector calculator 502 may treat the motion vector 714 as a concealment motion vector and treat a block 706 referenced for the motion vector 714 as an MB to be referenced.
In addition, the motion vector calculator 502 may apply, to the error MB 702, a motion vector 715 that is the same as the motion vector 714. The motion vector calculator 502 may treat the motion vector 715 as a concealment motion vector and treat a block 708 referenced for the motion vector 715 as an MB to be referenced.
The evaluating unit 510 performs the same process as described in the first embodiment so that one macro-block is selected when a plurality of macro-blocks to be evaluated exist.
In the second embodiment, even when parallax information for which an error MB is referenced cannot be acquired, the error concealment process may be performed on the error MB for the base side using a macro-block of a picture for the dependent side. In the second embodiment, even when data is coded according to the multi-view video coding in which parallax information is not used, the process illustrated in
Next, a video reproducing device that is included in the moving picture decoding device 100 according to the first or second embodiment is described.
The input unit 801 has a circuit that is connected to an antenna 805 and receives, through the antenna 805, a video signal carried by a wireless radio wave, for example. The input unit 801 performs analog-to-digital conversion on the received video signal. The input unit 801 demodulates the digitized video signal according to a demodulation method corresponding to a modulation method used for the carrier wave. The input unit 801 performs an error correction process, such as forward error correction, on the demodulated video signal. The input unit 801 outputs, to the signal processing unit 802, the video signal that has been demodulated and subjected to the error correction process.
The signal processing unit 802 has at least one processor and a circuit located near the processor. The signal processing unit 802 decodes coded data that expresses a moving picture and is included in the video signal. Also, the signal processing unit 802 decodes a coded audio signal included in the video signal. In this case, it is assumed that the coded data of the moving picture is data coded according to the multi-view video coding. The signal processing unit 802 outputs the decoded data of the moving picture to the display unit 803. The signal processing unit 802 outputs the decoded audio signal to the audio output unit 804.
The display unit 803 is a display device such as a liquid crystal display, for example. The display unit 803 displays the moving picture data received from the signal processing unit 802.
The audio output unit 804 is a speaker, for example. The audio output unit 804 outputs the audio signal received from the signal processing unit 802.
The demultiplexer 811 divides the video signal into the coded data of the moving picture and the coded audio signal. The demultiplexer 811 outputs the coded data to the moving picture decoder 812 on a picture basis. The demultiplexer 811 outputs the coded audio signal to the audio decoder 813.
The moving picture decoder 812 may serve as the moving picture decoding device 100 described in the first or second embodiment. The moving picture decoder 812 decodes the coded moving picture data received from the demultiplexer 811. When an error occurs in any of pictures included in the coded data, the moving picture decoder 812 performs the concealment process to conceal a region in which the error occurs in the aforementioned manner. The moving picture decoder 812 outputs the decoded moving picture data to the display unit 803.
The audio decoder 813 decodes the coded audio signal received from the demultiplexer 811 using a decoding method corresponding to a coding method used for the coded audio signal. The coding method is MPEG-2 AAC, MPEG-4 AAC or the like.
In a third embodiment, the video reproducing device 800 has the moving picture decoding device described in the first or second embodiment. Thus, in the video reproducing device 800, error resistance may be improved, and a degradation (caused by a transmission error or the like) of the quality of an image may be suppressed.
Next, a mobile terminal that has the processing functions of the moving picture decoding device 100 described in the first or second embodiment is described.
The input/output controller 901 controls input from an input device (key input unit of the mobile terminal 900 or the like). The input/output controller 901 controls output to the display (monitor or the like) 907. The CPU 902 executes various programs. The CPU 902 is a so-called processor and controls the processes to be performed by the parts of the mobile terminal 900.
The auxiliary storage unit 903 and the main storage unit 904 store data that is necessary for the programs and execution of the programs. The auxiliary storage unit 903 has a moving picture decoding program that is used to perform the aforementioned processes to be performed by the moving picture decoding device 100. The wireless communication circuit 905 performs communication through the antenna 906 and a network. The antenna 906 transmits and receives a wireless signal and a video signal.
The CPU 902 receives the coded moving picture data received by the antenna 906 through the wireless communication circuit 905. The CPU 902 loads the aforementioned moving picture decoding program onto the main storage unit 904. The CPU 902 may perform the error concealment process (described in the first or second embodiments) by executing the loaded program.
In a fourth embodiment, the mobile terminal 900 performs the processes of the moving picture decoding device described in the first or second embodiments. Thus, in the mobile terminal 900, error resistance may be improved, and a degradation (caused by a transmission error or the like) of the quality of an image may be suppressed.
Even when an image that is provided for the base side and coded according to the multi-view video coding is decoded and reproduced as a two-dimensional video, error resistance may be improved by performing the error concealment process on an image that is provided for the dependent side and is not reproduced.
According to an aspect of the embodiments of the invention, any combinations of one or more of the described features, functions, operations, and/or benefits can be provided. A combination can be one or a plurality. The embodiments can be implemented as an apparatus (a machine) that includes hardware for performing the described features, functions, operations, and/or benefits, for example, hardware to execute instructions, for example, computing hardware (i.e., computing apparatus), such as (in a non-limiting example) any computer that can store, retrieve, process and/or output data and/or communicate (network) with other computers. According to an aspect of an embodiment, the described features, functions, operations, and/or benefits can be implemented by and/or use computing hardware and/or software. The moving picture decoding device 100, video reproducing device, mobile terminal device 900, can comprise a controller (CPU) (e.g., a hardware logic circuitry based computer processor that processes or executes instructions, namely software/program), computer readable media, transmission communication interface (network interface), and/or an output device, for example, a display device, and which can be in communication among each other through one or more data communication buses. In addition, an apparatus can include one or more apparatuses in computer network communication with each other or other apparatuses. In addition, a computer processor can include one or more computer processors in one or more apparatuses or any combinations of one or more computer processors and/or apparatuses. An aspect of an embodiment relates to causing and/or configuring one or more apparatuses and/or computer processors to execute the described operations. The results produced can be output to an output device, for example, displayed on the display. An apparatus or device refers to a physical machine, for example, a computer (physical computing hardware or machinery) that implement or execute instructions, for example, by way of software, which is code executed by computing hardware, and/or by way of computing hardware (e.g., in circuitry, etc.), to achieve the functions or operations being described. The functions of embodiments described can be implemented in any type of apparatus that can execute instructions or code. More particularly, programming or configuring or causing an apparatus or device, for example, a computer, to execute the described functions of embodiments of the invention creates a new machine where in case of a computer a general purpose computer in effect becomes a special purpose computer once it is programmed or configured or caused to perform particular functions of the embodiments of the invention pursuant to instructions from program software.
A program/software implementing the embodiments may be recorded on a computer-readable media, e.g., a non-transitory or persistent computer-readable medium. Examples of the non-transitory computer-readable media include a magnetic recording apparatus, an optical disk, a magneto-optical disk, and/or volatile and/or non-volatile semiconductor memory (for example, RAM, ROM, etc.). Examples of the magnetic recording apparatus include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape (MT). Examples of the optical disk include a DVD (Digital Versatile Disc), DVD-ROM, DVD-RAM (DVD-Random Access Memory), BD (Blue-ray Disk), a CD-ROM (Compact Disc-Read Only Memory), and a CD-R (Recordable)/RW. The program/software implementing the embodiments may be transmitted over a transmission communication path, e.g., a wire and/or a wireless network implemented via hardware. An example of communication media via which the program/software may be sent includes, for example, a carrier-wave signal.
All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.
Claims
1. A moving picture decoding device comprising:
- an receiving unit that receives data that has been coded according to multi-view coding and includes data obtained by coding, on a predetermined region basis, pictures that are included in a moving picture and provided for a plurality of views;
- a first decoder that decodes, on a predetermined region basis, coded data that is included in the data coded according to the multi-view coding and provided for a first view;
- a second decoder that decodes, on a predetermined region basis, coded data that is included in the data coded according to the multi-view coding and provided for a second view;
- a detector that detects a predetermined region that cannot be decoded and is among a plurality of predetermined regions to be decoded by the first decoder; and
- a concealing unit that conceals the predetermined region detected by the detector using decoded data of a predetermined region for the second view decoded by the second decoder.
2. The device according to claim 1,
- wherein the data for the second view is coded using a coding method in which a picture that is obtained by decoding the coded data for the first view can be used as a reference picture for the second view,
- wherein when a picture that is decoded by the first decoder is a reference picture to the second view, the second decoder decodes a predetermined region expressed by the coded data for the second view using the decoded picture for the first view that is the reference picture, and
- wherein when the detected predetermined region exists in the decoded picture for the first view that is the reference picture, the concealing unit conceals the detected predetermined region using the decoded data of the predetermined region for the second view decoded by the second decoder.
3. The device according to claim 2, further comprising:
- an extractor that extracts, from the coded data for the second view, parallax information that is referenced for the detected predetermined region that cannot be decoded; and
- a calculator that calculates a motion vector for concealment of the detected predetermined region based upon a first motion vector and a second motion vector of the detected predetermined region, when a reference source region of the decoded picture for the second view has the first motion vector and the second motion vector for which the extracted parallax information is referenced.
4. The device according to claim 3,
- wherein before the first decoder decodes a first picture that includes the detected predetermined region that cannot be decoded, the second decoder decodes a second picture that corresponds to the first picture and is included in the coded data for the second view, and the second decoder decodes a motion vector and parallax information from a plurality of pictures to be decoded after the decoding of the second picture.
5. The device according to claim 3,
- wherein when a plurality of reference source regions exist from the plurality of decoded pictures by the second decoder, the calculator selects a reference source region that has the second motion vector for which a decoded picture is referenced.
6. The device according to claim 5,
- wherein when a plurality of reference source regions that have second motion vectors for which the decoded picture is referenced exist, the calculator selects a reference source region based upon evaluation values that are calculated from difference coefficients of the reference source regions and quantization scales of the reference source regions.
7. The device according to claim 1, further comprising:
- a specifying unit that specifies a predetermined region as a predetermined region incapable of being decoded and is among predetermined regions included in a picture that is provided for the second view and has been imaged at same time as a picture that includes the predetermined region that cannot be decoded,
- wherein the concealing unit treats a third motion vector of the specified predetermined region as a motion vector for concealment, and conceals the specified predetermined region using a predetermined region that is among predetermined regions of decoded pictures for the first view and referenced predetermined regions of the second view for the motion vector for concealment.
8. The device according to claim 1, further comprising:
- a specifying unit that specifies a predetermined region as a predetermined region incapable of being decoded and is among predetermined regions included in a picture that is provided for the second view and has been imaged at same time as a picture that includes the predetermined region that cannot be decoded; and
- a calculator that calculates a motion vector for concealment based upon a fourth motion vector and a fifth motion vector of the predetermined region of the second view, when a reference source region of decoded picture for the second view has the fourth motion vector and the fifth motion vector for which the specified predetermined region is referenced,
- wherein the concealing unit conceals the specified predetermined region using a predetermined region that is among predetermined regions of decoded pictures for the first view and referenced predetermined regions of the second view for the motion vector for concealment.
9. A moving picture decoding method comprising:
- receiving data that has been coded according to multi-view coding and includes data obtained by coding, on a predetermined region basis, pictures that are included in a moving picture and provided for a plurality of views;
- decoding, on a predetermined region basis, coded data that is included in the data coded according to the multi-view coding and provided for a first view;
- decoding, on a predetermined region basis, coded data that is included in the data coded according to the multi-view coding and provided for a second view;
- detecting a predetermined region that cannot be decoded and is among a plurality of predetermined regions expressed by the coded data that is to be decoded and is provided for the first view; and
- concealing the detected predetermined region using decoded data of a predetermined region for the second view obtained by decoding the coded data for the second view.
10. The method according to claim 9,
- wherein the data for the second view is coded using a coding method in which a picture that is obtained by decoding the coded data for the first view can be used as a reference picture, for the second view,
- wherein in the decoding the coded data for the second view on a predetermined region basis, when a picture that is obtained by decoding the coded data for the first view on a predetermined region basis is a reference picture, a predetermined region that is expressed by the coded data for the second view is decoded using the decoded picture that is the reference picture, and
- wherein when the detected predetermined region detected in the detecting exists in the decoded picture for the first view that is the reference picture, the detected predetermined region is concealed in the concealing using the decoded data of the predetermined region decoded in the decoding of the coded data for the second view on a predetermined for the second view region basis.
11. The method according to claim 10, further comprising:
- extracting, from the coded data for the second view, parallax information that is referenced for the detected predetermined region that cannot be decoded is referenced; and
- calculating a motion vector for concealment of the detected predetermined region based upon a first motion vector and a second motion vector of the detected predetermined region, when a reference source region has the second motion vector and the first motion vector for which a predetermined region that has the extracted parallax information is referenced,
- wherein the detected predetermined region is concealed in the concealing using decoded data of a predetermined region that is referenced for the motion vector for concealment and included in a decoded picture for the second view.
12. The method according to claim 11,
- wherein in the decoding of the coded data for the second view on a detected predetermined region basis, before a first picture that includes the predetermined region that cannot be decoded is decoded in the decoding of the coded data for the first view on a predetermined region basis, a second picture that corresponds to the first picture and is included in the coded data for the second view is decoded, and a motion vector and parallax information are decoded from a plurality of pictures to be decoded after the decoding of the second picture.
13. The method according to claim 11,
- wherein in the calculating, when a plurality of reference source regions exist from the plurality of decoded pictures by the second decoder, a reference source region that has the second motion vector for which a decoded picture is referenced is selected.
14. The method according to claim 13,
- wherein in the calculating, when a plurality of reference source regions that have second motion vectors for which the decoded picture is referenced exist, a reference source region is selected on the basis of evaluation values that are calculated from difference coefficients of the reference source regions and quantization scales of the reference source regions.
15. The method according to claim 9, further comprising
- specifying a predetermined region as a predetermined region incapable of being decoded and is among predetermined regions included in a picture that is provided for the second view and has been imaged at same time as a picture including the predetermined region that cannot be decoded,
- wherein in the concealing, a third motion vector of the specified predetermined regions is treated as a motion vector for concealment, and the specified predetermined region is concealed using a predetermined region that is referenced for the motion vector for concealment and is among predetermined regions included in a decoded picture for the first view.
16. The method according to claim 9, further comprising:
- specifying a predetermined region as a the predetermined region incapable of being decoded and is among predetermined regions included in a picture that is provided for the second view and has been imaged at the same time as a picture including the predetermined region that cannot be decoded; and
- calculating a motion vector for concealment based upon a fourth motion vector and a fifth motion vector of the predetermined region of the second view, when a reference source region has the fourth motion vector and the fifth motion vector for which the specified predetermined region is referenced,
- wherein in the concealing, the specified predetermined region is concealed using a predetermined region that is referenced predetermined regions of the second view for the motion vector for concealment and is among predetermined regions included in a decoded picture for the first view.
17. An integrated circuit comprising:
- an receiving unit that receives data that has been coded according to multi-view coding and includes data obtained by coding, on a predetermined region basis, pictures that are included in a moving picture and provided for a plurality of views;
- a first decoder that decodes, on a predetermined region basis, coded data that is included in the data coded according to the multi-view coding and provided for a first view;
- a second decoder that decodes, on a predetermined region basis, coded data that is included in the data coded according to the multi-view coding and provided for a second view;
- a detector that detects a predetermined region that cannot be decoded and is among a plurality of predetermined regions to be decoded by the first decoder; and
- a concealing unit that conceals the predetermined region detected by the detector using decoded data of a predetermined region for the second view decoded by the second decoder.
18. The integrated circuit according to claim 17,
- wherein the data for the second view is coded using a coding method in which a picture that is obtained by decoding the coded data for the first view can be used as a reference picture,
- wherein when a picture that is decoded by the first decoder is a reference picture, the second decoder decodes a predetermined region expressed by the coded data for the second view using the decoded picture that is the reference picture, and
- wherein when the predetermined region detected by the detector exists in the decoded picture that is the reference picture, the concealing unit conceals the detected predetermined region using the decoded data of the predetermined region decoded by the second decoder.
19. The integrated circuit according to claim 18, further comprising:
- an extractor that extracts, from the coded data for the second view, parallax information for which the predetermined region that cannot be decoded is referenced; and
- a calculator that calculates a motion vector for concealment on the basis of a first motion vector and a second motion vector when a reference source region has the second motion vector and the first motion vector for which a predetermined region that has the extracted parallax information is referenced,
- wherein the concealing unit conceals the detected predetermined region using decoded data of a predetermined region that is included in a decoded picture for the second view and referenced for the motion vector for concealment.
20. The integrated circuit according to claim 19,
- wherein before the first decoder decodes a first picture that includes the region that cannot be decoded, the second decoder decodes a second picture that corresponds to the first picture and is included in the coded data for the second view, and the second decoder decodes a motion vector and parallax information from a plurality of pictures to be decoded after the decoding of the second picture.
Type: Application
Filed: Aug 30, 2011
Publication Date: Mar 22, 2012
Patent Grant number: 9210448
Applicant: FUJITSU LIMITED (Kawasaki)
Inventor: Akihiro YAMORI (Kawasaki)
Application Number: 13/220,965
International Classification: H04N 7/26 (20060101);