Scalable Image Coding and Decoding

Abstract

The invention relates to a scalable video (de)coding method for wireless transmission of high definition television signals. Scalable means that the bitstream contains successively smaller quality refinements and that the bitstream can be truncated. The video images are divided in slices, and each slice is divided in blocks of 8×8 pixels. For each block, an optimal encoding method is chosen. Depending on whether the block is found to contain natural or synthetic image content, transform coding (DCT) or graphic coding is applied. Because the different encoding modes have different properties as regards picture quality, the bitstream format has to enable the encoder to very flexibly choose which bits to send first. The bitstream format in accordance with the invention consists of a multitude of scans (31-34) through the coded data of a series of individual blocks, e.g. a slice. In each scan, the encoder decides whether it will include data for natural blocks (BS1,BS2), for synthetic blocks (BS0,BS1), or both. For the DCT blocks, it can further choose between DC coefficient bits (22) and AC coefficient bits (23). For example, the encoder can choose to first send a few scans of DCT DC bits, then send a few scans of graphics bits and then send some DCT AC bits. The decisions that the encoder makes are signaled to the decoder in the form of flags that precede each block part (e.g. encoded bit plane) and/or plurality of blocks (e.g. a slice).

Description

FIELD OF THE INVENTION

The invention relates to a method and device for encoding images into a scalable bitstream. The invention also relates to a corresponding method and device for decoding such a scalable bitstream, an encoded image signal in the form of a such a (possibly truncated) scalable bitstream, and a storage medium having recorded thereon such an encoded image signal. The invention is particularly useful for encoding the video images of a high definition television (HDTV) signal.

BACKGROUND OF THE INVENTION

Scalable video coding is envisaged for wireless transmission of High Definition television signals (WirelessHD), in particular for the purpose of invisibly connecting a wall-mounted HDTV flat screen monitor to an external HDTV signal source. Scalable means that a bitstream produced by the encoder can be truncated.

A prior art method of obtaining a scalable bitstream is disclosed in U.S. Pat. No. 6,462,681. The prior art comprises dividing each image in image blocks, transform encoding each image block in corresponding block bitstreams, and forming the scalable bitstream by iteratively scanning the blockstreams, including in each scan a part of the data in the block bitstreams.

In experimental evaluations in the context of WirelessHD, the inventors found that an improvement of the compression performance was desired for certain computer-graphics content. For WirelessHD, both natural image content and graphics content should be supported. The prior art method is designed for natural image content and is not specifically optimized for graphics content.

OBJECT AND SUMMARY OF THE INVENTION

It is an object of the invention to further improve the prior art method.

To this end, the encoding method in accordance with the invention comprises the step of encoding each image block in accordance with one of a plurality of different encoding modes, to obtain corresponding block bitstreams comprising one or more types of data representative of said encoding mode. While forming the scalable bitstream by iteratively scanning the blockstreams, each scan comprising the steps of selecting at least one of said types of data, and including flags in the scalable bitstream to indicate the selected types of data.

The invention provides a scalable bitstream format that allows a very flexible combination of the various block bitstreams, which enables an optimized bit-allocation. The multiple encoding modes have different importance/priorities as regards the picture quality of the decoded image data. The invention allows the scalable video bitstream to be organized such that the largest quality improvements are achieved first, followed by successively smaller quality refinements when more of the bitstream is received. In this way, an optimal rate-distortion trade-off is achieved. Because of the different importance of the modes for the objective/subjective quality, the bitstream format enables the encoder to very flexibly choose which bits to send first.

Similar to known video coding methods, the video images are preferably divided in slices, and each slice is divided in blocks of 8×8 pixels. The 8×8 pixel blocks are intraframe coded, which means that no information from other frames is used.

Each block is encoded in accordance with one of three encoding modes:

In a natural mode, the block is considered to represent a part of a natural image. In that case, it is DCT transformed and converted into a scalable block bitstream. Preferably, scalable bit plane coding is used as disclosed in, for example, WO 2005/112467.

In a synthetic mode, or scalable graphics mode, the block is found to contain graphics contents that can more efficiently be coded as a specific graphic scalable block bitstream.

In a skip mode, a block is represented by a 1 bit flag indicating that it was found to be identical to the previous block.

The bitstream format in accordance with the invention consists of a multitude of scans/iterations through the scalable bitstream of the individual blocks. In each scan, the encoder decides whether it includes bits in the bitstream for the natural/DCT blocks, or for the synthetic/graphics blocks, or both. For the DCT blocks, it can further choose between DC coefficient bits and AC coefficient bits. For example, the encoder can choose to first send a few scans of DCT DC bits, then send a few scans of graphics bits and then send some DCT AC bits. The scans can be a mix of different mode bits or subsequent scans of single-mode bits. The decisions that the encoder makes are signaled to the receiver in the form of flags that precede each block part (e.g. encoded bit plane) and/or plurality of blocks (e.g. a slice).

In an embodiment of the invention, the bitstream format allows the block scan order to change for each scan. For example, a different scan order can be used for DCT blocks or graphics blocks. Typically, for natural images it is advantageous to first scan/allocate bits to the blocks in the center of the image.

In a further embodiment, the format allows to indicate per block whether bits are sent or not. This allows very detailed bit allocation (enabling e.g. emphasizing the quality of a region of interest).

The encoder may further have the choice per scan on whether or not to use arithmetic coding for more efficient compression of certain DCT-AC coefficient data.

The method may further comprise the step of dividing each image in image slices and include in each slice flags indicating the encoding modes applied to said slice. Indicating the modes that occur during compression of an image slice (8 image lines) in the beginning of the bitstream optimizes (i.e. minimizes the bits required for) the encoding of the block mode types.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a flow diagram illustrating the method of encoding a video signal in accordance with the invention.

FIG. 2 diagrammatically shows scalable block bitstreams obtained in a graphic encoding mode and a DCT encoding mode.

FIG. 3 shows an example illustrating the scanning process in accordance with the invention.

FIG. 4 shows a flow diagram illustrating the method of decoding a video bitstream signal in accordance with the invention.

FIG. 5 shows a block diagram of an entertainment system comprising a encoder device and a decoder device in accordance with the invention.

DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 shows a flow diagram illustrating the method of encoding a video signal in accordance with the invention. In a step 11, a video image is divided in 8-lines high slices, and each slice is divided into blocks of 8×8 pixels.

In a step 12, each block of the current slice is encoded by either a discrete cosine transform (DCT) or a graphic compression method, depending on whether the block is found to contain natural contents or graphic contents. A skip mode may also be defined, indicating that the block is identical to the previous one. The block encoding algorithms are chosen such that they yield scalable block bit streams. Such a scalable bitstream may be truncated at any position. The less is truncated, the smaller the residual error when the block is reconstructed.

FIG. 2 diagrammatically shows under A a scalable block bitstream obtained in the graphic encoding mode. In this mode, a graphic encoder applies a possibly lossless compression algorithm to each individual 8×8 bit plane of an image block. The block bitstream thus obtained comprises a series of eight (the pixel bitdepth) variable-length parts 21 of graphic data, each representing a bit plane of pixel values, in decreasing order of significance.

FIG. 2 diagrammatically shows under B a scalable block bit stream obtained in the natural encoding mode. In this mode, the 8×8 pixel block is subjected to a Discrete Cosine Transform (DCT), and then converted into a scalable block bitstream by individually compressing each bit plane of the coefficients as disclosed in Applicants International Patent Application WO 2005/112467. The block bit stream now comprises eleven (the coefficient bitdepth) bit planes of coefficient values, in decreasing order of significance. As shown in FIG. 2, each bit plane comprises a dc-bit 22 and a variable-length part 23 of ac data.

Returning now to FIG. 1, a routine is carried out in which a video bitstream representing the current slice is formed. First, in a step 13, an array block_modes( ) is generated to identify, for each block in the slice, which block encoding mode has been used to encode it. The routine then proceeds by iteratively scanning (step 14) the block bitstreams that were generated in step 12. In each scan 14, the encoding process selects a part of the graphic block bitstreams and/or the natural block bitstreams. With respect to the natural block bitstream, the process selects the dc-bit and/or the ac data part of the respective coefficient bit plane. Each selectable part of a block bitstream represents a given refinement of the image quality. The selections are such that successively smaller refinements are added to the video bitstream as the scanning process proceeds.

In each scan 14, flags indicating which types of encoded data (graphics, dc-bit, ac-data) were selected, and thus identifting the respective encoding modes, are added to the bitstream (step 141) prior to the actual refinement data parts (step 142). The selections made by the encoder are thus signaled to the receiver. The scanning process proceeds until all the block bitstreams of the slice have been completely processed (steps 15, 16). The above process is then carried out for the next slice (step 17).

FIG. 3 shows an example illustrating the scanning process in accordance with the invention. The figure shows four block bit streams BS0 . . . BS3 of the first four image blocks (B0 . . . B3) of a slice. The first and fourth blocks were found to be graphic blocks. The corresponding block bitstreams BS0 and BS3 have eight graphic encoded bit planes 21 in decreasing order of significance, some of which are shown in FIG. 3. The second and third blocks are natural blocks. The corresponding block bitstreams BS1 and BS2 have eleven dc-bits 22 and eleven encoded ac bit planes 23, some of which are shown in FIG. 3.

As defined by bold dots along an arrow 31, the first scan through the block streams conveys the most significant bit plane of the DCT coefficients of the natural blocks (B1, B2), that is, the respective bits of the DC coefficient as well as the AC coefficients. The second scan 32 defines the next (i.e. second) bit of the DC coefficient of the natural blocks. The third scan 33 provides graphic data as well as AC coefficient data of natural blocks, that is, the first graphic bit planes of the graphic blocks (B0,B3) and the next (i.e. second) AC coefficient bit planes of the natural blocks (B1,B2). In the fourth scan 34, the second graphic bit planes are conveyed, etc.

The following Table I shows the resulting video bitstream for the slice. The flags indicating in each scan the data types that follow are denoted dct_dc_data, dct_ac_data, and graph_data. The right column shows the actual bitstream data, where “1” and “0” denote flag values, “b” represents one data bit, and “d” denotes a (possibly variable) number of data bits.

TABLE I
block_modes( )       d
dct_dc_data          1
dct_ac_data          1
graph_data           0
BS1 1st dc bit       b
BS1 1st ac bit plane d
BS2 1st dc bit       b
BS2 1st ac bit plane d
end_of_stream        0
dct_dc_data          1
dct_ac_data          0
graph_data           0
BS1 2nd dc bit       b
BS2 2nd dc bit       b
end_of_stream        0
dct_dc_data          0
dct_ac_data          1
graph_data           1
BS0 1st bit plane    d
BS1 2nd ac bit plane d
BS2 2nd ac bit plane d
BS3 1st bit plane    d
end_of_stream        0
dct_dc_data          0
dct_ac_data          0
graph_data           1
BS0 2nd bit plane    d
BS3 2nd bit plane    d
end_of_stream        0
. . .
end_of_stream        1

FIG. 4 shows a flow diagram illustrating the method of decoding the (possibly truncated) video bitstream signal and reconstruct the image slice. Step 41 is an initialization step in which:

The block encoding modes are read and stored. For example, with respect to the bitstream shown in Table I, the receiver now knows that blocks B0,B3 are graphic blocks and blocks B1,B2 are natural blocks.

Respective block memory arrays are created.

For each block, a bit plane counter (or ‘block completed’ variable) is initialized. Natural blocks have separate bit plane counters for dc data and ac data.

In a step 42, the receiver identifies, on the basis of the data type flags and knowledge of the encoding modes, to which blocks the refinement data parts in a current scan apply. For example, the first scan in the bitstream of Table I is flagged to have dc data and ac data. As these data types apply to natural blocks, the receiver knows that the first scan will contain dc and ac data for blocks B1 and B2.

In step 43, a data part is taken from the bitstream. The data is stored in the current bit plane of the eligible block and the associated bit plane counter is subsequently incremented. This procedure is repeated (steps 44,45) for all blocks that are addressed in the current scan. In the present example, the first (most significant) coefficient bit plane of DCT blocks B1 and B2 is thus reconstructed in the first scan.

In a step 46, the end_of_stream flag that concludes each scan (cf. Table 1) is checked. As long as its value is ‘0’, another scan will follow and will be processed (step 47). The end_of_stream flag value ‘1’ signifies the end of the video bitstream for the current slice. All blocks of the slice can then be decoded and displayed (step 48).

The invention is also illustrated in the form of the following syntax description. The syntax elements are indicated in bold and the number of bits in the right column. The relevant flags are one bit. Entities having a length X denote coded data representing the contents (bit plane(s)) identified by the flags preceeding it. It describes the syntax of the compressed video data stream for a slice in a formal manner and, at the same time, also describes the decoding process.

TABLE II
slice(nblocks,bitdepth,color_format){
codec_version /* fixed to 0001 */ 4
nat_mode_used                    1
graph_mode_used                  1
skip_mode_used                   1
block_order_per_bit_scan         1
if (block_order_per_bit_scan==0) {
spatial_scan_order               1
comp_scan_order                  1
}
scan_order( )
block_modes( )                   x
if (nat_mode_used ==1) {
nat_mode_init( )
}
if (graph_mode_used == 1) {
graph_mode_init( )
}
for (i=0; i<nblocks; i++) {
block_completed_dct_dc[i]=0
block_completed_dct_ac[i]=0
block_completed_graph[i]=0
}
do {
block_bit_scan( )                x
end_of_stream                    1
} while (end_of_stream != 1)
if (nat_mode_used == 1) {
nat_mode_post( )
}
if (graph_mode_used == 1) {
graph_mode_post( )
}
if (skip_mode_used == 1) {
skip_mode_post( )
}
}

TABLE III
block_bit_scan( ){
if (nat_mode_used) {
dct_dc_data         1
dct_ac_data         1
} else {
dct_dc_data=0
dct_ac_data=0
}
if (dct_ac_data==1) {
arithcod            1
} else {
arithcod=0
}
if (graph_mode_used) {
graph_data          1
} else {
graph_data=0
}
data_each_block     1
if (block_order_per_bit_scan==1) {
spatial_scan_order  1
comp_scan_order     1
scan_order( )
}
if (arithcod==1) {
arithcod_bytes      9
for (i=0; i<arithcod_bytes; i++) {
arithcod_buf[i]     8
}
}
for (i=0; i<nblocks; i++) {
if (block_mode[scan_block[i]]==0) { /* DCT block */
if (((dct_dc_data==1) &&
(block_completed_dct_dc[scan_block[i]]==0)) ||
((dct_ac_data==1) &&
(block_completed_dct_ac[scan_block[i]]==0))) {
if (data_each_block==0) {
data_this_block     1
} else {
data_this_block=1
}
if (data_this_block==1) {
if ((dct_dc_data==1) &&
(block_completed_dct_dc[scan_block[i]]==0)){
scan_dct_dc_data( ) x
}
if ((dct_ac_data==1) &&
(block_completed_dct_ac[scan_block[i]]=0)){
scan_dct_ac_data( ) x
}
}
}
}
if (block_mode[scan_block[i]]==1) { /* graphic block */
if ((graph_data==1) &&
(block_completed_graph[scan_block[i]]==0)){
if (data_each_block==0) {
data_this_block     1
} else {
data_this_block=1
}
if (data_this_block==1) {
scan_graph_data( )  x
}
}
}
}
}

Table II defines the syntax at slice level. The table mentions, inter alia, the presence in the bitstream of the array block_modes( ) described above in connection with step 13 in FIG. 1. The iterative scans described above in connection with step 14 in FIG. 1 are denoted block_bit_scan( ).

Table III defines the syntax of block_bit_scan( ) in more details. The flags dct_dc_data, dct_ac_data, and graph_data in this table correspond to the flags described before in connection with step 14 in FIG. 1. The actual image data is denoted scan_dct_dc_data( ), scan_dct_ac_data( ), and scan_graph_data( ) in the table.

The above syntax also defines a number of additional features of further embodiments of the invention, for example:

Flags nat_mode_used, graph_mode_used, and skip_mode_used in Table II denote which coding modes apply to the part of the data that follows. Using such flag at slice level improves the efficiency of sending flags at scan level. For example, transmitting the flag graph_data in each scan may be omitted if the graphics encoding mode has not been used in a slice at all.

A subroutine scan_order( ) in Table II defines the order in which the blocks of a slice are scanned, by assigning to spatial order index i a scan order index scan_block[i]. This allows the encoder to scan the blocks in the center of an image first.

A block_order_per_bit_scan flag (Table II) denotes whether the order of blocks in the scan is the same for all the scans.

A flag data_each_block (Table III) denotes whether the relevant scan comprises data for each block.

A flag arithcod in Table III indicates per scan whether the more efficient arithmetic coding is used to compress the AC coefficient bit plane.

FIG. 5 shows a block diagram of an entertainment system comprising a encoder device and a decoder device in accordance with the invention. The entertainment system comprises a Bluray disc player 52 and a flat TV monitor 54. The player 52 plays a Bluray disc 51 and sends the HD signal through a wireless HD channel 53 to the monitor. The player 52 comprises a playback module 521, a video encoder 522, and a wireless transmitting unit 523. The monitor comprises a wireless receiving unit 541, a video decoder 542, and a display 543. The video encoder 522 encodes each video image into a scalable bitstream in accordance with the encoding method described hereinbefore with reference to FIG. 1. The video decoder 542 decodes the received bitstream in accordance with the decoding method described hereinbefore with reference to FIG. 4.

In summary, a flexible and efficient bitstream format for a multi-mode scalable image/video codec is presented. Scalable means that the bitstream contains successively smaller quality refinements and that the bitstream can be truncated. The video images are divided in slices, and each slice is divided in blocks of 8×8 pixels. For each block, an optimal encoding method is chosen. Depending on whether the block is found to contain natural or synthetic image content, transform coding (DCT) or graphic coding is applied. Because the different encoding modes have different properties as regards picture quality, the bitstream format has to enable the encoder to very flexibly choose which bits to send first.

The bitstream format in accordance with the invention consists of a multitude of scans (31-34) through the coded data of a series of individual blocks, e.g. a slice. In each scan, the encoder decides whether it will include data for the natural blocks (BS1,BS2), for the synthetic blocks (BS0,BS1), or both. For the DCT blocks, it can further choose between DC coefficient bits (22) and AC coefficient bits (23). For example, the encoder can choose to first send a few scans of DCT DC bits, then send a few scans of graphics bits and then send some DCT AC bits. The decisions that the encoder makes are signaled to the decoder in the form of flags that precede each block part (e.g. encoded bit plane) and/or plurality of blocks (e.g. a slice).

The invention can typically be applied in the areas of image- and video compression, particularly for the next generation of Wireless HD.

Claims

1. A method of encoding an image into a scalable bitstream, comprising the steps of:

dividing the image in image blocks;

encoding each image block in accordance with one of a plurality of different encoding modes, to obtain corresponding block bitstreams comprising one or more types of data representative of said encoding mode;

forming the scalable bitstream by iteratively scanning the block bitstreams, each scan comprising the steps of:

selecting at least one of said types of data;

including in the scalable bitstream data of the selected types from the block bitstreams;

including in the scalable bitstream flags indicating the selected types of data.

2. The method as claimed in claim 1, wherein one of said encoding modes is transform coding, the flags further indicating whether the included data comprises one or more bits of a DC coefficient and/or one or more bits of one or more AC coefficients.

3. The method as claimed in claim 2, wherein the AC coefficients of an image block are divided into bit planes, each bit plane is compressed, and the amount of data from a block bit stream being included in the scalable block bit stream corresponds to a compressed bit plane.

4. The method as claimed in claim 2, further comprising the step of including, in each scan, an indicator as to whether arithmetic coding is applied to said coefficients.

5. The method as claimed in claim 1, wherein one of said encoding modes is a graphics coding mode.

6. The method as claimed in claim 1, wherein one of said encoding modes is a skip mode in which an image block is represented by an indication that it is identical to the previous image block.

7. The method as claimed in claim 1, further comprising the step of including, in each scan, the order of image blocks being scanned.

8. The method as claimed in claim 1, further comprising the step of including, in each scan, a flag to indicate whether the scan includes data for each block.

9. The method as claimed in claim 8, further comprising the step of including, for each block in each scan, a flag to indicate whether the scan includes data for said block.

10. The method as claimed in claim 1, further comprising the step of dividing each image in image slices, the step of forming the scalable bitstream being successively applied to said image slices, and further including, in the scalable bitstream for each slice, flags to indicate the encoding modes applied to the image blocks of said slice.

11. A device for encoding images into a scalable bitstream, the device being arranged to carry out the method as claimed in claim 1.

12. A method of decoding an encoded image signal received in the form of a possibly truncated scalable bitstream comprising data of iteratively scanned block bitstreams representing image blocks being encoded in accordance with one of a plurality of different encoding modes and comprising one or more types of data representative of said encoding mode, each scan of the block streams including flags indicating the types of data in said scan, the method comprising the step of reconstructing the block bit streams in accordance with the flags, and decoding the reconstructed image blocks.

13. A device for decoding an encoded image signal received in the form of a possibly truncated bitstream, the device being arranged to carry out the method as claimed in claim 12.

14. An encoded image signal in the form of a possibly truncated scalable bitstream comprising data of iteratively scanned block bitstreams representing image blocks being encoded in accordance with one of a plurality of different encoding modes and comprising one or more types of data representative of said encoding mode, each scan of the block streams including flags indicating the types of data in said scan.

15. A storage medium having recorded thereon an encoded image signal as claimed in claim 14.