Data compression system and methods

- Realtime Data LLC

Systems and methods for providing fast and efficient data compression using a combination of content independent data compression and content dependent data compression. In one aspect, a method for compressing data comprises the steps of: analyzing a data block of an input data stream to identify a data type of the data block, the input data stream comprising a plurality of disparate data types; performing content dependent data compression on the data block, if the data type of the data block is identified; performing content independent data compression on the data block, if the data type of the data block is not identified.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No. 10/668,768, which is a Continuation of U.S. patent application Ser. No. 10/016,355, now U.S. Pat. No. 6,624,761, which is a Continuation-In-Part of U.S. patent application Ser. No. 09/705,446, now U.S. Pat. No. 6,309,424, which is a Continuation of U.S. patent application Ser. No. 09/210,491, now U.S. Pat. No. 6,195,024, each of which is hereby incorporated by reference herein in its entirety.

BACKGROUND

1. Technical Field

The present invention relates generally to a data compression and decompression and, more particularly, to systems and methods for data compression using content independent and content dependent data compression and decompression.

2. Description of Related Art

Information may be represented in a variety of manners. Discrete information such as text and numbers are easily represented in digital data. This type of data representation is known as symbolic digital data. Symbolic digital data is thus an absolute representation of data such as a letter, figure, character, mark, machine code, or drawing,

Continuous information such as speech, music, audio, images and video, frequently exists in the natural world as analog information. As is well known to those skilled in the art, recent advances in very large scale integration (VLSI) digital computer technology have enabled both discrete and analog information to be represented with digital data. Continuous information represented as digital data is often referred to as diffuse data. Diffuse digital data is thus a representation of data that is of low information density and is typically not easily recognizable to humans in its native form.

There are many advantages associated with digital data representation. For instance, digital data is more readily processed, stored, and transmitted due to its inherently high noise immunity. In addition, the inclusion of redundancy in digital data representation enables error detection and/or correction. Error detection and/or correction capabilities are dependent upon the amount and type of data redundancy, available error detection and correction processing, and extent of data corruption.

One outcome of digital data representation is the continuing need for increased capacity in data processing, storage, and transmittal. This is especially true for diffuse data where increases in fidelity and resolution create exponentially greater quantities of data. Data compression is widely used to reduce the amount of data required to process, transmit, or store a given quantity of information. In general, there are two types of data compression techniques that may be utilized either separately or jointly to encode/decode data: lossless and lossy data compression.

Lossy data compression techniques provide for an inexact representation of the original uncompressed data such that the decoded (or reconstructed) data differs from the original unencoded/uncompressed data. Lossy data compression is also known as irreversible or noisy compression. Entropy is defined as the quantity of information in a given set of data. Thus, one obvious advantage of lossy data compression is that the compression ratios can be larger than the entropy limit, all at the expense of information content. Many lossy data compression techniques seek to exploit various traits within the human senses to eliminate otherwise imperceptible data. For example, lossy data compression of visual imagery might seek to delete information content in excess of the display resolution or contrast ratio.

On the other hand, lossless data compression techniques provide an exact representation of the original uncompressed data. Simply stated, the decoded (or reconstructed) data is identical to the original unencoded/uncompressed data. Lossless data compression is also known as reversible or noiseless compression. Thus, lossless data compression has, as its current limit, a minimum representation defined by the entropy of a given data set.

There are various problems associated with the use of lossless compression techniques. One fundamental problem encountered with most lossless data compression techniques are their content sensitive behavior. This is often referred to as data dependency. Data dependency implies that the compression ratio achieved is highly contingent upon the content of the data being compressed. For example, database files often have large unused fields and high data redundancies, offering the opportunity to losslessly compress data at ratios of 5 to 1 or more. In contrast, concise software programs have little to no data redundancy and, typically, will not losslessly compress better than 2 to 1.

Another problem with lossless compression is that there are significant variations in the compression ratio obtained when using a single lossless data compression technique for data streams having different data content and data size. This process is known as natural variation.

A further problem is that negative compression may occur when certain data compression techniques act upon many types of highly compressed data. Highly compressed data appears random and many data compression techniques will substantially expand, not compress this type of data.

For a given application, there are many factors that govern the applicability of various data compression techniques. These factors include compression ratio, encoding and decoding processing requirements, encoding and decoding time delays, compatibility with existing standards, and implementation complexity and cost, along with the is adaptability and robustness to variations in input data. A direct relationship exists in the current art between compression ratio and the amount and complexity of processing required. One of the limiting factors in most existing prior art lossless data compression techniques is the rate at which the encoding and decoding processes are performed. Hardware and software implementation tradeoffs are often dictated by encoder and decoder complexity along with cost.

Another problem associated with lossless compression methods is determining the optimal compression technique for a given set of input data and intended application. To combat this problem, there are many conventional content dependent techniques that may be utilized. For instance, file type descriptors are typically appended to file names to describe the application programs that normally act upon the data contained within the file. In this manner data types, data structures, and formats within a given file may be ascertained. Fundamental limitations with this content dependent technique include:

(1) the extremely large number of application programs, some of which do not possess published or documented file formats, data structures, or data type descriptors;

(2) the ability for any data compression supplier or consortium to acquire, store, and access the vast amounts of data required to identify known file descriptors and associated data types, data structures, and formats; and

(3) the rate at which new application programs are developed and the need to update file format data descriptions accordingly.

An alternative technique that approaches the problem of selecting an appropriate lossless data compression technique is disclosed, for example, in U.S. Pat. No. 5,467,087 to Chu entitled “High Speed Lossless Data Compression System” (“Chu”). FIG. 1 illustrates an embodiment of this data compression and decompression technique. Data compression 1 comprises two phases, a data pre-compression phase 2 and a data compression phase 3. Data decompression 4 of a compressed input data stream is also comprised of two phases, a data type retrieval phase 5 and a data decompression phase 6. During the data compression process 1, the data pre-compressor 2 accepts an uncompressed data stream, identifies the data type of the input stream, and generates a data type identification signal. The data compressor 3 selects a data compression method from a preselected set of methods to compress the input data stream, with the intention of producing the best available compression ratio for that particular data type.

There are several limitations associated with the Chu method. One such limitation is the need to unambiguously identify various data types. While these might include such common data types as ASCII, binary, or unicode, there, in fact, exists a broad universe of data types that fall outside the three most common data types. Examples of these alternate data types include: signed and unsigned integers of various lengths, differing types and precision of floating point numbers, pointers, other forms of character text, and a multitude of user defined data types. Additionally, data types may be interspersed or partially compressed, making data type recognition difficult and/or impractical. Another limitation is that given a known data type, or mix of data types within a specific set or subset of input data, it may be difficult and/or impractical to predict which data encoding technique yields the highest compression ratio.

Accordingly, there is a need for a data compression system and method that would address limitations in conventional data compression techniques as described above.

SUMMARY OF THE INVENTION

The present invention is directed to systems and methods for providing fast and efficient data compression using a combination of content independent data compression and content dependent data compression. In one aspect of the invention, a method for compressing data comprises the steps of:

analyzing a data block of an input data stream to identify a data type of the data block, the input data stream comprising a plurality of disparate data types;

performing content dependent data compression on the data block, if the data type of the data block is identified;

performing content independent data compression on the data block, if the data type of the data block is not identified.

In another aspect, the step of performing content independent data compression comprises: encoding the data block with a plurality of encoders to provide a plurality of encoded data blocks; determining a compression ratio obtained for each of the encoders; comparing each of the determined compression ratios with a first compression threshold; selecting for output the input data block and appending a null compression descriptor to the input data block, if all of the encoder compression ratios do not meet the first compression threshold; and selecting for output the encoded data block having the highest compression ratio and appending a corresponding compression type descriptor to the selected encoded data block, if at least one of the compression ratios meet the first compression threshold.

In another aspect, the step of performing content dependent compression comprises the steps of: selecting one or more encoders associated with the identified data type and encoding the data block with the selected encoders to provide a plurality of encoded data blocks; determining a compression ratio obtained for each of the selected encoders; comparing each of the determined compression ratios with a second compression threshold; selecting for output the input data block and appending a null compression descriptor to the input data block, if all of the encoder compression do not meet the second compression threshold; and selecting for output the encoded data block having the highest compression ratio and appending a corresponding compression type descriptor to the selected encoded data block, if at least one of the compression ratios meet the second compression threshold.

In yet another aspect, the step of performing content independent data compression on the data block, if the data type of the data block is not identified, comprises the steps of: estimating a desirability of using of one or more encoder types based one characteristics of the data block; and compressing the data block using one or more desirable encoders.

In another aspect, the step of performing content dependent data compression on the data block, if the data type of the data block is identified, comprises the steps of: estimating a desirability of using of one or more encoder types based on characteristics of the data block; and compressing the data block using one or more desirable encoders.

In another aspect, the step of analyzing the data block comprises analyzing the data block to recognize one of a data type, data structure, data block format, file substructure, and/or file types. A further step comprises maintaining an association between encoder types and data types, data structures, data block formats, file substructure, and/or file types.

In yet another aspect of the invention, a method for compressing data comprises the steps of:

analyzing a data block of an input data stream to identify a data type of the data block, the input data stream comprising a plurality of disparate data types;

performing content dependent data compression on the data block, if the data type of the data block is identified;

determining a compression ratio of the compressed data block obtained using the content dependent compression and comparing the compression ratio with a first compression threshold; and

performing content independent data compression on the data block, if the data type of the data block is not identified or if the compression ratio of the compressed data block obtained using the content dependent compression does not meet the first compression threshold.

Advantageously, the present invention employs a plurality of encoders applying a plurality of compression techniques on an input data stream so as to achieve maximum compression in accordance with the real-time or pseudo real-time data rate constraint. Thus, the output bit rate is not fixed and the amount, if any, of permissible data quality degradation is user or data specified.

These and other aspects, features and advantages of the present invention will become apparent from the following detailed description of preferred embodiments, which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block/flow diagram of a content dependent high-speed lossless data compression and decompression system/method according to the prior art;

FIG. 2 is a block diagram of a content independent data compression system according to one embodiment of the present invention;

FIGS. 3a and 3b comprise a flow diagram of a data compression method according to one aspect of the present invention, which illustrates the operation of the data compression system of FIG. 2;

FIG. 4 is a block diagram of a content independent data compression system according to another embodiment of the present invention having an enhanced metric for selecting an optimal encoding technique;

FIGS. 5a and 5b comprise a flow diagram of a data compression method according to another aspect of the present invention, which illustrates the operation of the data compression system of FIG. 4;

FIG. 6 is a block diagram of a content independent data compression system according to another embodiment of the present invention having an a priori specified timer that provides real-time or pseudo real-time of output data;

FIGS. 7a and 7b comprise a flow diagram of a data compression method according to another aspect of the present invention, which illustrates the operation of the data compression system of FIG. 6;

FIG. 8 is a block diagram of a content independent data compression system according to another embodiment having an a priori specified timer that provides real-time or pseudo real-time of output data and an enhanced metric for selecting an optimal encoding technique;

FIG. 9 is a block diagram of a content independent data compression system according to another embodiment of the present invention having an encoding-architecture comprising a plurality of sets of serially cascaded encoders;

FIGS. 10a and 10b comprise a flow diagram of a data compression method according to another aspect of the present invention, which illustrates the operation of the data compression system of FIG. 9;

FIG. 11 is block diagram of a content independent data decompression system according to one embodiment of the present invention;

FIG. 12 is a flow diagram of a data decompression method according to one aspect of the present invention, which illustrates the operation of the data compression system of FIG. 11;

FIGS. 13a and 13b comprise a block diagram of a data compression system comprising content dependent and content independent data compression, according to an embodiment of the present invention;

FIGS. 14a-14d comprise a flow diagram of a data compression method using both content dependent and content independent data compression, according to one aspect of the present invention;

FIGS. 15a and 15b comprise a block diagram of a data compression system comprising content dependent and content independent data compression, according to another embodiment of the present invention;

FIGS. 16a-16d comprise a flow diagram of a data compression method using both content dependent and content independent data compression, according to another aspect of the present invention;

FIGS. 17a and 17b comprise a block diagram of a data compression system comprising content dependent and content independent data compression, according to another embodiment of the present invention; and

FIGS. 18a-18d comprise a flow diagram of a data compression method using both content dependent and content independent data compression, according to another aspect of the present invention.

 DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention is directed to systems and methods for providing data compression and decompression using content independent and content dependent data compression and decompression. In the following description, it is to be understood that system elements having equivalent or similar functionality are designated with the same reference numerals in the Figures. It is to be further understood that the present invention may be implemented in various forms of hardware, software, firmware, or a combination thereof. In particular, the system modules described herein are preferably implemented in software as an application program that is executable by, e.g., a general purpose computer or any machine or device having any suitable and preferred microprocessor architecture. Preferably, the present invention is implemented on a computer platform including hardware such as one or more central processing units (CPU), a random access memory (RAM), and input/output (I/O) interface(s). The computer platform also includes an operating system and microinstruction code. The various-processes and functions described herein may be either part of the microinstruction code or application programs which are executed via the operating system. In addition, various other peripheral devices may be connected to the computer platform such as an additional data storage device and a printing device.

It is to be further understood that, because some of the constituent system components described herein are preferably implemented as software modules, the actual system connections shown in the Figures may differ depending upon the manner in which the systems are programmed. It is to be appreciated that special purpose microprocessors may be employed to implement the present invention. Given the teachings herein, one of ordinary skill in the related art will be able to contemplate these and similar implementations or configurations of the present invention.

Referring now to FIG. 2 a block diagram illustrates a content independent data compression system according to one embodiment of the present invention. The data compression system includes a counter module 10 that receives as input an uncompressed or compressed data stream. It is to be understood that the system processes the input data stream in data blocks that may range in size from individual bits through complete files or collections of multiple files. Additionally, the data block size may be fixed or variable. The counter module 10 counts the size of each input data block (i.e., the data block size is counted in bits, bytes, words, any convenient data multiple or metric, or any combination thereof).

An input data buffer 20, operatively connected to the counter module 10, may be provided for buffering the input data stream in order to output an uncompressed data stream in the event that, as discussed in further detail below, every encoder fails to achieve a level of compression that exceeds an a priori specified minimum compression ratio threshold. It is to be understood that the input data buffer 20 is not required for implementing the present invention.

An encoder module 30 is operatively connected to the buffer 20 and comprises a set of encoders E1, E2, E3 . . . En. The encoder set E1, E2, E3 . . . En may include any number “n” of those lossless encoding techniques currently well known within the art such as run length, Huffman, Lempel-Ziv Dictionary Compression, arithmetic coding, data compaction, and data null suppression. It is to be understood that the encoding techniques are selected based upon their ability to effectively encode different types of input data. It is to be appreciated that a full complement of encoders are preferably selected to provide a broad coverage of existing and future data types.

The encoder module 30 successively receives as input each of the buffered input data blocks (or unbuffered input data blocks from the counter module 10). Data compression is performed by the encoder module 30 wherein each of the encoders E1 . . . En processes a given input data block and outputs a corresponding set of encoded data blocks. It is to be appreciated that the system affords a user the option to enable/disable any one or more of the encoders E1 . . . En prior to operation. As is understood by those skilled in the art, such feature allows the user to tailor the operation of the data compression system for specific applications. It is to be further appreciated that the is encoding process may be performed either in parallel or sequentially. In particular, the encoders E1 through En of encoder module 30 may operate in parallel (i.e., simultaneously processing a given input data block by utilizing task multiplexing on a single central processor, via dedicated hardware, by executing on a plurality of processor or dedicated hardware systems, or any combination thereof). In addition, encoders E1 through En may operate sequentially on a given unbuffered or buffered input data block. This process is intended to eliminate the complexity and additional processing overhead associated with multiplexing concurrent encoding techniques on a single central processor and/or dedicated hardware, set of central processors and/or dedicated hardware, or any achievable combination. It is to be further appreciated that encoders of the identical type may be applied in parallel to enhance encoding speed. For instance, encoder E1 may comprise two parallel Huffman encoders for parallel processing of an input data block.

A buffer/counter module 40 is operatively connected to the encoding module 30 for buffering and counting the size of each of the encoded data blocks output from encoder module 30. Specifically, the buffer/counter 30 comprises a-plurality of buffer/counters BC1, BC2, BC3 . . . BCn, each operatively associated with a corresponding one of the encoders E1 . . . En. A compression ratio module 50, operatively connected to the output buffer/counter 40, determines the compression ratio obtained for each of the enabled encoders E1 . . . En by taking the ratio of the size of the input data block to the size of the output data block stored in the corresponding buffer/counters BC1 . . . BCn. In addition, the compression ratio module 50 compares each compression ratio with an a priori-specified compression ratio threshold limit to determine if at least one of the encoded data blocks output from the enabled encoders E1 . . . En achieves a compression that exceeds an a priori-specified threshold. As is understood by those skilled in the art, the threshold limit may be specified as any value inclusive of data expansion, no data compression or expansion, or any arbitrarily desired compression limit. A description module 60, operatively coupled to the compression ratio module 50, appends a corresponding compression type descriptor to each encoded data block which is selected for-output so as to indicate the type of compression format of the encoded data block.

The operation of the data compression system of FIG. 2 will now be discussed in is further detail with reference to the flow diagram of FIGS. 3a and 3b. A data stream comprising one or more data blocks is input into the data compression system and the first data block in the stream is received (step 300). As stated above, data compression is performed on a per data block basis. Accordingly, the first input data block in the input data stream is input into the counter module 10 that counts the size of the data block (step 302). The data block is then stored in the buffer 20 (step 304). The data block is then sent to the encoder module 30 and compressed by each (enabled) encoder E1 . . . En (step 306). Upon completion of the encoding of the input data block, an encoded data block is output from each (enabled) encoder E1 . . . En and maintained in a corresponding buffer (step 308), and the encoded data block size is counted (step 310).

Next, a compression ratio is calculated for each encoded data block by taking the ratio of the size of the input data block (as determined by the input counter 10) to the size of each encoded data block output from the enabled encoders (step 312). Each compression ratio is then compared with an a priori-specified compression ratio threshold (step 314). It is to be understood that the threshold limit may be specified as any value inclusive of data expansion, no data compression or expansion, or any arbitrarily desired compression limit. It is to be further understood that notwithstanding that the current limit for lossless data compression is the entropy limit (the present definition of information content) for the data, the present invention does not preclude the use of future developments in lossless data compression that may increase lossless data compression ratios beyond what is currently known within the art.

After the compression ratios are compared with the threshold, a determination is made as to whether the compression ratio of at least one of the encoded data blocks exceeds the threshold limit (step 316). If there are no encoded data blocks having a compression ratio that exceeds the compression ratio threshold limit (negative determination in step 316), then the original unencoded input data block is selected for output and a null data compression type descriptor is appended thereto (step 318). A null data compression type descriptor is defined as any recognizable data token or descriptor that indicates no data encoding has been applied to the input data block. Accordingly, the unencoded input data block with its corresponding null data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 320).

On the other hand, if one or more of the encoded data blocks possess a compression ratio greater than the compression ratio threshold limit (affirmative result in step 316), then the encoded data block having the greatest compression ratio is selected (step 322). An appropriate data compression type descriptor is then appended (step 324). A data compression type descriptor is defined as any recognizable data token or descriptor that indicates which data encoding technique has been applied to the data. It is to be understood that, since encoders of the identical type may be applied in parallel to enhance encoding speed (as discussed-above), the data compression type descriptor identifies the corresponding encoding technique applied to the encoded data block, not necessarily the specific encoder. The encoded data block having the greatest compression ratio along with its corresponding data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 326).

After the encoded data block or the unencoded data input data block is output (steps 326 and 320), a determination is made as to whether the input data stream contains additional data blocks to be processed (step 328). If the input data stream includes additional data blocks (affirmative result in step 328), the next successive data block is received (step 330), its block size is counted (return to step 302) and the data compression process in repeated. This process is iterated for each data block in the input data stream. Once the final input data block is processed (negative result in step 328), data compression of the input data stream is finished (step 322).

Since a multitude of data types may be present within a given input data block, it is often difficult and/or impractical to predict the level of compression that will be achieved by-a specific encoder. Consequently, by processing the input data blocks with a plurality of encoding techniques and comparing the compression results, content free data compression is advantageously achieved. It is to be appreciated that this approach is scalable through future generations of processors, dedicated hardware, and software. As processing capacity increases and costs reduce, the benefits provided by the present invention will continue to increase. It should again be noted that the present invention may employ any lossless data encoding technique.

Referring now to FIG. 4, a block diagram illustrates a content independent data compression system according to another embodiment of the present invention. The data compression system depicted in FIG. 4 is similar to the data compression system of FIG. 2 except that the embodiment of FIG. 4 includes an enhanced metric functionality for selecting an optimal encoding technique. In particular, each of the encoders E1 . . . En in the encoder module 30 is tagged with a corresponding one of user-selected encoder desirability factors 70. Encoder desirability is defined as an a priori user specified factor that takes into account any number of user considerations including, but not limited to, compatibility of the encoded data with existing standards, data error robustness, or any other aggregation of factors that the user wishes to consider for a particular application. Each encoded data block output from the encoder module 30 has a corresponding desirability factor appended thereto. A figure of merit module 80, operatively coupled to the compression ratio module 50 and the descriptor module 60, is provided for calculating a figure of merit for each of the encoded data blocks which possess a compression ratio greater than the compression ratio threshold limit. The figure of merit for each encoded data block is comprised of a weighted average of the a priori user specified threshold and the corresponding encoder desirability factor. As discussed below in further detail with reference to FIGS. 5a and 5b, the figure of merit substitutes the a priori user compression threshold limit for selecting and outputting encoded data blocks.

The operation of the data compression system of FIG. 4 will now be discussed in further detail with reference to the flow diagram of FIGS. 5a and 5b. A data stream comprising one or more data blocks is input into the data compression system and the first data block in the stream is received (step 500). The size of the first data block is then determined by the counter module 10 (step 502). The data block is then stored in the buffer 20 (step 504). The data block is then sent to the encoder module 30 and compressed by each (enabled) encoder in the encoder set E1 . . . En (step 506). Each encoded data block processed in the encoder module 30 is tagged with an encoder desirability factor that corresponds the particular encoding technique applied to the encoded data block (step 508). Upon completion of the encoding of the input data block, an encoded data block with its corresponding desirability factor is output from each (enabled) encoder E1 . . . En and maintained in a corresponding buffer (step 510), and the encoded data block size is counted (step 512).

Next, a compression ratio obtained by each enabled encoder is calculated by taking the ratio of the size of the input data block (as determined by the input counter 10) to the size of the encoded data block output from each enabled encoder (step 514). Each compression ratio is then compared with an a priori-specified compression ratio threshold (step 516). A determination is made as to whether the compression ratio of at least one of the encoded data blocks exceeds the threshold limit (step 518). If there are no encoded data blocks having a compression ratio that exceeds the compression ratio threshold limit (negative determination in step 518), then the original unencoded input data block is selected for output and a null data compression type descriptor (as discussed above) is appended thereto (step 520). Accordingly, the original unencoded input data block with its corresponding null data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 522).

On the other hand, if one or more of the encoded data blocks possess a compression ratio greater than the compression ratio threshold limit (affirmative result in step 518), then a figure of merit is calculated for each encoded data block having a compression ratio which exceeds the compression ratio threshold limit (step 524). Again, the figure of merit for a given encoded data block is comprised of a weighted average of the a priori user specified threshold and the corresponding encoder desirability factor associated with the encoded data block. Next, the encoded data block having the greatest figure of merit is selected for output (step 526). An appropriate data compression type descriptor is then appended (step 528) to indicate the data encoding technique applied to the encoded data block. The encoded data block (which has the greatest figure of merit) along with its corresponding data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 530).

After the encoded data block or the unencoded input data block is output (steps 530 and 522), a determination is made as to whether the input data stream contains additional data blocks to be processed (step 532). If the input data stream includes additional data blocks (affirmative result in step 532), then the next successive data block is received (step 534), its block size is counted (return to step 502) and the data compression process is iterated for each successive data block in the input data stream. Once the final input data block is processed (negative result in step 532), data compression of the input data stream is finished (step 536).

Referring now to FIG. 6, a block diagram illustrates a data compression system according to another embodiment of the present invention. The data compression system depicted in FIG. 6 is similar to the data compression system discussed in detail above with reference to FIG. 2 except that the embodiment of FIG. 6 includes an a priori specified timer that provides real-time or pseudo real-time output data. In particular, an interval timer 90, operatively coupled to the encoder module 30, is preloaded with a user specified time value. The role of the interval timer (as will be explained in greater detail below with reference to FIGS. 7a and 7b) is to limit the processing time for each input data block processed by the encoder module 30 so as to ensure that the real-time, pseudo real-time, or other time critical nature of the data compression processes is preserved.

The operation of the data compression system of FIG. 6 will now be discussed in further detail with reference to the flow diagram of FIGS. 7a and 7b. A data stream comprising one or more data blocks is input into the data compression system and the first data block in the data stream is received (step 700), and its size is-determined by the counter module 10 (step 702). The data block is then stored in buffer 20 (step 704).

Next, concurrent with the completion of the receipt and counting of the first data block, the interval timer 90 is initialized (step 706) and starts counting towards a user-specified time limit. The input data block is then sent to the encoder module 30 wherein data compression of the data block by each (enabled) encoder E1 . . . En commences (step 708). Next, a determination is made as to whether the user specified time expires before the completion of the encoding process (steps 710 and 712). If the encoding process is completed before or at the expiration of the timer, i.e., each encoder (E1 through En) completes its respective encoding process (negative result in step 710 and affirmative result in step 712), then an encoded data block is output from each (enabled) encoder E1 . . . En and maintained in a corresponding buffer (step 714).

On the other hand, if the timer expires (affirmative result in 710), the encoding process is halted (step 716). Then, encoded data blocks from only those enabled encoders E1 . . . En that have completed the encoding process are selected and maintained in buffers (step 718). It is to be appreciated that it is not necessary (or in some cases desirable) that some or all of the encoders-complete the encoding process before the interval timer expires. Specifically, due to encoder data dependency and natural variation, it is possible that certain encoders may not operate quickly enough and, therefore, do not comply with the timing constraints of the end use. Accordingly, the time limit ensures that the real-time or pseudo real-time nature of the data encoding is preserved.

After the encoded data blocks are buffered (step 714 or 718), the size of each encoded data block is counted (step 720). Next, a compression ratio is calculated for each encoded data block by taking the ratio of the size of the input data block (as determined by the input counter 10) to the size of the encoded data block output from each enabled encoder (step 722). Each compression ratio is then compared with an a priori-specified compression ratio threshold (step 724). A determination is made as to whether the compression ratio of at least one of the encoded data blocks exceeds the threshold limit (step 726). If there are no encoded data blocks having a compression ratio that exceeds the-compression ratio threshold limit (negative determination in step 726), then the original unencoded input data block is selected for output and a null data compression type descriptor is appended thereto (step 728). The original unencoded input data block with its corresponding null data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 730).

On the other hand, if one or more of the encoded data blocks possess a compression ratio greater than the compression ratio threshold limit (affirmative result in step 726), then the encoded data block having the greatest compression ratio is selected (step 732). An appropriate data compression type descriptor is then appended (step 734). The encoded data block having the greatest compression ratio along with its corresponding data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 736).

After the encoded data block or the unencoded input data block is output (steps 730 or 736), a determination is made as to whether the input data stream contains additional data blocks to be processed (step 738). If the input data stream includes additional data blocks (affirmative result in step 738), the next successive data block is received (step 740), its block size is counted (return to step 702) and the data compression process in repeated. This process is iterated for each data block in the input data stream, with each data block being processed within the user-specified time limit as discussed above. Once the final input data block is processed (negative result in step 738), data compression of the input data stream is complete (step 742).

Referring now to FIG. 8, a block diagram illustrates a content independent data compression system according to another embodiment of the present system. The data compression system of FIG. 8 incorporates all of the features discussed above in connection with the system embodiments of FIGS. 2, 4, and 6. For example, the system of FIG. 8 incorporates both the a priori specified timer for providing real-time or pseudo real-time of output data, as well as the enhanced metric for selecting an optimal encoding technique. Based on the foregoing discussion, the operation of the system of FIG. 8 is understood by those skilled in the art.

Referring now to FIG. 9, a block diagram illustrates a data compression system according to a preferred embodiment of the present invention. The system of FIG. 9 contains many of the features of the previous embodiments discussed above. However, this embodiment advantageously includes a cascaded encoder module 30c having an encoding architecture comprising a plurality of sets of serially-cascaded encoders Em,n, where “m” refers to the encoding path (i.e., the encoder set) and where “n” refers to the number of encoders in the respective path. It is to be understood that each set of serially cascaded encoders can include any number of disparate and/or similar encoders (i.e., n can be any value for a given path m).

The system of FIG. 9 also includes a output buffer module 40c which comprises a plurality of buffer/counters B/C m,n, each associated with a corresponding one of the encoders Em,n. In this embodiment, an input data block is sequentially applied to successive encoders (encoder stages) in the encoder path so as to increase the data compression ratio. For example, the output data block from a first encoder E1,1, is buffered and counted in B/C1,1, for subsequent processing by a second encoder E1,2. Advantageously, these parallel sets of sequential encoders are applied to the input data stream to effect content free lossless data compression. This embodiment provides for multi-stage sequential encoding of data with the maximum number of encoding steps subject to the available real-time, pseudo real-time, or other timing constraints.

As with each previously discussed embodiment, the encoders Em,n may include those lossless encoding techniques currently well known within the art, including: run length, Huffman, Lempel-Ziv Dictionary Compression, arithmetic coding, data compaction, and data null suppression. Encoding techniques are selected based upon their ability to effectively encode different types of input data. A full complement of encoders provides for broad coverage of existing and future data types. The input data blocks may be applied simultaneously to the encoder paths (i.e., the encoder paths may operate in parallel, utilizing task multiplexing on a single central processor, or via dedicated hardware, or by executing on a plurality of processor or dedicated hardware systems, or any combination thereof). In addition, an input data block may be sequentially applied to the encoder paths. Moreover, each serially cascaded encoder path may comprise a fixed (predetermined) sequence of encoders or a random sequence of encoders. Advantageously, by simultaneously or sequentially processing input data blocks via a plurality of sets of serially cascaded encoders, content free data compression is achieved.

The operation of the data compression system of FIG. 9 will now be discussed in further detail with reference to the flow diagram of FIGS. 10a and 10b. A data stream comprising one or more data blocks is input into the data compression system and the first data block in the data stream is received (step 100), and its size is determined by the counter module 10 (step 102). The data block is then stored in buffer 20 (step 104).

Next, concurrent with the completion of the receipt and counting of the first data block, the interval timer 90 is initialized (step 106) and starts counting towards a user-specified time limit. The input data block is then sent to the cascade encoder module 30C wherein the input data block is applied to the first encoder (i.e., first encoding stage) in each of the cascaded encoder paths E1,1 . . . Em,1 (step 108). Next, a determination is made as to whether the user specified time expires before the completion of the first stage encoding process (steps 110 and 112). If the first stage encoding process is completed before the expiration of the timer, i.e., each encoder (E1,1 . . . Em,1) completes its respective encoding process (negative result in step 110 and affirmative result in step 112), then an encoded data block is output from each encoder E1,1 . . . Em,1 and maintained in a corresponding buffer (step 114). Then for each cascade encoder path, the output of the completed encoding stage is applied to the next successive encoding stage in the cascade path (step 116). This process (steps 110, 112, 114, and 116) is repeated until the earlier of the timer expiration (affirmative result in step 110) or the completion of encoding by each encoder stage in the serially cascaded paths, at which time the encoding process is halted (step 118).

Then, for each cascade encoder path, the buffered encoded data block output by the last encoder stage that completes the encoding process before the expiration of the timer is selected for further processing (step 120). Advantageously, the interim stages of the multi-stage data encoding process are preserved. For example, the results of encoder E1,1 are preserved even after encoder E1,2 begins encoding the output of encoder E1,1 . If the-interval timer expires after encoder E1,1 completes its respective encoding process but before encoder E1,2 completes its respective encoding process, the encoded data block from encoder E1,1 is complete and is utilized for calculating the compression-ratio for the corresponding encoder path. The incomplete encoded data block from encoder E1,2 is either discarded or ignored.

It is to be appreciated that it is not necessary (or in some cases desirable) that some or all of the encoders in the cascade encoder paths complete the encoding process before the interval timer expires. Specifically, due to encoder data dependency, natural variation and the sequential application of the cascaded encoders, it is possible that certain encoders may not operate quickly enough and therefore do not comply with the timing constraints of the end use. Accordingly, the time limit ensures that the real-time or pseudo real-time nature of the data encoding is preserved.

After the encoded data blocks are selected (step 120), the size of each encoded data block is counted (step 122). Next, a compression ratio is calculated for each encoded data block by taking the ratio of the size of the input data block (as determined by the input counter 10) to the size of the encoded data block output from each encoder (step 124). Each compression ratio is then compared with an a priori-specified compression ratio threshold (step 126). A determination is made as to whether the compression ratio of at least one of the encoded data blocks exceeds the threshold limit (step 128). If there are no encoded data blocks having a compression ratio that exceeds the compression ratio threshold limit (negative determination in step 128), then the original unencoded input data block is selected for output and a null data compression type descriptor is appended thereto (step 130). The original unencoded data block and its corresponding null data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 132).

On the other hand, if one or more of the encoded data blocks possess a compression ratio greater than the compression ratio threshold limit (affirmative result in step 128), then a figure of merit is calculated for each encoded data block having a compression ratio which exceeds the compression ratio threshold limit (step 134). Again, the figure of merit for a given encoded data block is comprised of a weighted average of the a priori user specified threshold and the corresponding encoder desirability factor associated with the encoded data block. Next, the encoded data block having the greatest figure of merit is selected (step 136). An appropriate data compression type descriptor is then appended (step 138) to indicate the data encoding technique applied to the encoded data block. For instance, the data type compression descriptor can indicate that the encoded data block was processed by either a single encoding type, a plurality of sequential encoding types, and a plurality of random encoding types. The encoded data block (which has the greatest figure of merit) along with its corresponding data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 140).

After the unencoded data block or the encoded data input data block is output (steps 132 and 140), a determination is made as to whether the input data stream contains additional data blocks to be processed (step 142). If the input data stream includes additional data blocks (affirmative result in step 142), then the next successive data block is received (step 144), its block size is counted (return to step 102) and the data compression process is iterated for each successive data block in the input data stream. Once the final input data block is processed (negative result in step 142), data compression-of the input data stream is finished (step 146).

Referring now to FIG. 11, a block diagram illustrates a data decompression system according to one embodiment of the present invention. The data decompression system preferably includes an input buffer 1100 that receives as input an uncompressed or compressed data stream comprising one or more data blocks. The data blocks may range in size from individual bits through complete files or collections of multiple files. Additionally, the data block size may be fixed or variable. The input data buffer 1100 is preferably included (not required) to provide storage of input data for various hardware implementations. A descriptor extraction module 1102 receives the buffered (or unbuffered) input data block and then parses, lexically, syntactically, or otherwise analyzes the input data block using methods known by those skilled in the art to extract the data compression type descriptor associated with the data block. The data compression type descriptor may possess values corresponding to null (no encoding applied), a single applied encoding technique, or multiple encoding techniques applied in a specific or random order (in accordance with the data compression system embodiments and methods discussed above).

A decoder module 1104 includes a plurality of decoders D1 . . . Dn for decoding the input data block using a-decoder, set of decoders, or a sequential set of decoders corresponding to the extracted compression type descriptor. The decoders D1 . . . Dn may include those lossless encoding techniques currently well known within the art, including: run length, Huffman, Lempel-Ziv Dictionary Compression, arithmetic coding, data compaction, and data null suppression. Decoding techniques are selected based upon their ability to effectively decode the various different types of encoded input data generated by the data compression systems described above or originating from any other desired source. As with the data compression systems discussed above, the decoder module 1104 may include multiple decoders of the same type applied in parallel so as to reduce the data decoding time.

The data decompression system also includes an output data buffer 1106 for buffering the decoded data block output from the decoder module 1104.

The operation of the data decompression system of FIG. 11 will be discussed in further detail with reference to the flow diagram of FIG. 12. A data stream comprising one or more data blocks of compressed or uncompressed data is input into the data decompression system and the first data block in the stream is received (step 1200) and maintained in the buffer (step 1202). As with the data compression systems discussed above, data decompression is performed on a per data block basis. The data compression type descriptor is then extracted from the input data block (step 1204). A determination is then made as to whether the data compression type descriptor is null (step 1206). If the data compression type descriptor is determined to be null (affirmative result in step 1206), then no decoding is applied to the input data block and the original undecoded data block is output (or maintained in the output buffer) (step 1208).

On the other hand, if the data compression type descriptor is determined to be any value other than null (negative result in step 1206), the corresponding decoder or decoders are then selected (step 1210) from the available set of decoders D1 . . . Dn in the decoding module 1104. It is to be understood that the data compression type descriptor may mandate the application of: a single specific decoder, an ordered sequence of specific decoders, a random order of specific decoders, a class or family of decoders, a mandatory or optional application of parallel decoders, or any combination or permutation thereof. The input data block is then decoded using the selected decoders (step 1212), and output (or maintained in the output buffer 1106) for subsequent data processing, storage, or transmittal (step 1214). A determination is then made as to whether the input data stream contains additional data blocks to be processed (step 1216). If the input data stream includes additional data blocks (affirmative result in step 1216), the next successive data block is received (step 1220), and buffered (return to step 1202). Thereafter, the data decompression process is iterated for each data block in the input data stream. Once the final input data block is processed (negative result in step 1216), data decompression of the input data stream is finished (step 1218).

In other embodiments of the present invention described below, data compression is achieved using a combination of content dependent data compression and content independent data compression. For example, FIGS. 13a and 13b are block diagrams illustrating a data compression system employing both content independent and content dependent data compression according to one embodiment of the present invention, wherein content independent data compression is applied to a data block when the content of the data block cannot be identified or is not associable with a specific data compression algorithm. The data compression system comprises a counter module 10 that receives as input an uncompressed or compressed data stream. It is to be understood that the system processes the input data stream in data blocks that may range in size from individual bits through complete files or collections of multiple files. Additionally, the data block size may be fixed or variable. The counter module 10 counts the size of each input data block (i.e., the data block size is counted in bits, bytes, words, any convenient data multiple or metric, or any combination thereof).

An input data buffer 20, operatively connected to the counter module 10, may be provided for buffering the input data stream in order to output an uncompressed data stream in the event that, as discussed in further detail below, every encoder fails to achieve a level of compression that exceeds a priori specified content independent or content dependent minimum compression ratio thresholds. It is to be understood that the input data buffer 20 is not required for implementing the present invention.

A content dependent data recognition module 1300 analyzes the incoming data stream to recognize data types, data structures, data block formats, file substructures, file types, and/or any other parameters that may be indicative of either the data type/content of a given data block or the appropriate data compression algorithm or algorithms (in serial or in parallel) to be applied. Optionally, a data file recognition list(s) or algorithm(s) 1310 module may be employed to hold and/or determine associations between recognized data parameters and appropriate algorithms. Each data block that is recognized by the content data compression module 1300 is routed to a content dependent encoder module 1320, if not the data is routed to the content independent encoder module 30.

A content dependent encoder module 1320 is operatively connected to the content dependent data recognition module 1300 and comprises a set of encoders D1, D2, D3 . . . Dm. The encoder set D1, D2, D3 . . . Dm may include any number “n” of those lossless or lossy encoding techniques currently well known within the art such as MPEG4, various voice codecs, MPEG3, AC3, AAC, as well as lossless algorithms such as run length, Huffman, Lempel-Ziv Dictionary Compression, arithmetic coding, data compaction, and data null suppression. It is to be understood that the encoding techniques are selected based upon their ability to effectively encode different types of input data. It is to be appreciated that a full complement of encoders and or codecs are preferably selected to provide a broad coverage of existing and future data types.

The content independent encoder module 30, which is operatively connected to the content dependent data recognition module 1300, comprises a set of encoders E1, E2, E3 . . . En. The encoder set E1, E2, E3 . . . En may include any number “n” of those lossless encoding techniques currently well known within the art such as run length, Huffman, Lempel-Ziv Dictionary Compression, arithmetic coding, data compaction, and data null suppression. Again, it is to be understood that the encoding techniques are selected based upon their ability to effectively encode different types of input data. It is to be appreciated that a full complement of encoders are preferably selected to provide a broad coverage of existing and future data types.

The encoder modules (content dependent 1320 and content independent 30) selectively receive the buffered input data blocks (or unbuffered input data blocks from the counter module 10) from module 1300 based on the results of recognition. Data compression is performed by the respective encoder modules wherein some or all of the encoders D1 . . . Dm or E1 . . . En processes a given input data block and outputs a corresponding set of encoded data blocks. It is to be appreciated that the system affords a user the option to enable/disable any one or more of the encoders D1 . . . Dm and E1 . . . En prior to operation. As is understood by those skilled in the art, such feature allows the user to tailor the operation of the data compression system for specific applications. It is to be further appreciated that the encoding process may be performed either in parallel or sequentially. In particular, the encoder set D1 through Dm of encoder module 1320 and/or the encoder set E1 through En of encoder module 30 may operate in parallel (i.e., simultaneously processing a given input data block by utilizing task multiplexing on a single central processor, via dedicated hardware, by executing on a plurality of processor or dedicated hardware systems, or any combination thereof). In addition, encoders D1 through Dm and E1 through En may operate sequentially on a given unbuffered or buffered input data block. This process is intended to eliminate the complexity and additional processing overhead associated with multiplexing concurrent encoding techniques on a single central processor and/or dedicated hardware, set of central processors and/or dedicated hardware, or any achievable combination. It is to be further appreciated that encoders of the identical type may be applied in parallel to enhance encoding speed. For instance, encoder E1 may comprise two parallel Huffman encoders for parallel processing of an input data block. It should be further noted that one or more algorithms may be implemented in dedicated hardware such as an MPEG4 or MP3 encoding integrated circuit.

Buffer/counter modules 1330 and 40 are operatively connected to their respective encoding modules 1320 and 30, for buffering and counting the size of each of the encoded data blocks output from the respective encoder modules. Specifically, the content dependent buffer/counter 1330 comprises a plurality of buffer/counters BCD1, BCD2, BCD3 . . . BCDm, each operatively associated with a corresponding one of the encoders D1 . . . Dm. Similarly the content independent buffer/counters BCE1, BCE2, BCE3 . . . BCEn, each operatively associated with a corresponding one of the encoders E1 . . . En. A compression ratio module 1340, operatively connected to the content dependent output buffer/counters 1330 and content independent buffer/counters 40 determines the compression ratio obtained for each of the enabled encoders D1 . . . Dm and or E1 . . . En by taking the ratio of the size of the input data block to the size of the output data block stored in the corresponding buffer/counters BCD1, BCD2, BCD3 . . . BCDm and or BCE 1, BCE2, BCE3 . . . BCEn. In addition, the compression ratio module 1340 compares each compression ratio with an a priori-specified compression ratio threshold limit to determine if at least one of the encoded data blocks output from the enabled encoders BCD1, BCD2, BCD3 . . . BCDm and or BCE1, BCE2, BCE3 . . . BCEn achieves a compression that meets an a prior-i-specified threshold. As is understood by those skilled in the art, the threshold limit maybe specified as any value inclusive of data expansion, no data compression or expansion, or any arbitrarily desired compression limit. It should be noted that different threshold values may be applied to content dependent and content independent encoded data. Further these thresholds may be adaptively modified based upon enabled encoders in either or both the content dependent or content independent encoder sets, along with any associated parameters. A compression type description module 1350, operatively coupled to the compression ratio module 1340, appends a corresponding compression type descriptor to each encoded data block which is selected for output so as to indicate the type of compression format of the encoded data block.

A mode of operation of the data compression system of FIGS. 13a and 13b will now be discussed with reference to the flow diagrams of FIGS. 14a-14d, which illustrates a method for performing data compression using a combination of content dependent and content independent data compression. In general, content independent data compression is applied to a given data block when the content of a data block cannot be identified or is not associated with a specific data compression algorithm. More specifically, referring to FIG. 14a, a data stream comprising one or more data blocks is input into the data compression system and the first data block in the stream is received (step 1400). As stated above, data compression is performed on a per data block basis. As previously stated a data block may represent any quantity of data from a single bit through a multiplicity of files or packets and may vary from block to block. Accordingly, the first input data block in the input data stream is input into the counter module 10 that counts the size of the data block (step 1402). The data block is then stored in the buffer 20 (step 1404). The data block is then analyzed on a per block or multi-block basis by the content dependent data recognition module 1300 (step 1406). If the data stream content is not recognized utilizing the recognition list(s) or algorithms(s) module 1310 (step 1408) the data is routed to the content independent encoder module 30 and compressed by each (enabled) encoder E1 . . . En (step 1410). Upon completion of the encoding of the input data block, an encoded data block is output from each (enabled) encoder E1 . . . En and maintained in a corresponding buffer (step 1412), and the encoded data block size is counted (step 1414).

Next, a compression ratio is calculated for each encoded data block by taking the ratio of the size of the input data block (as determined by the input counter 10 to the size of each encoded data block output from the enabled encoders (step 1416). Each compression ratio is then compared with an a priori-specified compression ratio threshold (step 1418). It is to be understood that the threshold limit may be specified as any value inclusive of data expansion, no data compression or expansion, or any arbitrarily desired compression limit. It is to be further understood that notwithstanding that the current limit for lossless data compression is the entropy limit (the present definition of information content) for the data, the present invention does not preclude the use of future developments in lossless data compression that may increase lossless data compression ratios beyond what is currently known within the art. Additionally the content independent data compression threshold may be different from the content dependent threshold and either may be modified by the specific enabled encoders.

After the compression ratios are compared with the threshold, a determination is made as to whether the compression ratio of at least one of the encoded data blocks exceeds the threshold limit (step 1420). If there are no encoded data blocks having a compression ratio that exceeds the compression ratio threshold limit (negative determination in step 1420), then the original unencoded input data block is selected for output and a null data compression type descriptor is appended thereto (step 1434). A null data compression type descriptor is defined as any recognizable data token or descriptor that indicates no data encoding has been applied to the input data block. Accordingly, the unencoded input data block with its corresponding null data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 1436).

On the other hand, if one or more of the encoded data blocks possess a compression ratio greater than the compression ratio threshold limit (affirmative result in step 1420), then the encoded data block having the greatest compression ratio is selected (step 1422). An appropriate data compression type descriptor is then appended (step 1424). A data compression type descriptor is defined as any recognizable data token or descriptor that indicates which data encoding technique has been applied to the data. It is to be understood that, since encoders of the identical type may be applied in parallel to enhance encoding speed (as discussed above), the data compression type descriptor identifies the corresponding encoding technique applied to the encoded data block, not necessarily the specific encoder. The encoded data block having the greatest compression ratio along with its corresponding data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 1426).

As previously stated the data block stored in the buffer 20 (step 1404) is analyzed on a per block or multi-block basis by the content dependent data recognition module 1300 (step 1406). If the data stream content is recognized utilizing the recognition list(s) or algorithms(s) module 1310 (step 1434) the appropriate content dependent algorithms are enabled and initialized (step 1436), and the data is routed to the content dependent encoder module 1320 and compressed by each (enabled) encoder D1 . . . Dm (step 1438). Upon completion of the encoding of the input data block, an encoded data block is output from each (enabled) encoder D1 . . . Dm and maintained in a corresponding buffer (step 1440), and the encoded data block size is counted (step 1442).

Next, a compression ratio is calculated for each encoded data block by taking the ratio of the size of the input data block (as determined by the input counter 10 to the size of each encoded data block output from the enabled encoders (step 1444). Each compression ratio is then compared with an a priori-specified compression ratio threshold (step 1448). It is to be understood that the threshold limit may be specified as any value inclusive of data expansion, no data compression or expansion, or any arbitrarily desired compression limit. It is to be further understood that many of these algorithms may be lossy, and as such the limits may be subject to or modified by an end target storage, listening, or viewing device. Further notwithstanding that the current limit for lossless data compression is the entropy limit (the present definition of information content) for the data, the present invention does not preclude the use of future developments in lossless data compression that may increase lossless data compression ratios beyond what is currently known within the art. Additionally the content independent data compression threshold may be different from the content dependent threshold and either may be modified by the specific enabled encoders.

After the compression ratios are compared with the threshold, a determination is made as to whether the compression ratio of at least one of the encoded data blocks exceeds the threshold limit (step 1420). If there are no encoded data-blocks having a compression ratio that exceeds the compression ratio threshold limit (negative determination in step 1420), then the original unencoded input data block is selected for output and a null data compression type descriptor is appended thereto (step 1434). A null data compression type descriptor is defined as any recognizable data token or descriptor that indicates no data encoding has been applied to the input data block. Accordingly, the unencoded input data block with its corresponding null data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 1436).

On the other hand, if one or more of the encoded data blocks possess a compression ratio greater than the compression ratio threshold limit (affirmative result in step 1420), then the encoded data block having the greatest compression ratio is selected (step 1422). An appropriate data compression type descriptor is then appended (step 1424). A data compression type descriptor is defined as any recognizable data token or descriptor that indicates which data encoding technique has been applied to the data. It is to be understood that, since encoders of the identical type may be applied in parallel to enhance encoding speed (as discussed above), the data compression type descriptor identifies the corresponding encoding technique applied to the encoded data block, not necessarily the specific encoder. The encoded data block having the greatest compression ratio along with its corresponding data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 1426).

After the encoded data block or the unencoded data input data block is output (steps 1426 and 1436), a determination is made as to whether the input data stream contains additional data blocks to be processed (step 1428). If the input data stream includes additional data blocks (affirmative result in step 1428), the next successive data block is received (step 1432), its block size is counted (return to step 1402) and the data compression process in repeated. This process is iterated for each data block in the input data stream. Once the final input data block is processed (negative result in step 1428), data compression of the input data stream is finished (step 1430).

Since a multitude of data types may be present within a given input data block, it is often difficult and/or impractical to predict the level of compression that will be achieved by a specific encoder. Consequently, by processing the input data blocks with a plurality of encoding techniques and comparing the compression results, content free data compression is advantageously achieved. Further the encoding may be lossy or lossless dependent upon the input data types. Further if the data type is not recognized the default content independent lossless compression is applied. It is not a requirement that this process be deterministic—in fact a certain probability may be applied if occasional data loss is permitted. It is to be appreciated that this approach is scalable through future generations of processors, dedicated hardware, and software. As processing capacity increases and costs reduce, the benefits provided by the present invention will continue to increase. It should again be noted that the present invention may employ any lossless data encoding technique.

FIGS. 15a and 15b are block diagrams illustrating a data compression system employing both content independent and content dependent data compression according to another embodiment of the present invention. The system in FIGS. 15a and 15b is similar in operation to the system of FIGS. 13a and 13b in that content independent data compression is applied to a data block when the content of the data block cannot be identified or is not associable with a specific data compression algorithm. The system of FIGS. 15a and 15b additionally performs content independent data compression on a data block when the compression ratio obtained for the data block using the content dependent data compression does not meet a specified threshold.

A mode of operation of the data compression system of FIGS. 15a and 15b will now be discussed with reference to the flow diagram of FIGS. 16a-16d, which illustrates a method for performing data compression using a combination of content dependent and content independent data compression. A data stream comprising one or more data blocks is input into the data compression system and the first data block in the stream is received (step 1600). As stated above, data compression is performed on a per data block basis. As previously stated a data block may represent any quantity of data from a single bit through a multiplicity of files or packets and may vary from block to block. Accordingly, the first input data block in the input data stream is input into the counter module 10 that counts the size of the data block (step 1602). The data block is then stored in the buffer 20 (step 1604). The data block is then analyzed on a per block or multi-block basis by the content dependent data recognition module 1300 (step 1606). If the data stream content is not recognized utilizing the recognition list(s) or algorithms(s) module 1310 (Step 1608) the data is routed to-the content independent encoder module 30 and compressed by each (enabled) encoder E1 . . . En (step 1610). Upon completion of the encoding of the input data block, an encoded data block is output from each (enabled) encoder E1 . . . En and maintained in a corresponding buffer (step 1612), and the encoded data block size is counted (step 1614).

Next, a compression ratio is calculated for each encoded data block by taking the ratio of the size of the input data block (as determined by the input counter 10 to the size of each encoded data block output from the enabled encoders (step 1616). Each compression ratio is then compared with an a priori-specified compression ratio threshold (step 1618). It is to be understood that the threshold limit may be specified as any value inclusive of data expansion, no data compression or expansion, or any arbitrarily desired compression limit. It is to be further understood that notwithstanding that the current limit for lossless data compression is the entropy limit (the present definition of information content) for the data, the present invention does not preclude the use of future developments in lossless data compression that may increase lossless data compression ratios-beyond what is currently known within the art. Additionally the content independent data compression threshold may be different from the content dependent threshold and either may be modified by the specific enabled encoders.

After the compression ratios are compared with the threshold, a determination is made as to whether the compression ratio of at least one of-the encoded data blocks exceeds the threshold limit (step 1620). If there are no encoded data blocks having a compression ratio that exceeds the compression ratio threshold limit (negative determination in step 1620), then the original unencoded input data block is selected for output and a null data compression type descriptor is appended thereto (step 1634). A null data compression type descriptor is defined as any recognizable data token or descriptor that indicates no data encoding has been applied to the input data block. Accordingly, the unencoded input data block with its corresponding null data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 1636).

On the other hand, if one or more of the encoded data blocks possess a compression ratio greater than the compression ratio threshold limit (affirmative result in step 1620), then the encoded data block having the greatest compression ratio is selected (step 1622). An appropriate data compression type descriptor is then appended (step 1624). A data compression type descriptor is defined as any recognizable data token or descriptor that indicates which data encoding technique has been applied to the data. It is to be understood that, since encoders of the identical type may be applied in parallel to enhance encoding speed (as discussed above), the data compression type descriptor identifies the corresponding encoding technique applied to the encoded data block, not necessarily the specific encoder. The encoded data block having the greatest compression ratio along with its corresponding data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 1626).

As previously stated the data block stored in the buffer 20 (step 1604) is analyzed on a per block or multi-block basis by the content dependent data recognition module 1300 (step 1606). If the data stream content is recognized utilizing the recognition list(s) or algorithms(s) module 1310 (step 1634) the appropriate content dependent algorithms are enabled and initialized (step 1636) and the data is routed to the content dependent encoder module 1620 and compressed by each (enabled) encoder D1 . . . Dm (step 1638). Upon completion of the encoding of the input data block, an encoded data block is output from each (enabled) encoder D1 . . . Dm and maintained in a corresponding buffer (step 1640), and the encoded data block size is counted (step 1642).

Next, a compression ratio is calculated for each encoded data block by taking the ratio of the size of the input data block (as determined by the input counter 10 to the size of each encoded data block output from the enabled encoders (step 1644). Each compression ratio is then compared with an a priori-specified compression ratio threshold (step 1648). It is to be understood that the threshold limit may be specified as any value inclusive of data expansion, no data compression or expansion, or any arbitrarily desired compression limit. It is to be further understood that many of these algorithms may be lossy, and as such the limits may be subject to or modified by an end target storage, listening, or viewing device. Further notwithstanding that the current limit for lossless data compression is the entropy limit (the present definition of information content) for the data, the present invention does not preclude the use of future developments in lossless data compression that may increase lossless data compression ratios beyond what is currently known within the art. Additionally the content independent data compression threshold may be different from the content dependent threshold and either may be modified by the specific enabled encoders.

After the compression ratios are compared with the threshold, a determination is made as to whether the compression ratio of at least one of the encoded data blocks exceeds the threshold limit (step 1648). If there are no encoded data blocks having a compression ratio that exceeds the compression ratio threshold limit (negative determination in step 1620), then the original unencoded input data block is routed to the content independent encoder module 30 and the process resumes with compression utilizing content independent encoders (step 1610).

After the encoded data block or the unencoded data input data block is output (steps 1626 and 1636), a determination is made as to whether the input data stream contains additional data blocks to be processed (step 1628). If the input data stream includes additional data blocks (affirmative result in step 1628), the next successive data block is received (step 1632), its block size is counted (return to step 1602) and the data compression process in repeated. This process is iterated for each data block in the input data stream. Once the final input data block is processed (negative result in step 1628), data compression of the input data stream is finished (step 1630).

FIGS. 17a and 17b are block diagrams illustrating a data compression system employing both content independent and content dependent data compression according to another embodiment of the present invention. The system in FIGS. 17a and 17b is similar in operation to the system of FIGS. 13a and 13b in that content independent data compression is applied to a data block when the content of the data block cannot be identified or is not associable with a specific data compression algorithm. The system of FIGS. 17a and 17b additionally uses a priori estimation algorithms or look-up tables to estimate the desirability of using content independent data compression encoders and/or content dependent-data compression encoders and selecting appropriate algorithms or subsets thereof based on such estimation.

More specifically, a content dependent data recognition and or estimation module 1700 is utilized to analyze the incoming data stream for recognition of data types, data strictures, data block formats, file substructures, file types, or any other parameters that may be indicative of the appropriate data compression algorithm or algorithms (in serial or in parallel) to be applied. Optionally, a data file recognition list(s) or algorithm(s) 1710 module may be employed to hold associations between recognized data parameters and appropriate algorithms. If the content data compression module recognizes a portion of the data, that portion is routed to the content dependent encoder module 1320, if not the data is routed to the content independent encoder module 30. It is to be appreciated that process of recognition (modules 1700 and 1710) is not limited to a deterministic recognition, but may further comprise a probabilistic estimation of which encoders to select for compression from the set of encoders of the content dependent module 1320 or the content independent module 30. For example, a method may be employed to compute statistics of a data block whereby a determination that the locality of repetition of characters in a data stream is determined is high can suggest a text document, which may be beneficially compressed with a lossless dictionary type algorithm. Further the statistics of repeated characters and relative frequencies may suggest a specific type of dictionary algorithm. Long strings will require a wide dictionary file while a wide diversity of strings may suggest a deep dictionary. Statistics may also be utilized in algorithms such as Huffman where various character statistics will dictate the choice of different Huffman compression tables. This technique is not limited to lossless algorithms but may be widely employed with lossy algorithms. Header information in frames for video files can imply a specific data resolution. The estimator then may select the appropriate lossy compression algorithm and compression parameters (amount of resolution desired). As shown in previous embodiments of the present invention, desirability of various algorithms and now associated resolutions with lossy type algorithms may also be applied in the estimation selection process.

A mode of operation of the data compression system of FIGS. 17a and 17b will now be discussed with reference to the flow diagrams of FIGS. 18a-18d. The method of FIGS. 18a-18d use a priori estimation algorithms or look-Lip tables to estimate the desirability or probability of using content independent data compression encoders or content dependent data compression encoders, and select appropriate or desirable algorithms or subsets thereof based on such estimates. A data stream comprising one or more data blocks is input into the data compression system and the first data block in the stream is received (step 1800). As stated above, data compression is performed on a per data block basis. As previously stated a data block may represent any quantity of data from a single bit through a multiplicity of files or packets and may vary from block to block. Accordingly, the first input data block in the input data stream is input into the counter module 10 that counts the size of the data block (step 1802). The data block is then stored in the buffer 20 (step 1804). The data block is then analyzed on a per block or multi-block basis by the content dependent/content independent data recognition module 1700 (step 1806). If the data stream content is not recognized utilizing the recognition list(s) or algorithms(s) module 1710 (step 1808) the data is to the content independent encoder module 30. An estimate of the best content independent encoders is performed (step 1850) and the appropriate encoders are enabled and initialized as applicable. The data is then compressed by each (enabled) encoder E1 . . . En (step 1810). Upon completion of the encoding of the input data block, an encoded data block is output from each (enabled) encoder E1 . . . En and maintained in a corresponding buffer (step 1812), and the encoded data block size is counted (step 1814).

Next, a compression ratio is calculated for each encoded data block by taking the ratio of the size of the input data block (as determined by the input counter 10 to the size of each encoded data block output from the enabled encoders (step 1816). Each compression ratio is then compared with an a priori-specified compression ratio threshold (step 1818). It is to be understood that the threshold limit may be specified as any value inclusive of data expansion, no data compression or expansion, or any arbitrarily desired compression limit. It is to be further understood that notwithstanding that the current limit for lossless data-compression is the entropy limit (the present definition of information content) for the data, the present invention does not preclude the use of future developments in lossless data compression that may increase lossless data compression ratios beyond what is currently known within the art. Additionally the content independent data compression threshold may be different from the content dependent threshold and either may be modified by the specific enabled encoders.

After the compression ratios are compared with the threshold, a determination is made as to whether the compression ratio of at least one of the encoded data blocks exceeds the threshold limit (step 1820). If there are no encoded data blocks having a compression ratio that exceeds the compression ratio threshold limit (negative determination in step 1820), then the original unencoded input data block is selected for output and a null data compression type descriptor is appended thereto (step 1834). A null data compression type descriptor is defined as any recognizable data token or descriptor that indicates no data encoding has been applied to the input data block. Accordingly, the unencoded input data block with its corresponding null data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 1836).

On the other hand, if one or more of the encoded data blocks possess a compression ratio greater than the compression ratio threshold limit (affirmative result in step 1820), then the encoded data block having the greatest compression ratio is selected (step 1822). An appropriate data compression type descriptor is then appended (step 1824). A data compression type descriptor is defined as any recognizable data token or descriptor that indicates which data encoding technique has been applied to the data. It is to be understood that, since encoders of the identical type may be applied in parallel to enhance encoding speed (as discussed above), the data compression type descriptor identifies the corresponding encoding technique applied-to the encoded data block, not necessarily the specific encoder. The encoded data block having the greatest compression ratio along with its corresponding data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 1826).

As previously stated the data block stored in the buffer 20 (step 1804) is analyzed on a per block or multi-block basis by the content dependent data recognition module 1300 (step 1806). If the data stream content is recognized or estimated utilizing the recognition list(s) or algorithms(s) module 1710 (affirmative result in step 1808) the recognized data type/file or block is selected based on a list or algorithm (step 1838) and an estimate of the desirability of using the associated content dependent algorithms can be determined (step 1840). For instance, even though a recognized data type may be associated with three different encoders, an estimation of the desirability of using each encoder may result in only one or two of the encoders being actually selected for use. The data is routed to the content dependent encoder module 1320 and compressed by each (enabled) encoder D1 . . . Dm (step 1842). Upon completion of the encoding of the input data block, an encoded data block is output from each (enabled) encoder D1 . . . Dm and maintained in a corresponding buffer (step 1844), and the encoded data block size is counted (step 1846).

Next, a compression-ratio is calculated for each encoded data block by taking the ratio of the size of the input data block (as determined by the input counter 10 to the size of each encoded data block output from the enabled encoders (step 1848). Each compression ratio is then compared with an a priori-specified compression ratio threshold (step 1850). It is to be understood that the threshold limit may be specified as any value inclusive of data expansion, no data compression or expansion, or any arbitrarily desired compression limit. It is to be further understood that many of these algorithms may be lossy, and as such the limits may be subject to or modified by an end target storage, listening, or viewing device. Further notwithstanding that the current limit for lossless data compression is the entropy-limit (the present definition of information content) for the data, the present invention does not preclude the use of future developments in lossless data compression that may-increase lossless data compression ratios beyond what is currently known within the art. Additionally the content independent data compression threshold may be different from the content dependent threshold and either may be modified by the specific enabled encoders.

After the compression ratios are compared with the threshold, a determination is made as to whether the compression ratio of at least one of the encoded data blocks exceeds the threshold limit (step 1820). If there are no encoded data blocks having a compression ratio that exceeds the compression ratio threshold limit (negative determination in step 1820), then the original unencoded input data block is selected for output and a null data compression type descriptor is appended thereto (step 1834). A null data compression type descriptor is defined as any recognizable data token or descriptor that indicates no data encoding has been applied to the input data block. Accordingly, the unencoded input data block with its corresponding null data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 1836).

On the other hand, if one or more of the encoded data blocks possess a compression ratio greater than the compression ratio threshold limit (affirmative result in step 1820), then the encoded data block having the greatest compression ratio is selected (step 1822). An appropriate data compression type descriptor is then appended (step 1824). A data compression type descriptor is defined as any recognizable data token or descriptor that indicates which data encoding technique has been applied to the data. It is to be understood that, since encoders of the identical type may be applied in parallel to enhance encoding speed (as discussed above), the data compression type descriptor identifies the corresponding encoding technique applied to the encoded data block, not necessarily the specific encoder. The encoded data block having the greatest compression ratio along with its corresponding data compression type descriptor is then output for subsequent data processing, storage, or transmittal (step 1826).

After the encoded data block or the unencoded data input data block is output (steps 1826 and 1836), a determination is made as to whether the input data stream contains additional data blocks to be processed (step 1828). If the input data stream includes additional data blocks (affirmative result in step 1428), the next successive data block is received (step 1832), its block size is counted (return to step 1802) and the data compression process in repeated. This process is iterated for each data block in the input data stream. Once the final input data block is processed (negative result in step 1828), data compression of the input data stream is finished (step 1830).

It is to be appreciated that in the embodiments described above with reference to FIGS. 13-18, an a priori specified time limit or any other real-time requirement may be employed to achieve practical and efficient real-time operation.

Although illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the present invention is not limited to those precise embodiments, and that various other changes and modifications may be affected therein by one skilled in the art without departing from the scope or spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as defined by the appended claims.

Claims

1. A method comprising:

associating at least one encoder of a plurality of encoders to each one of a plurality of data types;
determining a data type of a data block, wherein said data block is provided as part of a data stream;
if said determined data type is associated with at least one of said plurality of encoders, compressing said data block with said at least one associated encoder to provide a compressed data block;
determining whether said compressed data block is indicative of data expansion; and
outputting a descriptor with said compressed data block that is indicative of how said compressed data block was compressed.

2. The method of claim 1, further comprising selecting resolution parameters for at least one of said at least one associated encoder.

3. The method of claim 1, further comprising selecting resolution parameters for at least one of said at least one associated encoder, wherein said compressing with said at least one of said at least one associated encoders is performed at said selected resolution.

4. The method of claim 1, further comprising selecting resolution parameters for at least one of said at least one associated encoder, wherein said compressing with said at least one of said at least one associated encoders is performed at said selected resolution and said selected resolution is indicative of lossy compression.

5. The method of claim 1, further comprising determining that no data type is associated with a second data block, wherien said second data block is is provided as part of said data stream.

6. The method of claim 1, further comprising;

determining that no data type is associated with a second data block, wherien said second data block is is provided as part of said data stream; and
compressing said second data block with at least one second encoder.

7. The method of claim 1, wherein said at least one associated encoder is lossless.

8. The method of claim 1, wherein said at least one associated encoder is lossy.

9. The method of claim 1, wherein said at least one associated encoder comprises a plurality of associated encoders.

10. The method of claim 1, wherein said data block is a fixed-size data block.

11. The method of claim 1, wherein said data block is a variable-size data block.

12. The method of claim 1, wherein said at least one associated encoder is lossy or lossless dependent on said data type.

13. The method of claim 1, further comprising transmitting said compressed data block.

14. The method of claim 1, further comprising:

transmitting said compressed data block; and
decompressing said compressed data block based on said descriptor.

15. The method of claim 1, wherein said at least one associated encoder comprises a Huffman encoder.

16. The method of claim 1, wherein said at least one associated encoder comprises a Lempel-Ziv encoder.

17. The method of claim 1, wherein said at least one associated encoder comprises a run length encoder.

18. The method of claim 1, wherein said at least one associated encoder comprises an arithmetic encoder.

19. The method of claim 1, wherein said at least one associated encoder comprises a plurality of encoders that operate in parallel.

20. The method of claim 1, wherein said at least one associated encoder comprises a plurality of encoders that operate sequentially.

21. A method comprising:

associating at least one encoder of a plurality of encoders to each one of a plurality of data types;
determining a data type of a data block, wherein said data block is provided as part of a data stream;
if said determined data type is associated with at least one of said plurality of encoders, compressing said data block with said at least one associated encoder to provide a compressed data block;
determining whether said compressed data block is representative of a data compression ratio above a threshold and, if said determination is above said threshold, outputting said compressed data block; and
outputting a descriptor that is indicative of how said compressed data block was compressed.

22. The method of claim 21, further comprising selecting resolution parameters for at least one of said at least one associated encoder.

23. The method of claim 21, further comprising selecting resolution parameters for at least one of said at least one associated encoder, wherein said compressing with said at least one of said at least one associated encoders is performed at said selected resolution.

24. The method of claim 21, further comprising selecting resolution parameters for at least one of said at least one associated encoder, wherein said compressing with said at least one of said at least one associated encoders is performed at said selected resolution and said selected resolution is indicative of lossy compression.

25. The method of claim 21, further comprising determining that no data type is associated with a second data block, wherien said second data block is is provided as part of said data stream.

26. The method of claim 21, further comprising;

determining that no data type is associated with a second data block, wherien said second data block is provided as part of said data stream; and
compressing said second data block with at least one second encoder.

27. The method of claim 21, wherein said at least one associated encoder is lossless.

28. The method of claim 21, wherein said at least one associated encoder is lossy.

29. The method of claim 21, wherein said at least one associated encoder comprises a plurality of associated encoders.

30. The method of claim 21, wherein said data block is a fixed-size data block.

31. The method of claim 21, wherein said data block is a variable-size data block.

32. The method of claim 21, wherein said at least one associated encoder is lossy or lossless dependent on said data type.

33. The method of claim 21, further comprising transmitting said compressed data block.

34. The method of claim 21, further comprising:

transmitting said compressed data block; and
decompressing said compressed data block based on said descriptor.

35. The method of claim 21, wherein said at least one associated encoder comprises a Huffman encoder.

36. The method of claim 21, wherein said at least one associated encoder comprises a Lempel-Ziv encoder.

37. The method of claim 21, wherein said at least one associated encoder comprises a run length encoder.

38. The method of claim 21, wherein said at least one associated encoder comprises an arithmetic encoder.

39. The method of claim 21, wherein said at least one associated encoder comprises a plurality of encoders that operate in parallel.

40. The method of claim 21, wherein said at least one associated encoder comprises a plurality of encoders that operate sequentially.

41. A machine-readable storage medium encoded with machine executable instructions for performing the method comprising:

associating at least one encoder of a plurality of encoders to each one of a plurality of data types;
determining a data type of a data block, wherein said data block is provided as part of a data stream;
if said determined data type is associated with at least one of said plurality of encoders, compressing said data block with said at least one associated encoder to provide a compressed data block;
determining whether said compressed data block is indicative of data expansion; and
outputting a descriptor with said compressed data block that is indicative of how said compressed data block was compressed.

42. The machine-readable storage medium of claim 41, further comprising instructions for selecting resolution parameters for at least one of said at least one associated encoder.

43. The machine-readable storage medium of claim 41, further comprising instructions for selecting resolution parameters for at least one of said at least one associated encoder, wherein said compressing with said at least one of said at least one associated encoders is performed at said selected resolution.

44. The machine-readable storage medium of claim 41, further comprising instructions for selecting resolution parameters for at least one of said at least one associated encoder, wherein said compressing with said at least one of said at least one associated encoders is performed at said selected resolution and said selected resolution is indicative of lossy compression.

45. The machine-readable storage medium of claim 41, further comprising instructions for determining that no data type is associated with a second data block, wherein said second data block is provided as part of said data stream.

46. The machine-readable storage medium of claim 41, further comprising instructions for:

determining that no data type is associated with a second data block, wherein said second data block is provided as part of said data stream; and
compressing said second data block with at least one second encoder.

47. The machine-readable storage medium of claim 41, wherein said at least one associated encoder is lossless.

48. The machine-readable storage medium of claim 41, wherein said at least one associated encoder is lossy.

49. The machine-readable storage medium of claim 41, wherein said at least one associated encoder comprises a plurality of associated encoders.

50. The machine-readable storage medium of claim 41, wherein said data block is a fixed-size data block.

51. The machine-readable storage medium of claim 41, wherein said data block is a variable-size data block.

52. The machine-readable storage medium of claim 41, wherein said at least one associated encoder is lossy or lossless dependent on said data type.

53. The machine-readable storage medium of claim 41, further comprising instructions for transmitting said compressed data block.

54. The machine-readable storage medium of claim 41, further comprising instructions for:

transmitting said compressed data block; and
decompressing said compressed data block based on said descriptor.

55. The machine-readable storage medium of claim 41, wherein said at least one associated encoder comprises a Huffman encoder.

56. The machine-readable storage medium of claim 41, wherein said at least one associated encoder comprises a Lempel-Ziv encoder.

57. The machine-readable storage medium of claim 41, wherein said at least one associated encoder comprises a run length encoder.

58. The machine-readable storage medium of claim 41, wherein said at least one associated encoder comprises an arithmetic encoder.

59. The machine-readable storage medium of claim 41, wherein said at least one associated encoder comprises a plurality of encoders that operate in parallel.

60. The machine-readable storage medium of claim 41, wherein said at least one associated encoder comprises a plurality of encoders that operate sequentially.

61. A machine-readable storage medium encoded with machine executable instructions for performing the method comprising:

associating at least one encoder of a plurality of encoders to each one of a plurality of data types;
determining a data type of a data block, wherein said data block is provided as part of a data stream;
if said determined data type is associated with at least one of said plurality of encoders, compressing said data block with said at least one associated encoder to provide a compressed data block;
determining whether said compressed data block is representative of a data compression ratio above a threshold and, if said determination is above said threshold, outputting said compressed data block; and
outputting a descriptor that is indicative of how said compressed data block was compressed.

62. The machine-readable storage medium of claim 61, further comprising instructions for selecting resolution parameters for at least one of said at least one associated encoder.

63. The machine-readable storage medium of claim 61, further comprising instructions for selecting resolution parameters for at least one of said at least one associated encoder, wherein said compressing with said at least one of said at least one associated encoders is performed at said selected resolution.

64. The machine-readable storage medium of claim 61, further comprising instructions for selecting resolution parameters for at least one of said at least one associated encoder, wherein said compressing with said at least one of said at least one associated encoders is performed at said selected resolution and said selected resolution is indicative of lossy compression.

65. The machine-readable storage medium of claim 61, further comprising instructions for determining that no data type is associated with a second data block, wherein said second data block is provided as part of said data stream.

66. The machine-readable storage medium of claim 61, further comprising instructions for:

determining that no data type is associated with a second data block, wherein said second data block is provided as part of said data stream; and
compressing said second data block with at least one second encoder.

67. The machine-readable storage medium of claim 61, wherein said at least one associated encoder is lossless.

68. The machine-readable storage medium of claim 61, wherein said at least one associated encoder is lossy.

69. The machine-readable storage medium of claim 61, wherein said at least one associated encoder comprises a plurality of associated encoders.

70. The machine-readable storage medium of claim 61, wherein said data block is a fixed-size data block.

71. The machine-readable storage medium of claim 61, wherein said data block is a variable-size data block.

72. The machine-readable storage medium of claim 61, wherein said at least one associated encoder is lossy or lossless dependent on said data type.

73. The machine-readable storage medium of claim 61, further comprising instructions for transmitting said compressed data block.

74. The machine-readable storage medium of claim 61, further comprising instructions for:

transmitting said compressed data block; and
decompressing said compressed data block based on said descriptor.

75. The machine-readable storage medium of claim 61, wherein said at least one associated encoder comprises a Huffman encoder.

76. The machine-readable storage medium of claim 61, wherein said at least one associated encoder comprises a Lempel-Ziv encoder.

77. The machine-readable storage medium of claim 61, wherein said at least one associated encoder comprises a run length encoder.

78. The machine-readable storage medium of claim 61, wherein said at least one associated encoder comprises an arithmetic encoder.

79. The machine-readable storage medium of claim 61, wherein said at least one associated encoder comprises a plurality of encoders that operate in parallel.

80. The machine-readable storage medium of claim 61, wherein said at least one associated encoder comprises a plurality of encoders that operate sequentially.

Referenced Cited
U.S. Patent Documents
4302775 November 24, 1981 Widergren et al.
4394774 July 19, 1983 Widergren et al.
4574351 March 4, 1986 Dang et al.
4593324 June 3, 1986 Ohkubo et al.
4682150 July 21, 1987 Mathes et al.
4730348 March 8, 1988 MacCrisken
4754351 June 28, 1988 Wright
4804959 February 14, 1989 Makansi et al.
4870415 September 26, 1989 Van Maren et al.
4872009 October 3, 1989 Tsukiyama et al.
4876541 October 24, 1989 Storer
4888812 December 19, 1989 Dinan et al.
4906995 March 6, 1990 Swanson
4929946 May 29, 1990 O'Brien et al.
4953324 September 4, 1990 Herrmann
4965675 October 23, 1990 Hori et al.
4988998 January 29, 1991 O'Brien
5028922 July 2, 1991 Huang
5045848 September 3, 1991 Fascenda
5045852 September 3, 1991 Mitchell et al.
5046027 September 3, 1991 Taaffe et al.
5049881 September 17, 1991 Gibson et al.
5091782 February 25, 1992 Krause et al.
5097261 March 17, 1992 Langdon, Jr. et al.
5113522 May 12, 1992 Dinwiddie, Jr. et al.
5121342 June 9, 1992 Szymborski et al.
5150430 September 22, 1992 Chu
5159336 October 27, 1992 Rabin et al.
5175543 December 29, 1992 Lantz
5179651 January 12, 1993 Taaffe et al.
5187793 February 16, 1993 Keith et al.
5191431 March 2, 1993 Hasegawa et al.
5204756 April 20, 1993 Chevion et al.
5209220 May 11, 1993 Hiyama et al.
5212742 May 18, 1993 Normile et al.
5226176 July 6, 1993 Westaway et al.
5227893 July 13, 1993 Ett
5231492 July 27, 1993 Dangi et al.
5237460 August 17, 1993 Miller et al.
5237675 August 17, 1993 Hannon, Jr.
5243341 September 7, 1993 Seroussi et al.
5243348 September 7, 1993 Jackson
5247638 September 21, 1993 O'Brien
5247646 September 21, 1993 Osterlund et al.
5263168 November 16, 1993 Toms et al.
5270832 December 14, 1993 Balkanski et al.
5287420 February 15, 1994 Barrett
5293379 March 8, 1994 Carr
5307497 April 26, 1994 Feigenbaum et al.
5309555 May 3, 1994 Akins et al.
5355498 October 11, 1994 Provino et al.
5357614 October 18, 1994 Pattisam et al.
5373290 December 13, 1994 Lempel et al.
5379036 January 3, 1995 Storer
5379757 January 10, 1995 Hiyama et al.
5381145 January 10, 1995 Allen et al.
5394534 February 28, 1995 Kulakowski et al.
5396228 March 7, 1995 Garahi
5400401 March 21, 1995 Wasilewski et al.
5403639 April 4, 1995 Belsan et al.
5406278 April 11, 1995 Graybill et al.
5406279 April 11, 1995 Anderson et al.
5412384 May 2, 1995 Chang et al.
5414850 May 9, 1995 Whiting
5420639 May 30, 1995 Perkins
5434983 July 18, 1995 Yaso et al.
5452287 September 19, 1995 DiCecco et al.
5455576 October 3, 1995 Clark, II et al.
5461679 October 24, 1995 Normile et al.
5467087 November 14, 1995 Chu
5471206 November 28, 1995 Allen et al.
5479587 December 26, 1995 Campbell et al.
5483470 January 9, 1996 Alur et al.
5486826 January 23, 1996 Remillard
5495244 February 27, 1996 Jeong et al.
5506844 April 9, 1996 Rao
5506872 April 9, 1996 Mohler
5530845 June 25, 1996 Hiatt et al.
5533051 July 2, 1996 James
5535356 July 9, 1996 Kim et al.
5537658 July 16, 1996 Bakke et al.
5557551 September 17, 1996 Craft
5557668 September 17, 1996 Brady
5557749 September 17, 1996 Norris
5561824 October 1, 1996 Carreiro et al.
5563961 October 8, 1996 Rynderman et al.
5574952 November 12, 1996 Brady et al.
5574953 November 12, 1996 Rust et al.
5583500 December 10, 1996 Allen et al.
5590306 December 31, 1996 Watanabe et al.
5596674 January 21, 1997 Bhandari et al.
5604824 February 18, 1997 Chui et al.
5606706 February 25, 1997 Takamoto et al.
5611024 March 11, 1997 Campbell et al.
5612788 March 18, 1997 Stone
5613069 March 18, 1997 Walker
5615017 March 25, 1997 Choi
5621820 April 15, 1997 Rynderman et al.
5623623 April 22, 1997 Kim et al.
5623701 April 22, 1997 Bakke et al.
5627534 May 6, 1997 Craft
5627995 May 6, 1997 Miller et al.
5629732 May 13, 1997 Moskowitz et al.
5630092 May 13, 1997 Carreiro et al.
5635632 June 3, 1997 Fay et al.
5635932 June 3, 1997 Shinagawa et al.
5638498 June 10, 1997 Tyler et al.
5640158 June 17, 1997 Okayama et al.
5642506 June 24, 1997 Lee
5649032 July 15, 1997 Burt et al.
5652795 July 29, 1997 Dillon et al.
5652857 July 29, 1997 Shimoi et al.
5652917 July 29, 1997 Maupin et al.
5654703 August 5, 1997 Clark, II
5655138 August 5, 1997 Kikinis
5666560 September 9, 1997 Moertl et al.
5668737 September 16, 1997 Iler
5671389 September 23, 1997 Saliba
5675333 October 7, 1997 Boursier et al.
5686916 November 11, 1997 Bakhmutsky
5694619 December 2, 1997 Konno
5696927 December 9, 1997 MacDonald et al.
5703793 December 30, 1997 Wise et al.
5715477 February 3, 1998 Kikinis
5717393 February 10, 1998 Nakano et al.
5717394 February 10, 1998 Schwartz et al.
5719862 February 17, 1998 Lee et al.
5721958 February 24, 1998 Kikinis
5724475 March 3, 1998 Kirsten
5729228 March 17, 1998 Franaszek et al.
5748904 May 5, 1998 Huang et al.
5757852 May 26, 1998 Jericevic et al.
5771340 June 23, 1998 Nakazato et al.
5778411 July 7, 1998 DeMoss et al.
5781767 July 14, 1998 Inoue et al.
5784572 July 21, 1998 Rostoker et al.
5787487 July 28, 1998 Hashimoto et al.
5796864 August 18, 1998 Callahan
5799110 August 25, 1998 Israelsen et al.
5805932 September 8, 1998 Kawashima et al.
5808660 September 15, 1998 Sekine et al.
5809176 September 15, 1998 Yajima
5809337 September 15, 1998 Hannah et al.
5812789 September 22, 1998 Diaz et al.
5818368 October 6, 1998 Langley
5818369 October 6, 1998 Withers
5818530 October 6, 1998 Canfield et al.
5819215 October 6, 1998 Dobson et al.
5825424 October 20, 1998 Canfield et al.
5825830 October 20, 1998 Kopf
5832037 November 3, 1998 Park
5832126 November 3, 1998 Tanaka
5836003 November 10, 1998 Sadeh
5838996 November 17, 1998 deCarmo
5839100 November 17, 1998 Wegener
5841979 November 24, 1998 Schulhof et al.
5847762 December 8, 1998 Canfield et al.
5861824 January 19, 1999 Ryu et al.
5861920 January 19, 1999 Mead et al.
5864342 January 26, 1999 Kajiya et al.
5867167 February 2, 1999 Deering
5867602 February 2, 1999 Zandi et al.
5870036 February 9, 1999 Franaszek et al.
5870087 February 9, 1999 Chau
5872530 February 16, 1999 Domyo et al.
5883975 March 16, 1999 Narita et al.
5886655 March 23, 1999 Rust
5889961 March 30, 1999 Dobbek
5915079 June 22, 1999 Vondran, Jr. et al.
5917438 June 29, 1999 Ando
5920326 July 6, 1999 Rentschler et al.
5936616 August 10, 1999 Torborg, Jr. et al.
5949355 September 7, 1999 Panaoussis
5955976 September 21, 1999 Heath
5960465 September 28, 1999 Adams
5964842 October 12, 1999 Packard
5968149 October 19, 1999 Jaquette et al.
5973630 October 26, 1999 Heath
5974235 October 26, 1999 Nunally et al.
5974471 October 26, 1999 Belt
5978483 November 2, 1999 Thompson, Jr. et al.
5982723 November 9, 1999 Kamatani
5991515 November 23, 1999 Fall et al.
5996033 November 30, 1999 Chiu-Hao
6000009 December 7, 1999 Brady
6002411 December 14, 1999 Dye
6003115 December 14, 1999 Spear et al.
6008743 December 28, 1999 Jaquette
6011901 January 4, 2000 Kirsten
6014694 January 11, 2000 Aharoni et al.
6026217 February 15, 2000 Adiletta
6028725 February 22, 2000 Blumenau
6031939 February 29, 2000 Gilbert et al.
6032148 February 29, 2000 Wilkes
6061398 May 9, 2000 Satoh et al.
6073232 June 6, 2000 Kroeker et al.
6075470 June 13, 2000 Little et al.
6091777 July 18, 2000 Guetz et al.
6094634 July 25, 2000 Yahagi et al.
6097520 August 1, 2000 Kadnier
6104389 August 15, 2000 Ando
6105130 August 15, 2000 Wu et al.
6128412 October 3, 2000 Satoh
6141053 October 31, 2000 Saukkonen
6145069 November 7, 2000 Dye
6169241 January 2, 2001 Shimizu
6172936 January 9, 2001 Kitazaki
6173381 January 9, 2001 Dye
6182125 January 30, 2001 Borella et al.
6192082 February 20, 2001 Moriarty et al.
6195024 February 27, 2001 Fallon
6195465 February 27, 2001 Zandi et al.
6222886 April 24, 2001 Yogeshwar
6225922 May 1, 2001 Norton
6226667 May 1, 2001 Matthews et al.
6226740 May 1, 2001 Iga
6253264 June 26, 2001 Sebastian
6272178 August 7, 2001 Nieweglowski et al.
6272627 August 7, 2001 Mann
6272628 August 7, 2001 Aguilar et al.
6282641 August 28, 2001 Christensen
6308311 October 23, 2001 Carmichael et al.
6309424 October 30, 2001 Fallon
6317714 November 13, 2001 Del Castillo et al.
6330622 December 11, 2001 Schaefer
6345307 February 5, 2002 Booth
6392567 May 21, 2002 Satoh
6404931 June 11, 2002 Chen et al.
6421387 July 16, 2002 Rhee
6434168 August 13, 2002 Kari
6434695 August 13, 2002 Esfahani et al.
6442659 August 27, 2002 Blumenau
6449682 September 10, 2002 Toorians
6452602 September 17, 2002 Morein
6463509 October 8, 2002 Teoman et al.
6487640 November 26, 2002 Lipasti
6489902 December 3, 2002 Heath
6513113 January 28, 2003 Kobayashi
6529633 March 4, 2003 Easwar et al.
6532121 March 11, 2003 Rust et al.
6539456 March 25, 2003 Stewart
6542644 April 1, 2003 Satoh
6577254 June 10, 2003 Rasmussen
6590609 July 8, 2003 Kitade et al.
6597812 July 22, 2003 Fallon et al.
6601104 July 29, 2003 Fallon
6604040 August 5, 2003 Kawasaki et al.
6604158 August 5, 2003 Fallon
6606040 August 12, 2003 Abdat
6606413 August 12, 2003 Zeineh
6609223 August 19, 2003 Wolfgang
6618728 September 9, 2003 Rail
6624761 September 23, 2003 Fallon
6650261 November 18, 2003 Nelson et al.
6661839 December 9, 2003 Ishida et al.
6661845 December 9, 2003 Herath
6704840 March 9, 2004 Nalawadi et al.
6711709 March 23, 2004 York
6717534 April 6, 2004 Yokose
6731814 May 4, 2004 Zeck et al.
6745282 June 1, 2004 Okada et al.
6748457 June 8, 2004 Fallon
6756922 June 29, 2004 Ossia
6810434 October 26, 2004 Muthujumaraswathy et al.
6856651 February 15, 2005 Singh
6885316 April 26, 2005 Mehring
6885319 April 26, 2005 Geiger et al.
6888893 May 3, 2005 Li et al.
6909383 June 21, 2005 Shokrollahi et al.
6944740 September 13, 2005 Abali et al.
7054493 May 30, 2006 Schwartz
7102544 September 5, 2006 Liu
7130913 October 31, 2006 Fallon
7161506 January 9, 2007 Fallon
7181608 February 20, 2007 Fallon et al.
7190284 March 13, 2007 Dye et al.
7321937 January 22, 2008 Fallon
20010031092 October 18, 2001 Zeck et al.
20010032128 October 18, 2001 Kepecs
20010052038 December 13, 2001 Fallon et al.
20020037035 March 28, 2002 Singh
20020080871 June 27, 2002 Fallon et al.
20020101367 August 1, 2002 Geiger et al.
20020104891 August 8, 2002 Otto
20020126755 September 12, 2002 Li et al.
20020191692 December 19, 2002 Fallon et al.
20030030575 February 13, 2003 Frachtenberg et al.
20030034905 February 20, 2003 Anton et al.
20030084238 May 1, 2003 Okada et al.
20030090397 May 15, 2003 Rasmussen
20030142874 July 31, 2003 Schwartz
20030191876 October 9, 2003 Fallon
20040042506 March 4, 2004 Fallon et al.
20040073710 April 15, 2004 Fallon
20060015650 January 19, 2006 Fallon
20060181441 August 17, 2006 Fallon
20060181442 August 17, 2006 Fallon
20060184696 August 17, 2006 Fallon
20060190644 August 24, 2006 Fallon
20060195601 August 31, 2006 Fallon
20070043939 February 22, 2007 Fallon et al.
20070050514 March 1, 2007 Fallon
20070050515 March 1, 2007 Fallon
20070067483 March 22, 2007 Fallon
20070083746 April 12, 2007 Fallon et al.
20070109154 May 17, 2007 Fallon
20070109155 May 17, 2007 Fallon
20070174209 July 26, 2007 Fallon et al.
Foreign Patent Documents
4127518 February 1992 DE
0164677 December 1985 EP
0185098 June 1986 EP
0283798 September 1988 EP
0405572 September 1988 EP
0595406 May 1994 EP
0718751 June 1996 EP
0493130 June 1997 EP
0587437 February 2002 EP
2162025 January 1986 GB
6051989 February 1994 JP
9188009 July 1997 JP
11149376 June 1999 JP
WO 9414273 June 1994 WO
WO 9429852 December 1994 WO
WO 9502873 January 1995 WO
WO 9748212 December 1997 WO
Other references
  • Rice, Robert F., “Some Practical Universal Noiseless Coding Techniques”, Jet Propulsion Laboratory, Pasadena, California, JPL Publication 79-22, Mar. 15, 1979.
  • Anderson, J., et al., “Codec squeezes color teleconferencing through digital telephone lines”, Electronics 1984, pp. 113-115.
  • Venbrux, Jack, “A VLSI Chip Set for High-Speed Lossless Data Compression”, IEEE Trans. On Circuits and Systems for Video Technology, vol. 2, No. 44, Dec. 1992, pp. 381-391.
  • “Fast Dos Soft Boot”, IBM Technical Disclosure Bulletin, Feb. 1994, vol. 37, Issue No. 2B, pp. 185-186.
  • “Operating System Platform Abstraction Method”, IBM Technical Disclosure Bulletin, Feb. 1995, vol. 38, Issue No. 2, pp. 343-344.
  • Murashita, K., et al., “High-Speed Statistical Compression using Self-Organized Rules and Predetermined Code Tables”, IEEE, 1996 Data Compression Conference.
  • Coene, W., et al., “A Fast Route For Application of Rate-distortion Optimal Quantization in an MPEG Video Encoder” Proceedings of the International Conference on Image Processing, US., New York, IEEE, Sep. 16, 1996, pp. 825-828.
  • Rice, Robert, “Lossless Coding Standards for Space Data Systems”, IEEE 1058-6393/97, pp. 577-585, no date.
  • Millman, Howard, “Image and video compression”, Computerworls, vol. 33, Issue No. 3, Jan. 18, 1999, pp. 78.
  • “IBM boosts your memory”, Geek.com [online], Jun. 26, 2000 [retrieved on Jul. 6, 2007], <URL: http://www.geek.com/ibm-boosts-your-memory/>.
  • “IBM Research Breakthrough Doubles Computer Memory Capacity”, IBM Press Release [online], Jun. 26, 2000 [retrieved on Jul. 6, 2007], <URL: http://www-03.ibm.com/press/us/en/pressrelease /1653.wss>.
  • “ServerWorks To Deliver IBM's Memory eXpansion Technology in Next-Generation Core Logic for Servers”, ServerWorks Press Release [online], Jun. 27, 2000 [retrieved on Jul. 14, 2000], <URL: http://www.serverworks.com/news/press/ 000627.html>.
  • Abali, B., et al., “Memory Expansion Technology (MXT) Software support and performance”, IBM Journal of Research and Development, vol. 45, Issue No. 2, mar. 2001, pp. 287-301.
  • Franaszek, P.A., et al., “Algorithms and data structures for compressed memory machines”,IBM Journal of Research and Development, vol. 45, Issue No. 2, Mar. 2001, pp. 245-258.
  • Franaszek, P.A. et al., “On internal organization in compressed random access memories”, IBM Journal of Research and Development, vol. 45, Issue No. 2, Mar. 2001, pp. 259-270.
  • Smith, T.B., et al., “Memory Expansion Technology (MXT) Competitve impact”, IBM Journal of Research and Development, Vo. 45, Issue No. 2, Mar. 2001, pp. 303-309.
  • Tremaine, R. B., et al., “IBM Memory Expansion Technology (MXT)”, IBM Journal of Research and Development, vol. 45, Issue No. 2, Mar. 2001, pp. 271-285.
  • Yeh, Pen-Shu, “The CCSDS Lossless Data Compression Recommendation for Space Applcations”, Chapter 16, Lossless Compress Handbook, Elsevier Science (USA), 2003, pp. 311-326.
Patent History
Patent number: 7432833
Type: Grant
Filed: Jan 8, 2007
Date of Patent: Oct 7, 2008
Patent Publication Number: 20070109156
Assignee: Realtime Data LLC (New York, NY)
Inventor: James J. Fallon (Armonk, NY)
Primary Examiner: Linh V Nguyen
Attorney: Ropes & Gray LLP
Application Number: 11/651,366