PERFORMING DE-DUPLICATION FOR AT LEAST ONE COMPUTER FILE IN A COMPUTER SYSTEM
The present invention provides a method and system of performing de-duplication for at least one computer file in a computer system. In an exemplary embodiment, the method and system include (1) tuning a rolling-hash algorithm for the de-duplication, (2) chunking the data in the file into chunks of data by using the tuned algorithm, (3) producing a content identifier for each of the chunks, and (4) processing the chunks that are unique, the content identifier for each of the chunks that are unique, and references to the chunks that are unique. In an exemplary embodiment, the computer system includes a de-duplication-enabled data store. In an exemplary embodiment, the computer system includes (a) a transferor computer system that is configured to transfer the file to a de-duplication-enabled computer system and (b) the de-duplication-enabled computer system.
Latest IBM Patents:
- Shareable transient IoT gateways
- Wide-base magnetic tunnel junction device with sidewall polymer spacer
- AR (augmented reality) based selective sound inclusion from the surrounding while executing any voice command
- Confined bridge cell phase change memory
- Control of access to computing resources implemented in isolated environments
1. Problems with De-Duplicated File-Systems
Computer systems (e.g. server computer systems) need the ability to perform efficient data de-duplication on data. Backup solutions for computer file-system have been including some form of data “de-duplication” or data “redundancy elimination” algorithms. These algorithms can be used at the whole-file or at the sub-file level.
2. Prior Art Systems
Referring to
These unique identifiers are then placed into an index that can be queried. When a chunk is found which already exists in the system (found by querying the index or attempting an insert and getting a collision), that chunk can be replaced by a reference to that chunk, and “de-duplication” occurs. For each file that is chunked, a “blueprint” (references to unique chunks) is produced identifying how to reconstruct the file from its constituent parts or chunks.
The size of the data chunks has a dramatic effect on the de-duplication rates. As the size of the chunks decreases, the de-duplication rate increases, and the size of the unique content identifier index and the size of the “blueprints” used to reconstruct objects from their parts increase, resulting in slower object “reconstruction” from the “blueprints”. As the size of the chunks increases, the size of the unique content identifier index and the size of the “blueprints” decrease while the object “reconstruction” from the “blueprints” becomes faster.
One problem with this prior art de-duplication system relates to certain types of files (e.g., JPEGs, MPEGs, other similarly compressed media files) that are very unlikely to have duplicate sub-file chunks with other files. Specifically, these types of files can use a very large chunk size, whereas other files in the system can use a smaller chunk size. Other factors may also influence whether a certain file type is a candidate for smaller or larger chunking. For example, file types that are typically “consumed” by the user(s) of the system containing these files are generally poor candidates for data de-duplication, and, thus, would benefit from larger block or whole-file de-duplication.
File types that are typically “produced” by the user(s) of the system containing those files would typically be good candidates for data de-duplication. This is because such files will often be found in different stages of preparedness (e.g., drafts, modifications, alterations). For example, movie files (such as those available from YouTube, LLC) often have multiple versions of the same movie and would benefit from smaller-block data de-duplication.
A solution is required which can achieve high speeds without sacrificing de-duplication rates. Therefore, a method and system of performing de-duplication for at least one computer file in a computer system is needed.
SUMMARY OF THE INVENTIONThe present invention provides a method and system of performing de-duplication for at least one computer file in a computer system. In an exemplary embodiment, the method and system include (1) tuning a rolling-hash algorithm for the de-duplication, (2) chunking the data in the file into chunks of data by using the tuned algorithm, (3) producing a content identifier for each of the chunks, and (4) processing the chunks that are unique, the content identifier for each of the chunks that are unique, and references to the chunks that are unique. In an exemplary embodiment, the computer system includes a de-duplication-enabled data store. In an exemplary embodiment, the computer system includes (a) a transferor computer system that is configured to transfer the file to a de-duplication-enabled computer system and (b) the de-duplication-enabled computer system.
In an exemplary embodiment, the tuning includes tuning the algorithm based on the type of content in the file. In an exemplary embodiment, the tuning includes tuning the algorithm based on the size of the file. In an exemplary embodiment, the tuning includes tuning the algorithm based on the resource constraints of the system. In a specific embodiment, the tuning includes tuning the algorithm based on the remaining space in a data store of the system. In a specific embodiment, the tuning includes tuning the algorithm based on the bandwidth of a transmission medium of the system.
In an exemplary embodiment, the tuning includes tuning the algorithm based on file type usage in the system. In a specific embodiment, the tuning includes, if the file type usage is produced, decreasing a chunk size that the algorithm will produce. In a specific embodiment, the tuning includes, if the file type usage is consumed, increasing a chunk size that the algorithm will produce.
In an exemplary embodiment, the tuning includes adjusting a minimum chunk size that the algorithm will produce. In an exemplary embodiment, the tuning includes adjusting an average expected chunk size that the algorithm will produce. In an exemplary embodiment, the tuning includes adjusting a maximum chunk size that the algorithm will produce. In an exemplary embodiment, the tuning includes adjusting an input window size upon which the algorithm will operate.
In an exemplary embodiment, the producing includes applying a collision-resistant hashing algorithm to each of the chunks. In a specific embodiment, the applying includes applying SHA-1 to each of the chunks.
In an exemplary embodiment, the processing includes, if an index of the computer system does not include the content identifier, (a) identifying the chunk corresponding to the content identifier as being a unique chunk, (b) identifying the content identifier as a unique content identifier, and (c) adding the unique content identifier to the index. In an exemplary embodiment, the processing includes, if an index of the computer system includes the content identifier, (a) identifying the chunk corresponding to the content identifier as being a duplicated chunk, (b) removing the duplicated chunk from the system, and (c) adding a reference to the identified chunk.
In an exemplary embodiment, the processing includes, if an index of the computer system does not include the content identifier, (a) identifying the chunk corresponding to the content identifier as being a unique chunk, (b) receiving the identified chunk, (c) identifying the content identifier as a unique content identifier, and (d) adding the unique content identifier to the index. In an exemplary embodiment, the processing includes, if an index of the computer system includes the content identifier, (a) identifying the chunk corresponding to the content identifier as being a duplicated chunk and (b) adding a reference to the identified chunk.
The present invention also provides a computer program product usable with a programmable computer having readable program code embodied therein of performing de-duplication for at least one computer file in a computer system. In an exemplary embodiment, the computer program product includes (1) computer readable code for tuning a rolling-hash algorithm for the de-duplication, (2) computer readable code for chunking the data in the file into chunks of data by using the tuned algorithm, (3) computer readable code for producing a content identifier for each of the chunks, and (4) computer readable code for processing the chunks that are unique, the content identifier for each of the chunks that are unique, and references to the chunks that are unique.
The present invention provides a method and system of performing de-duplication for at least one computer file in a computer system. In an exemplary embodiment, the method and system include (1) tuning a rolling-hash algorithm for the de-duplication, (2) chunking the data in the file into chunks of data by using the tuned algorithm, (3) producing a content identifier for each of the chunks, and (4) processing the chunks that are unique, the content identifier for each of the chunks that are unique, and references to the chunks that are unique. In an exemplary embodiment, the computer system includes a de-duplication-enabled data store. In an exemplary embodiment, the computer system includes (a) a transferor computer system that is configured to transfer the file to a de-duplication-enabled computer system and (b) the de-duplication-enabled computer system.
Referring to
In an exemplary embodiment, tuner 222 tunes the rolling-hash algorithm 232 based attributes of the computer file 230 and system properties 231 (e.g., resource constraints of the system). In an exemplary embodiment, Chunker 224 then chunks computer file 230 into chunks, such as chunk 234 and chunk 236, via tuned rolling-hash algorithm 232. In an exemplary embodiment, Content Identifier Producer 226 thereafter produces a content identifier for each of the chunks, content identifier 238 for chunk 234 and content identifier 239 for chunk 236. In an exemplary embodiment, the present invention, via step 218, then processes the chunks that are unique, the content identifier for each of the chunks that are unique, and references to the chunks that are unique, via a De-duplicator 240, an Index of Unique Chunks 242, a References to Unique Chunks 244 (e.g., a “blueprint”), and a Data Store 246.
Tuning the AlgorithmReferring to
Referring to
Referring to
In an exemplary embodiment, the tuning includes setting the minimum chunk size, the average expected chunk size, and the maximum chunk size to the same value. This setting could constitute fixed-block chunking. In an exemplary embodiment, the tuning includes setting the minimum chunk size, the average expected chunk size, the maximum chunk size, and the maximum chunk size to their maximum values. This setting could constitute whole-file de-duplication.
Producing a Content IdentifierReferring to
Referring to
Referring to
The present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment containing both hardware and software elements. In an exemplary embodiment, the present invention is implemented in software, which includes but is not limited to firmware, resident software, and microcode.
Furthermore, the present invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer system or any instruction execution system. The computer program product includes the instructions that implement the method of the present invention. A computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid-state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, and an optical disk. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read/write (CD-R/W), and DVD.
A computer system suitable for storing and/or executing program code includes at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements include local memory employed during actual execution of the program code, bulk storage, and cache memories that provide temporary storage of at least some program code to reduce the number of times code is retrieved from bulk storage during execution. Input/output (I/O) devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the computer system either directly or through intervening I/O controllers. Network adapters may also be coupled to the computer system in order to enable the computer system to become coupled to other computer systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters. The computer system can also include an operating system and a compute file-system.
CONCLUSIONHaving fully described a preferred embodiment of the invention and various alternatives, those skilled in the art will recognize, given the teachings herein, that numerous alternatives and equivalents exist which do not depart from the invention. It is therefore intended that the invention not be limited by the foregoing description, but only by the appended claims.
Claims
1. A method of performing de-duplication for at least one computer file in a computer system, the method comprising:
- tuning a rolling-hash algorithm for the de-duplication;
- chunking the data in the file into chunks of data by using the tuned algorithm;
- producing a content identifier for each of the chunks; and
- processing the chunks that are unique, the content identifier for each of the chunks that are unique, and references to the chunks that are unique.
2. The method of claim 1 wherein the tuning comprises tuning the algorithm based on the type of content in the file.
3. The method of claim 1 wherein the tuning comprises tuning the algorithm based on the size of the file.
4. The method of claim 1 wherein the tuning comprises tuning the algorithm based on the resource constraints of the system.
5. The method of claim 4 wherein the tuning comprises tuning the algorithm based on the remaining space in a data store of the system.
6. The method of claim 4 wherein the tuning comprises tuning the algorithm based on the bandwidth of a transmission medium of the system.
7. The method of claim 1 wherein the tuning comprises tuning the algorithm based on file type usage in the system.
8. The method of claim 7 wherein the tuning comprises, if the file type usage is produced, decreasing a chunk size that the algorithm will produce.
9. The method of claim 7 wherein the tuning comprises, if the file type usage is consumed, increasing a chunk size that the algorithm will produce.
10. The method of claim 1 wherein the tuning comprises adjusting a minimum chunk size that the algorithm will produce.
11. The method of claim 1 wherein the tuning comprises adjusting an average expected chunk size that the algorithm will produce.
12. The method of claim 1 wherein the tuning comprises adjusting a maximum chunk size that the algorithm will produce.
13. The method of claim 1 wherein the tuning comprises adjusting an input window size upon which the algorithm will operate.
14. The method of claim 1 wherein the producing comprises applying a collision-resistant hashing algorithm to each of the chunks.
15. The method of claim 14 wherein the applying comprises applying SHA-1 to each of the chunks.
16. The method of claim 1 wherein the processing comprises, if an index of the computer system does not include the content identifier:
- identifying the chunk corresponding to the content identifier as being a unique chunk;
- identifying the content identifier as a unique content identifier; and
- adding the unique content identifier to the index.
17. The method of claim 1 wherein the processing comprises, if an index of the computer system includes the content identifier:
- identifying the chunk corresponding to the content identifier as being a duplicated chunk;
- removing the duplicated chunk from the system; and
- adding a reference to the identified chunk.
18. The method of claim 1 wherein the processing comprises, if an index of the computer system does not include the content identifier:
- identifying the chunk corresponding to the content identifier as being a unique chunk;
- receiving the identified chunk;
- identifying the content identifier as a unique content identifier; and
- adding the unique content identifier to the index.
19. The method of claim 1 wherein the processing comprises, if an index of the computer system includes the content identifier:
- identifying the chunk corresponding to the content identifier as being a duplicated chunk; and
- adding a reference to the identified chunk.
20. A system of performing de-duplication for at least one computer file in a computer system, the system comprising:
- a tuning module configured to tune a rolling-hash algorithm for the de-duplication;
- a chunking module configured to chunk the data in the file into chunks of data by using the tuned algorithm;
- a producing module configured to produce a content identifier for each of the chunks; and
- a processing module configured to process the chunks that are unique, the content identifier for each of the chunks that are unique, and references to the chunks that are unique.
21. The system of claim 20 wherein the tuning module comprises a tuning module configured to tune the algorithm based on the type of content in the file.
22. The system of claim 20 wherein the tuning module comprises a tuning module configured to tune the algorithm based on the size of the file.
23. The system of claim 20 wherein the tuning module comprises a tuning module configured to tune the algorithm based on the resource constraints of the system.
24. The system of claim 23 wherein the tuning module comprises a tuning module configured to tune the algorithm based on the remaining space in a data store of the system.
25. The system of claim 23 wherein the tuning module comprises a tuning module configured to tune the algorithm based on the bandwidth of a transmission medium of the system.
26. The system of claim 20 wherein the tuning module comprises a tuning module configured to tune the algorithm based on file type usage in the system.
27. The system of claim 26 wherein the tuning module comprises, if the file type usage is produced, a decreasing module configured to decrease a chunk size that the algorithm will produce.
28. The system of claim 26 wherein the tuning module comprises, if the file type usage is consumed, an increasing module configured to increase a chunk size that the algorithm will produce.
Type: Application
Filed: Apr 30, 2008
Publication Date: Nov 5, 2009
Patent Grant number: 8620877
Applicant: INTERNATIONAL BUSINESS MACHINES CORPORATION (ARMONK, NY)
Inventor: Mark Andrew Smith (Los Gatos, CA)
Application Number: 12/113,136
International Classification: G06F 17/30 (20060101);