Method and system for data reduction
A “forward” delta data management technique uses a “sparse” index associated with a delta file to achieve both delta management efficiency and to eliminate read latency while accessing history data. The invention may be implemented advantageously in a data management system that provides real-time data services to data sources associated with a set of application host servers. A host driver embedded in an application server connects an application and its data to a cluster. The host driver captures real-time data transactions, preferably in the form of an event journal that is provided to the data management system. In particular, the driver functions to translate traditional file/database/block I/O into a continuous, application-aware, output data stream. A given application-aware data stream is processed through a multi-stage data reduction process to produce a compact data representation from which an “any point-in-time” reconstruction of the original data can be made.
Latest Dell Software Inc. Patents:
- Systems and methods for predictive logins to session(s) or resource(s)
- Secure shell authentication
- System and method for enforcing access control to publicly-accessible web applications
- System for Rapid Identification of Sources of Variation in Complex Manufacturing Processes
- System for Managing Effective Self-Service Analytic Workflows
The present application is a continuation of U.S. patent application Ser. No. 12/901,824, filed on Oct. 11, 2010, which is a continuation of U.S. patent application Ser. No. 10/943,541, filed on Sep. 17, 2004, now U.S. Pat. No. 7,979,404, issued Jul. 12, 2011, each of which is hereby incorporated herein by reference in its entirety. The present application is related to commonly-owned U.S. Pat. No. 7,096,392, issued Aug. 22, 2006 and U.S. Pat. No. 7,565,661, issued Jul. 21, 2009, each of which is hereby incorporated herein by reference in its entirety.
BACKGROUND OF THE INVENTION1. Technical Field
The present invention relates generally to enterprise data protection and data management.
2. Background of the Related Art
Techniques for managing data history in distributed computing systems are known in the art. In particular, traditional content management systems typically manage file history by using either “forward delta” management, “reverse delta” management, or a combination of both techniques. A forward delta management system maintains an initial baseline of the file as well as a list of deltas (changes to the file) that occur after the baseline is created. In a forward delta management system, deltas are appended to a delta document sequentially. An advantage of such a system is that, as deltas arrive, the system only needs to append them to an end of a delta document. However, when a user tries to access a file (or when a host needs to recover its lost data to a specific point-in-time, version, or the most current point-in-time), the forward delta management system must (at runtime) take the baseline and apply the necessary delta strings “on the fly” to generate the requested point-in-time data. If there is a long list of delta strings, the read latency of such an operation may be very long; in addition, the cache required to process the delta strings during the read operation may be unacceptably high.
A reverse delta management system maintains the most current point-in-time data and a list of reverse deltas (an “undo” list) in a delta management file. A reverse delta management system first takes a given forward delta and applies the delta to last point-in-time data to generate the most current point-in-time data; it then uses the most current point-in-time data to compare with the last point-in-time data to generate an undo (reverse) delta. This type of system only keeps the most current data file and a list of undo deltas. If the most current data is requested, the data can be retrieved instantly. If, however, data from a previous point-in-time is requested, this system must take the most current data file and apply the necessary undo delta(s) to generate the requested point-in-time data. The baseline copy in this system is the most current point-in-time copy. In many cases, there may be a significant read latency for previous data. In addition, the computing power needed for ongoing data updates in such a data management system is very significant. This technique also does not support data replication over an unreliable network, as the baseline copy of the data is constantly changing.
When performing incremental data protection, traditional data management systems copy the entire contents of a changed file into a protection repository, where the file history is saved. These systems, however, do not apply any delta management techniques, such as those described above, to manage the file history. Moreover, because these systems are not storage and bandwidth efficient, they are not suitable for performing real-time data services.
The traditional content management systems can manage file history, but they are not capable of managing unstructured and dynamic data. Further, a traditional system of this type requires that its data source be well-structured, i.e., having directories that are created and configured in advance. In most cases, a given content management system is designed to manage a specific content type as opposed to dynamic data. Thus, for example, a given source control system may be designed to manage design documents or source code, but that same system cannot manage data that changes constantly. These systems also are not capable of protecting changing data in real-time. To the extent they include delta management schemes, such schemes do not enable efficient any-point-in-time data recovery.
There remains a need in the art to provide distributed data management systems that can efficiently manage real-time history of a large amount of unstructured and dynamic data with minimal storage and bandwidth usage.
There also remains a need in the art to provide such a distributed data management system that can perform virtual-on-demand recovery of consistent data at any point-in-time in the past.
The present invention addresses these deficiencies in the art.
SUMMARY OF THE INVENTIONIt is a general object of the present invention to provide for efficient transfer of real-time data changes over a local and/or wide area network.
It is also a general object of the invention to provide for efficient storage of data history over a given time period.
It is a more specific object of the present invention to provide novel data reduction techniques that facilitate any-point-in-time virtual on-demand data recovery in a data management system.
A specific object of the invention is to implement an improved “forward” delta data management technique wherein a “sparse” index is associated with a delta file to achieve both delta management efficiency and to eliminate read latency while accessing history data of any point-in-time.
Another more specific object of the present invention is to provide a novel data management technique to create a given data structure for use in managing data history for a file that is constantly changing. According to the invention, the given data structure need not include the actual contents of later-created versions of a particular file as that file is updated in the system. Nevertheless, the information in the given data structure is such that each of the given versions can be readily reconstructed “on-the-fly” (as-needed) without extensive read latency to apply deltas to the baseline data.
The present invention may be implemented advantageously in a data management system or “DMS” that provides a wide range of data services to data sources associated with a set of application host servers. The data management system typically comprises one or more regions, with each region having one or more clusters. A given cluster has one or more nodes that share storage. To facilitate a given data service, a host driver embedded in an application server connects an application and its data to a cluster. The host driver captures real-time data transactions, preferably in the form of an event journal that is provided to the data management system. In particular, the driver functions to translate traditional file/database/block I/O into a continuous, application-aware, output data stream. Application aware event journaling is a technique to create real-time data capture so that, among other things, consistent data checkpoints of an application can be identified and metadata can be extracted. Application aware event journaling tracks granular application consistent checkpoints. Thus, when a DMS is used to provide file system data protection to a given data source, it is capable of reconstructing an application data state to a consistent point-in-time in the past.
According to an illustrative embodiment, a given application aware data stream is processed through a multi-stage data reduction process to produce a compact data representation from which an “any point-in-time” reconstruction of the original data can be made.
The foregoing has outlined some of the more pertinent features of the invention. These features should be construed to be merely illustrative. Many other beneficial results can be attained by applying the disclosed invention in a different manner or by modifying the invention as will be described.
For a more complete understanding of the present invention and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:
As illustrated in commonly-owned, U.S. Pat. No. 7,565,661, issued Jul. 21, 2009, a “host driver” 128 is associated with one or more of the application(s) running in the application servers 116 to transparently and efficiently capture the real-time, continuous history of all (or substantially all) transactions and changes to data associated with such application(s) across the enterprise network. As will be described below, this facilitates real-time, so-called “application aware” protection, with substantially no data loss, to provide continuous data protection and other data services including, without limitation, data distribution, data replication, data copy, data access, and the like. In operation, a given host driver 128 intercepts data events between an application and its primary data storage, and it may also receive data and application events directly from the application and database. In a representative embodiment, the host driver 128 is embedded in the host application server 116 where the application resides; alternatively, the host driver is embedded in the network on the application data path. By intercepting data through the application, fine grain (but opaque) data is captured to facilitate the data service(s). To this end, and as also illustrated in
Referring now to
The DMS provides these and other business continuity data services in real-time with data and application awareness to ensure continuous application data consistency and to allow for fine grain data access and recovery. To offer such application and data aware services, the DMS has the capability to capture fine grain and consistent data. As will be illustrated and described, a given DMS host driver uses an I/O filter to intercept data events between an application and its primary data storage. The host driver also receives data and application events directly from the application and database.
Referring now to
In this embodiment, a host server embedded host driver is used for illustrating the driver behavior. In particular, the host driver 500 in a host server connects to one of the DMS nodes in a DMS cluster (in a DMS region) to perform or facilitate a data service. The host driver preferably includes two logical subsystems, namely, an I/O filter 502, and at least one data agent 504. An illustrative data agent 504 preferably includes one or more modules, namely, an application module 506, a database module 508, an I/O module 510, and an event processor or event processing engine 512. The application module 506 is configured with an application 514, one or more network devices and/or the host system itself to receive application level events 516. These events include, without limitation, entry or deletion of some critical data, installation or upgrade of application software or the operating system, a system alert, detecting of a virus, an administrator generated checkpoint, and so on. One or more application events are queued for processing into an event queue 518 inside or otherwise associated with the data agent. The event processor 512 over time may instruct the application module 506 to re-configure with its event source to capture different application level events.
If an application saves its data into a database, then a database module 508 is available for use. The database module 508 preferably registers with a database 520 to obtain notifications from a database. The module 508 also may integrate with the database 520 through one or more database triggers, or it may also instruct the database 520 to generate a checkpoint 522. The database module 508 also may lock the database 520 (or issue a specific API) to force a database manager (not shown) to flush out its data from memory to disk, thereby generating a consistent disk image (a binary table checkpoint). This process of locking a database is also known as “quiescing” the database. An alternative to quiescing a database is to set the database into a warm backup mode. After a consistent image is generated, the database module 508 then lifts a lock to release the database from its quiescent state. The database events preferably are also queued for processing into the event queue 518. Generalizing, database events include, without limitation, a database checkpoint, specific database requests (such as schema changes or other requests), access failure, and so on. As with application module, the event processor 512 may be used to re-configure the events that will be captured by the database module.
The I/O module 510 instructs the I/O filter 502 to capture a set of one or more I/O events that are of interest to the data agent. For example, a given I/O module 510 may control the filter to capture I/O events synchronously, or the module 510 may control the filter to only capture several successful post I/O events. When the I/O module 510 receives I/O events 524, it forwards the I/O events to the event queue 518 for processing. The event processor 512 may also be used to re-configure the I/O module 510 and, thus, the I/O filter 502.
The event processor 512 functions to generate an application aware, real-time event journal (in effect, a continuous stream) for use by one or more DMS nodes to provide one or more data services. Application aware event journaling is a technique to create real-time data capture so that, among other things, consistent data checkpoints of an application can be identified and metadata can be extracted. For example, application awareness is the ability to distinguish a file from a directory, a journal file from a control or binary raw data file, or to know how a file or a directory object is modified by a given application. Thus, when protecting a general purpose file server, an application aware solution is capable of distinguishing a file from a directory, and of identifying a consistent file checkpoint (e.g., zero-buffered write, flush or close events), and of interpreting and capturing file system object attributes such as an access control list. By interpreting file system attributes, an application aware data protection may ignore activities applied to a temporary file. In general, application aware event journaling tracks granular application consistent checkpoints; thus, when used in conjunction with data protection, the event journal is useful in reconstructing an application data state to a consistent point-in-time in the past, and it also capable of retrieving a granular object in the past without having to recover an entire data volume. In the DMS, data protection typically begins with an initial upload phase, when a full copy of a host data source is uploaded to a DMS cluster. During and after the upload is completed, application(s) may continue to update the data, in which case event journals are forwarded to the DMS as data is modified. Further details of the event journaling technique are described in commonly-owned, U.S. Pat. No. 7,565,661, issued Jul. 21, 2009, which is incorporated herein by reference.
With the above as background, the multi-stage data reduction process of the present invention can now be described. A preferred multi-stage data reduction has a first stage, and a second stage. Typically, a first-stage data reduction takes place at a given host driver, whereas a second-stage data reduction takes place at a given DMS node of a given DMS cluster at which the first-stage data is delivered initially. This approach (which is not to be taken by way of limitation) is illustrated diagrammatically in
As data is changed in the protected host server, a new version of the data is created. This version, however, need not actually be stored in the DMS cluster, as will now be seen with reference to
By structuring the data history object in the manner illustrated in
As an example, if a user file is 10K bytes in length but the update involves just 2 bytes, a typical application would write an entire file locally; in the DMS, however, only the associated new metadata (which includes the new sparse index) is written to disk along with the second-stage delta string (that represents the 2 bytes). As additional updates occur, each subsequent new version is managed in the same way, i.e., without storing (in DMS) the actual binary content of the update and with only the simple creation of new metadata (including the new sparse index) and additional sequencing of the dfile. When it comes time to reconstruct a given version, the layout of the flat file (with the metadata version blocks preferably reverse ordered) provides for efficient file read operation. In particular, during the read, the actual data bytes are located using the sparse indices (of that version), which point to information in the bfile and dfile as needed. The information in the bfile and dfile is then used to create the version under reconstruction.
Thus, according to the present invention, a given version (an updated file) need not be stored in the DMS cluster; rather, as long as the bfile, the dfile and the sparse index (for that version) exist, the actual contents of the version can be reconstructed efficiently and reliably.
As noted above, preferably the first-stage data reduction uses a signature-based algorithm to extract changed data ranges instead of comparing the current changes to the previous data version. This operation minimizes both bandwidth utilization and storage overhead. A convenient algorithm to perform the first-stage data reduction operation is Rsync, which is available as an open source implementation from several online locations, e.g., http://samba.anu.edu.au/rsync/. In an alternative embodiment, or if bandwidth is not a concern, the first-stage data reduction can operate by using any delta differencing algorithm that merely compares the current changes to the previous data version. More generally, any known or later-developed checksum-based delta extraction algorithm may be used.
As noted above, an important goal of the present invention is to reduce significantly the amount of storage required for storing data history in an environment where data is consistently changing and the data must be available over a wide area. As will be seen, this goal is achieved by the present invention through the combination of the first-stage and second-stage data reduction, especially where the latter data reduction step is associated with a sparse indexing technique. This multi-stage data reduction ensures that only minimal storage is required for storing data history and that only minimal wide-area-network bandwidth is required for distribution and replication.
The first and second stage data reduction is now illustrated. In an illustrated embodiment, each version of a binary object (such as a file or a database volume) in the DMS has an associated sparse index in their version metadata defined by the following syntax:
In addition, according to the invention, preferably each version (other than the first version) of the binary file also has an associated first stage delta string, together with an associated second stage delta string. As each version of a binary file is generated (typically in the host server), a data reduction method is implemented to generate the first stage delta string and the second stage delta string. Further, as each version of a binary file is generated (preferably at the DMS), the resulting second stage delta string is concatenated into a composite stream, which was identified above as the delta file (dfile). The dfile contains a sequence of second-stage delta strings. When there is only one changed version, the delta file includes just one (or one or more related) second stage delta string(s). However, as additional versions are created, the delta file becomes a composite of the second stage delta strings (i.e., it includes all the second-stage delta strings for all the version updates). The dfile is a highly compact encoding of the version differences, as will be seen below. As described above in connection with
The following table describes a representative delta string syntax that may be implemented to generate the first and second stage delta strings according to the present invention:
The above-described syntax should not be taken to limit the present invention. Any syntax that defines given data insertions, deletions, replacements or other data comparison operations may be used.
The application of the above-identified syntax according to the present invention can be illustrated by way of example. Assume that the original data range stored in the host server (e.g, cluster 644 of
By way of example only, the following chart assumes that each content character in the example represents 100 bytes and that the signature-block size used by the first-stage data reduction checksum based algorithm is 400 bytes. This means that a checksum is generated for each 400 bytes of data. It is also assumed that each delta string symbols (+, −, R) representation is 1 byte and that the offset and length are 4 byte numbers.
As can be seen, the delta file (dfile) is a string into which the second stage delta reduction strings are concatenated as new versions are created. This is a forward delta encoding format. Stated another way, the delta file becomes a “composite” string (or stream) over time, with highly compact encoding. In this example, after Version 3 has been generated, the delta file is a composite of the two (2) second stage delta strings, viz., +400200xx|R200200300yyz.
As can also be seen, a sparse index associated with a given version is a byte range description of the particular version of the file (i.e., the version that exists at a given point in time). Stated another way, the delta file and the associated sparse index enable the system to determine byte level contents of each version of the file at any given point-in-time. Thus, the encoding techniques described by the present invention facilitate any point-in-time “on-demand” recovery of the data.
The above-described examples show one delta string being produced for each version change. This is not a limitation. In practice, typically a new file update may result in one or more delta strings being generated. In addition, the number of first stage delta strings need not be the same as the number of second stage delta strings for a given update.
Thus, in the DMS cluster (and in this example), Version 2 generates a first stage delta string of +400 200 “xx,” which indicates that the data “xx” is of length 400 and is inserted at a given offset 200. The second stage delta string has a similar value, as typically an “insertion” does not reduce the size of the first stage delta string. As can be seen, the Version 2 sparse index corresponds to the Version 2 content (with “+” being one byte, and both 400 and 200 being 4 byte numbers). In particular, the Version 2 sparse index identifies that the first four character positions (byte range 0-399) of the Version 2 content are found in the original binary file (bfile); that the next 2 characters (byte range 400-599) of the Version 2 content are found in the delta file for this version at offset “9” (in this encoding “+” is represented as 1 byte and both “400” and “200” are represented as 4 byte numbers so that +400200 represents “9”); and that the final four character positions (byte range 600-999) of the Version 2 content are found in the original binary file (bfile) in the final four character positions of that file. Thus, as can be seen, the sparse index provides byte level descriptions from which the actual data comprising the Version 2 content can be reconstructed.
With Version 3, the first stage delta string reflects a replace function R, in this case that the new data (aayyz) is of length 500 and is replacing old data (aaaa) of length 400 at a given offset (0) (at the front of the binary file). The second stage delta string is then generated by comparing the first stage delta string R 0 400 500 “aayyz” with the original binary string to create a further reduced string, in this case a string that reflects that new data (yyz) is of length 300 and is replacing old data (aa) of length 200 at a given offset (200). Once again, the Version 3 sparse index provides the byte range descriptions of the Version 3 content. Thus, the first two characters (byte range 0-199) are from the original binary file at the positions indicated, the next three characters (byte range 200-499) are identified from the composite delta file (dfile) at the identified offset, the next two characters (byte range 500-699) are identified from the composite delta file at the identified offset, and then the final characters (byte range 700-1099) are identified from the original binary file as indicated.
As can be seen, in the host server a large portion of a file or a database may be modified, while at the DMS typically only a small amount of data is written to the storage. Moreover, as noted above, the data written to storage is typically just new metadata (including the sparse index for the version) and a new sequence (the one or more second-stage delta strings) appended to the dfile. To generate a new sparse index, only the last version of the sparse index needs to be retrieved and modified according to the semantics indicated in the new second-stage delta string(s). As compared to the host server, however, only a very small amount of storage and I/O bandwidth is used in the DMS.
Of course, the above-identified description is merely representative of the desired encoding properties provided by the second stage delta string and the associated sparse index. The specific examples are not meant to limit the present invention in any way.
The following table illustrates several additional examples of how the delta string syntax is used to generate representative first stage and second stage delta strings:
Once configured, the Delta1-handle continues to accept requests from the host driver. This is step 714. At step 716, a test is performed to determine the request type. If the request type is a WRITE request (a request to accumulate updated data) in the form of WRITE (offset, length, data), the routine branches to step 718. At this step, any data changes to the version are accumulated. If the request type is SIGNATURE request (a request to accumulate the signatures from the last version for delta computation in the form of SIGNATURE (blockOffset, blockRange, arrayofSignatures), the routine branches to step 720. At this step, the signatures of the previous data version that are relevant to the changes are accumulated. As will be described in more detail below, the host driver can determine if the needed signatures are available with the delta1-handle; if not, the host driver preferably obtains the signature from the DMS core. Once changes are completed (typically upon a checkpoint event) and the relevant signatures of the last version are acquired, the routine performs a COMPUTE function at step 722 using the data changes and the signatures as needed. This generates the first stage delta string. Once the first-stage delta strings are generated, the host driver can forward those delta strings to the DMS core and terminate the process, which is indicated by step 724.
If the outcome of the test at step 820 indicates that the event type is XDMP, the routine performs a test at step 838 to determine if the response corresponds to a request for signatures and if signature(s) are available. If yes, the routine branches to step 839 to put the signature(s) into the Delta1-handle (as in step 718 in
The use of forward delta encoding in combination with the sparse index provide significant advantages of the prior art. In particular, the multi-stage delta reduction as has been described significantly reduces the amount of storage required for storing data history. As is known, a forward delta management system maintains an initial baseline of the file, as well as a list of deltas that come after the baseline. In a forward delta management system, deltas are always appended at a delta document sequentially. The advantage of such system is that, as deltas arrived, the system needs only to append the deltas to the tail end of a delta document. As applied in the present invention, a given DMS node can simply append the delta strings to a delta file as it provides a data protection service. The DMS system can also transfer the delta strings to other DMS nodes or external hosts to provide a data replication service. When a user tries to access a file (or when a host needs to recover its lost data at a specific point-in-time or the most current point-in-time), the DMS node must, at runtime, take the baseline of the file and then apply the necessary delta strings “on the fly” to generate the requested point-in-time data. As mentioned above, this process is quite difficult to accomplish in an efficient manner in the prior art because read latency is very high as the number of delta strings increases. The present invention, however, solves this problem by providing the associated sparse index. By using the sparse index, the DMS can identify the exact contents of a particular version of the file at any given point-in-time in a computationally-efficient manner. Moreover, because the invention uses significantly lower I/O bandwidth at the DMS (as compared to the I/O bandwidth requirements of the corresponding update at the host server), many servers can be protected (by DMS) concurrently. The present invention also allows the DMS nodes to perform data replication over local or wide area networks with minimal bandwidth.
Each of the first and second stage data reduction modules as described above are conveniently implemented as computer software, i.e., a set of program instructions and associated data structures. This is not a requirement of the invention, as the functionality described above (or any part thereof) may be implemented in firmware, in general purpose or special-purpose hardware, or in combinations of hardware and software.
While a multi-stage data reduction approach is desirable, this is not a limitation of the present invention. In an alternative embodiment, there are no second-stage delta strings, in which case only the first-stage delta strings are maintained with the sparse index. In yet another alternative embodiment, changes collected from the host are sent to the DMS without performing first-stage delta reduction, in which case the changes are compared against the previous data version in the DMS to generate delta strings and the sparse index.
Yet other variants are also within the scope of the present invention. Thus, while the delta file (dfile) has been described as a concatenation of the second stage delta strings (i.e., a stream), this is not necessarily a requirement of the invention. The second stage delta strings may be stored separately or otherwise disassociated from one another if desired.
While the present invention has been described in the context of a method or process, the present invention also relates to apparatus for performing the operations herein. As described above, this apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including an optical disk, a CD-ROM, a magnetic-optical disk, a read-only memory (ROM), a random access memory (RAM), a magnetic or optical card, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
While the above written description also describes a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary, as alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, or the like. References in the specification to a given embodiment indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic.
While given components of the system have been described separately, one of ordinary skill also will appreciate that some of the functions may be combined or shared in given instructions, program sequences, code portions, and the like.
One of ordinary skill will also appreciate that the techniques of the present invention can be implemented in any data storage device or system, or across sets of such devices or systems. More generally, the present invention can be applied on a file system, on a raw volume, or with respect to any storage devices in which any logical or physical data structures are used.
Claims
1. A method for capturing and storing a data history of a file to enable reconstruction of versions of the file, comprising:
- storing a copy of a first version of a file;
- comparing with one or more computer processors, a second version of the file to the first version of the file, or the copy of the first version of the file, to generate one or more delta strings associated with the second version of the file;
- generating a byte range index that refers to bytes in the copy of the first version of the file and to bytes in the one or more delta strings associated with the second version of the file;
- wherein the byte range index references entire contents of the second version of the file;
- storing the byte range index; and
- using the byte range index to enable reconstruction of the second version of the file without having to apply to the copy of the first version of the file the one or more delta strings associated with the second version of the file.
2. The method of claim 1 further comprising:
- comparing an nth version of the file to an (n−1)th version of the file, or a copy of the (n−1)th version of the file, to generate one or more delta strings associated with the nth version of the file; and
- generating an nth byte range index that refers to bytes in the copy of the first version of the file, to bytes in the one or more delta strings associated with any of the second to (n−1)th versions of the file, and to bytes in the one or more delta strings associated with the nth version of the file.
3. The method of claim 2 further comprising:
- using the nth byte range index to enable reconstruction of the nth version of the file without having to apply to the copy of the first version of the file the one or more delta strings associated with any of the second to nth versions of the file.
4. The method of claim 1 wherein comparing utilizes a signature-based algorithm that performs checksum-based delta extraction to generate the one or more delta strings.
5. The method of claim 1 wherein comparing comprises a first stage and a second stage, wherein the first stage generates a first, coarse data reduction and the second stage generates a second, more fine-grained data reduction.
6. The method of claim 5 wherein, during the first stage, a digital signature of a block of an nth version of the file is compared to a digital signature of a corresponding block of an (n−1)th version of the file to generate a delta string.
7. The method of claim 6 wherein, during the second stage, a digital signature of the block of the nth version is compared to a digital signature of the corresponding block of the (n−1)th version of the file to generate another delta string, where the block of the nth version that is processed during the second stage is smaller than the block of the nth version that is processed in the first stage.
8. The method of claim 1 wherein the byte range index comprises a set of one or more entries, each entry comprising a start byte-end byte sequence and a reference value that represents contents of the file within the start byte-end byte sequence.
9. The method of claim 8 wherein the reference refers to a data range of the copy of the first version of the file or any delta string of any version of the file.
10. The method of claim 8 wherein the reference refers to a content file associated with the file, an offset in the content file, and a length or end byte of the content file.
11. The method of claim 8 wherein the reference refers to a physical storage location.
12. The method of claim 1 wherein a given version of the file represents an application checkpoint, a user file save, or a file close.
13. An apparatus for capturing and storing a data history of a file to enable reconstruction of versions of the file, comprising:
- computer hardware comprising one or more computer processors configured to: store a copy of a first version of a file; compare a second version of the file to the first version of the file, or the copy of the first version of the file, to generate one or more delta strings associated with the second version of the file; generate a byte range index that refers to bytes in the copy of the first version of the file and to bytes in the one or more delta strings associated with the second version of the file; wherein the byte range index references entire contents of the second version of the file; store the byte range index; and use the byte range index to enable reconstruction of the second version of the file without having to apply to the copy of the first version of the file the one or more delta strings associated with the second version of the file.
14. The apparatus of claim 13 wherein the computer hardware is further configured to:
- compare an nth version of the file to an (n−1)th version of the file, or a copy of the (n−1)th version of the file, to generate one or more delta strings associated with the nth version of the file; and
- generate an nth byte range index that refers to bytes in the copy of the first version of the file, to bytes in the one or more delta strings associated with any of the second to (n−1)th versions of the file, and to bytes in the one or more delta strings associated with the nth version of the file.
15. The apparatus of claim 14 wherein the computer hardware is further configured to:
- use the nth byte range index to enable reconstruction of the nth version of the file without having to apply to the copy of the first version of the file the one or more delta strings associated with any of the second to nth versions of the file.
16. The apparatus of claim 13 wherein comparing utilizes a signature-based algorithm that performs checksum-based delta extraction to generate the one or more delta strings.
17. The apparatus of claim 13 wherein comparing comprises a first stage and a second stage, wherein the first stage generates a first, coarse data reduction and the second stage generates a second, more fine-grained data reduction.
18. The apparatus of claim 17 wherein, during the first stage, a digital signature of a block of an nth version of the file is compared to a digital signature of a corresponding block of an (n−1)th version of the file to generate a delta string.
19. The apparatus of claim 18 wherein, during the second stage, a digital signature of the block of the nth version is compared to a digital signature of the corresponding block of the (n−1)th version of the file to generate another delta string, where the block of the nth version that is processed during the second stage is smaller than the block of the nth version that is processed in the first stage.
20. The apparatus of claim 13 wherein the byte range index comprises a set of one or more entries, each entry comprising a start byte-end byte sequence and a reference value that represents contents of the file within the start byte-end byte sequence.
3555184 | January 1971 | Townley |
3555195 | January 1971 | Rester et al. |
3555204 | January 1971 | Braun |
3555251 | January 1971 | Shavit |
3648250 | March 1972 | Low et al. |
4162536 | July 24, 1979 | Morley |
4402045 | August 30, 1983 | Krol |
4415792 | November 15, 1983 | Jordan |
4450556 | May 22, 1984 | Boleda et al. |
4451108 | May 29, 1984 | Skidmore |
4455483 | June 19, 1984 | Schonhuber |
4502082 | February 26, 1985 | Ragle et al. |
4512020 | April 16, 1985 | Krol et al. |
4796260 | January 3, 1989 | Schilling et al. |
4882737 | November 21, 1989 | Dzung |
4916450 | April 10, 1990 | Davis |
4972474 | November 20, 1990 | Sabin |
5005197 | April 2, 1991 | Parsons et al. |
5148479 | September 15, 1992 | Bird et al. |
5177796 | January 5, 1993 | Feig et al. |
5224212 | June 29, 1993 | Rosenthal et al. |
5274508 | December 28, 1993 | Tan et al. |
5280584 | January 18, 1994 | Caesar et al. |
5287504 | February 15, 1994 | Carpenter et al. |
5303393 | April 12, 1994 | Noreen et al. |
5305326 | April 19, 1994 | Solomon et al. |
5311197 | May 10, 1994 | Sorden et al. |
5319395 | June 7, 1994 | Larky et al. |
5321699 | June 14, 1994 | Endoh et al. |
5363371 | November 8, 1994 | Roy et al. |
5365516 | November 15, 1994 | Jandrell |
5373372 | December 13, 1994 | Loewen |
5377102 | December 27, 1994 | Nishiishigaki |
5382508 | January 17, 1995 | Ikenoue |
5386422 | January 31, 1995 | Endoh et al. |
5387994 | February 7, 1995 | McCormack et al. |
5388074 | February 7, 1995 | Buckenmaier |
5392209 | February 21, 1995 | Eason et al. |
5396600 | March 7, 1995 | Thompson et al. |
5416831 | May 16, 1995 | Chewning, III et al. |
5424778 | June 13, 1995 | Sugiyama et al. |
5430830 | July 4, 1995 | Frank et al. |
5440686 | August 8, 1995 | Dahman et al. |
5469444 | November 21, 1995 | Endoh et al. |
5477492 | December 19, 1995 | Ohsaki et al. |
5479654 | December 26, 1995 | Squibb |
5481531 | January 2, 1996 | Yamamuro |
5499512 | March 19, 1996 | Jurewicz et al. |
5502491 | March 26, 1996 | Sugiyama et al. |
5506965 | April 9, 1996 | Naoe |
5507024 | April 9, 1996 | Richards, Jr. |
5511212 | April 23, 1996 | Rockoff |
5526357 | June 11, 1996 | Jandrell |
5537945 | July 23, 1996 | Sugihara et al. |
5560033 | September 24, 1996 | Doherty et al. |
5561671 | October 1, 1996 | Akiyama |
5583975 | December 10, 1996 | Naka et al. |
5602638 | February 11, 1997 | Boulware |
5606601 | February 25, 1997 | Witzman et al. |
5640159 | June 17, 1997 | Furlan et al. |
5644763 | July 1, 1997 | Roy |
5651129 | July 22, 1997 | Yokote et al. |
5657398 | August 12, 1997 | Guilak |
5678042 | October 14, 1997 | Pisello et al. |
5684536 | November 4, 1997 | Sugiyama et al. |
5684693 | November 4, 1997 | Li |
5684774 | November 4, 1997 | Yamamuro |
5724241 | March 3, 1998 | Wood et al. |
5729743 | March 17, 1998 | Squibb |
5737399 | April 7, 1998 | Witzman et al. |
5742509 | April 21, 1998 | Goldberg et al. |
5742915 | April 21, 1998 | Stafford |
5754772 | May 19, 1998 | Leaf |
5764691 | June 9, 1998 | Hennedy et al. |
5768159 | June 16, 1998 | Belkadi et al. |
5778370 | July 7, 1998 | Emerson |
5781612 | July 14, 1998 | Choi et al. |
5784366 | July 21, 1998 | Apelewicz |
5794252 | August 11, 1998 | Bailey et al. |
5805155 | September 8, 1998 | Allibhoy et al. |
5812130 | September 22, 1998 | Van Huben et al. |
RE35920 | October 13, 1998 | Sorden et al. |
5819020 | October 6, 1998 | Beeler, Jr. |
5822749 | October 13, 1998 | Agarwal |
5826265 | October 20, 1998 | Van Huben et al. |
5831903 | November 3, 1998 | Ohuchi et al. |
5841717 | November 24, 1998 | Yamaguchi |
5841771 | November 24, 1998 | Irwin et al. |
5848072 | December 8, 1998 | Prill et al. |
5854834 | December 29, 1998 | Gottlieb et al. |
5862136 | January 19, 1999 | Irwin |
5864875 | January 26, 1999 | Van Huben et al. |
5877742 | March 2, 1999 | Klink |
5878408 | March 2, 1999 | Van Huben et al. |
5893119 | April 6, 1999 | Squibb |
5894494 | April 13, 1999 | Davidovici |
5909435 | June 1, 1999 | Apelewicz |
5915251 | June 22, 1999 | Burrows et al. |
5917429 | June 29, 1999 | Otis et al. |
5918248 | June 29, 1999 | Newell et al. |
5920867 | July 6, 1999 | Van Huben et al. |
5920873 | July 6, 1999 | Van Huben et al. |
5928327 | July 27, 1999 | Wang et al. |
5930732 | July 27, 1999 | Domanik et al. |
5930762 | July 27, 1999 | Masch |
5931928 | August 3, 1999 | Brennan et al. |
5937168 | August 10, 1999 | Anderson et al. |
5940823 | August 17, 1999 | Schreiber et al. |
5950201 | September 7, 1999 | Van Huben et al. |
5953729 | September 14, 1999 | Cabrera et al. |
5958010 | September 28, 1999 | Agarwal et al. |
5966707 | October 12, 1999 | Van Huben et al. |
5974563 | October 26, 1999 | Beeler, Jr. |
5980096 | November 9, 1999 | Thalhammer-Reyero |
5999562 | December 7, 1999 | Hennedy et al. |
6005846 | December 21, 1999 | Best et al. |
6005860 | December 21, 1999 | Anderson et al. |
6031848 | February 29, 2000 | Brennan |
6035297 | March 7, 2000 | Van Huben et al. |
6047323 | April 4, 2000 | Krause |
6065018 | May 16, 2000 | Beier et al. |
6072185 | June 6, 2000 | Arai et al. |
6088693 | July 11, 2000 | Van Huben et al. |
6094654 | July 25, 2000 | Van Huben et al. |
6108318 | August 22, 2000 | Kolev et al. |
6108410 | August 22, 2000 | Reding et al. |
6154847 | November 28, 2000 | Schofield et al. |
6158019 | December 5, 2000 | Squibb |
6163856 | December 19, 2000 | Dion et al. |
6178121 | January 23, 2001 | Maruyama |
6181609 | January 30, 2001 | Muraoka |
6189016 | February 13, 2001 | Cabrera et al. |
6237122 | May 22, 2001 | Maki |
6243348 | June 5, 2001 | Goodberlet |
6249824 | June 19, 2001 | Henrichs |
6366926 | April 2, 2002 | Pohlmann et al. |
6366988 | April 2, 2002 | Skiba et al. |
6389427 | May 14, 2002 | Faulkner |
6393582 | May 21, 2002 | Klecka et al. |
6397242 | May 28, 2002 | Devine et al. |
6446136 | September 3, 2002 | Pohlmann et al. |
6460055 | October 1, 2002 | Midgley et al. |
6463565 | October 8, 2002 | Kelly et al. |
6487561 | November 26, 2002 | Ofek et al. |
6487581 | November 26, 2002 | Spence et al. |
6496944 | December 17, 2002 | Hsiao et al. |
6502133 | December 31, 2002 | Baulier et al. |
6519612 | February 11, 2003 | Howard et al. |
6526418 | February 25, 2003 | Midgley et al. |
6549916 | April 15, 2003 | Sedlar |
6611867 | August 26, 2003 | Bowman-Amuah |
6625623 | September 23, 2003 | Midgley et al. |
6629109 | September 30, 2003 | Koshisaka |
6640145 | October 28, 2003 | Hoffberg |
6670974 | December 30, 2003 | McKnight et al. |
RE38410 | January 27, 2004 | Hersch et al. |
6751753 | June 15, 2004 | Nguyen et al. |
6769074 | July 27, 2004 | Vaitzblit |
6779003 | August 17, 2004 | Midgley et al. |
6785786 | August 31, 2004 | Gold et al. |
6807550 | October 19, 2004 | Li et al. |
6816872 | November 9, 2004 | Squibb |
6823336 | November 23, 2004 | Srinivasan et al. |
6826711 | November 30, 2004 | Moulton et al. |
6836756 | December 28, 2004 | Gruber |
6839721 | January 4, 2005 | Schwols |
6839740 | January 4, 2005 | Kiselev |
6847984 | January 25, 2005 | Midgley et al. |
6907551 | June 14, 2005 | Katagiri et al. |
6941310 | September 6, 2005 | Ahad et al. |
6968464 | November 22, 2005 | Gardiner et al. |
6983286 | January 3, 2006 | Sinha et al. |
6993706 | January 31, 2006 | Cook |
7003688 | February 21, 2006 | Pittelkow et al. |
7028078 | April 11, 2006 | Sharma et al. |
7039663 | May 2, 2006 | Federwisch et al. |
7054913 | May 30, 2006 | Kiselev |
7069579 | June 27, 2006 | Delpuch |
7080081 | July 18, 2006 | Agarwal et al. |
7092396 | August 15, 2006 | Lee et al. |
7096392 | August 22, 2006 | Sim-Tang |
7200233 | April 3, 2007 | Keller et al. |
7206805 | April 17, 2007 | McLaughlin et al. |
7207224 | April 24, 2007 | Rutt et al. |
7272613 | September 18, 2007 | Sim et al. |
7290056 | October 30, 2007 | McLaughlin et al. |
7325159 | January 29, 2008 | Stager et al. |
7363549 | April 22, 2008 | Sim-Tang |
7519870 | April 14, 2009 | Sim-Tang |
7526541 | April 28, 2009 | Roese et al. |
7565661 | July 21, 2009 | Sim-Tang |
7680834 | March 16, 2010 | Sim-Tang |
7689602 | March 30, 2010 | Sim-Tang |
7788521 | August 31, 2010 | Sim-Tang |
7904913 | March 8, 2011 | Sim-Tang et al. |
7979404 | July 12, 2011 | Sim-Tang |
7979441 | July 12, 2011 | Sim-Tang |
8060889 | November 15, 2011 | Sim-Tang |
8094647 | January 10, 2012 | Elliott et al. |
8108429 | January 31, 2012 | Sim-Tang et al. |
8131723 | March 6, 2012 | Sim-Tang |
8151140 | April 3, 2012 | Sim-Tang |
8195628 | June 5, 2012 | Sim-Tang |
8200706 | June 12, 2012 | Sim-Tang |
20010029520 | October 11, 2001 | Miyazaki et al. |
20010043522 | November 22, 2001 | Park |
20010056362 | December 27, 2001 | Hanagan et al. |
20020022982 | February 21, 2002 | Cooperstone et al. |
20020064149 | May 30, 2002 | Elliott et al. |
20020091722 | July 11, 2002 | Gupta et al. |
20020107860 | August 8, 2002 | Gobeille et al. |
20020144177 | October 3, 2002 | Kondo et al. |
20020147807 | October 10, 2002 | Raguseo |
20020172222 | November 21, 2002 | Ullmann et al. |
20020178397 | November 28, 2002 | Ueno et al. |
20020199152 | December 26, 2002 | Garney et al. |
20030004947 | January 2, 2003 | Coverston |
20030009552 | January 9, 2003 | Benfield et al. |
20030051026 | March 13, 2003 | Carter et al. |
20030088372 | May 8, 2003 | Caulfield |
20030117916 | June 26, 2003 | Makela et al. |
20030200098 | October 23, 2003 | Geipel et al. |
20030204515 | October 30, 2003 | Shadmon et al. |
20030225825 | December 4, 2003 | Healey et al. |
20040010544 | January 15, 2004 | Slater et al. |
20040036716 | February 26, 2004 | Jordahl |
20040047354 | March 11, 2004 | Slater et al. |
20040080504 | April 29, 2004 | Salesky et al. |
20040098458 | May 20, 2004 | Husain et al. |
20040098717 | May 20, 2004 | Husain et al. |
20040098728 | May 20, 2004 | Husain et al. |
20040098729 | May 20, 2004 | Husain et al. |
20040117715 | June 17, 2004 | Ha et al. |
20040133487 | July 8, 2004 | Hanagan et al. |
20040193594 | September 30, 2004 | Moore et al. |
20040199486 | October 7, 2004 | Gopinath et al. |
20040250212 | December 9, 2004 | Fish |
20050001911 | January 6, 2005 | Suzuki |
20050021690 | January 27, 2005 | Peddada |
20050076066 | April 7, 2005 | Stakutis et al. |
20050166179 | July 28, 2005 | Vronay et al. |
20050240592 | October 27, 2005 | Mamou et al. |
20050251540 | November 10, 2005 | Sim-Tang |
20050262097 | November 24, 2005 | Sim-Tang et al. |
20050262188 | November 24, 2005 | Mamou et al. |
20050286440 | December 29, 2005 | Strutt et al. |
20060020586 | January 26, 2006 | Prompt et al. |
20060026220 | February 2, 2006 | Margolus |
20060050970 | March 9, 2006 | Gunatilake |
20060101384 | May 11, 2006 | Sim-Tang et al. |
20060130002 | June 15, 2006 | Hirayama et al. |
20060137024 | June 22, 2006 | Kim et al. |
20060236149 | October 19, 2006 | Nguyen et al. |
20060259820 | November 16, 2006 | Swoboda |
20060278004 | December 14, 2006 | Rutt et al. |
20070067278 | March 22, 2007 | Borodziewicz et al. |
20070094312 | April 26, 2007 | Sim-Tang |
20070168692 | July 19, 2007 | Quintiliano |
20070185922 | August 9, 2007 | Kapoor et al. |
20070214191 | September 13, 2007 | Chandrasekaran |
20100198788 | August 5, 2010 | Sim-Tang |
20110252004 | October 13, 2011 | Sim-Tang |
20110252432 | October 13, 2011 | Sim-Tang et al. |
20120254659 | October 4, 2012 | Sim-Tang |
20120266019 | October 18, 2012 | Sim-Tang |
WO 98/19262 | May 1998 | WO |
WO 02/25443 | March 2002 | WO |
WO 03/060774 | July 2003 | WO |
- Housel et al., “WebExpress: A client/intercept based system for optimizing Web browsing in a wireless environment”, Mobile Networks and Applications, vol. 3, No. 4, pp. 419-431, 1998, Baltzer Science Publishers BV.
- Psounis, “Class-based Delta-encoding: A Scalable Scheme for Caching Dynamic Web Content”, Proceedings of the 22 nd International Conference on Distributed Computing Systems Workshops, 2002 IEEE).
- Reichenberger, “Delta Storage for Arbitrary Non-Text Files”, SCM '91 Proceedings of the 3rd international workshop on Software configuration management, pp. 144-152, ACM, 1991.
- Otlu, “A New Technique: Replace Algorithm to Retrieve a Version from a Repository Instead od Delta Application”, Thesis, The Graduate School of Natural and Applied Sciences, The Middle East Technical University, 2004.
- Kurtz et al., “Delta Compression Algorithms for Diverse Environments”.
- Psounis, “Class-based Delta-encoding: A Scalable Scheme for Caching Dynamic Web Content”, Proceedings of the 22 nd International Conference on Distributed Computing Systems Workshops, 2002 IEEE.
- U.S. Appl. No. 12/099,837, filed Apr. 9, 2008, Sim-Tang, Siew Yong.
- U.S. Appl. No. 13/242,916, filed Sep. 23, 2011, Sim-Tang, Siew Yong.
- U.S. Appl. No. 13/436,624, filed Mar. 30, 2012, Sim-Tang, Siew Yong.
- U.S. Appl. No. 13/489,751, filed Jun. 6, 2012, Sim-Tang, Siew Yong.
- Bloom, Burton H., Space/Time Trade-offs in Hash Coding with Allowable Errors, Communications of the ACM, vol. 13, No. 7, Jul. 1970, pp. 422-426.
- Housel, et al., “WebExpress: A client/intercept based system for optimizing Web browsing in a wireless environment”, Mobile Network and Applications, vol. 3, pp. 419-431, 1998, Baltzer Science Publishers.
- International Preliminary Report and Written Opinion for PCT/US05/15651 (Beausoliel, Rob) mailed Sep. 29, 2005 (13 pages).
- International Preliminary Report and Written Opinion for PCT/US05/15653 (Zhen, Li B.) mailed Nov. 30, 2006 (13 pages).
- International Preliminary Report and Written Opinion for PCT/US05/15662 (Vo, Tim) mailed Feb. 1, 2008 (11 pages).
- International Preliminary Report and Written Opinion for PCT/US05/32958 (Abel-Jalil, Neveen) mailed Jul. 8, 2008 (8 pages).
- International Preliminary Report and Written Opinion for PCT/US05/39709 (Thomson, William) mailed Oct. 16, 2007 (7 pages).
- Microsoft Windows Storage Server 2003, NSI Software, “Improving Business Continuity Using Windows Storage Server 2003”, White Paper, Oct. 2003, pp. 1-15.
- Plagemann, Thomas, Towards Middleware Services for Mobile ADHOC Network Applications, May 2003, IEEE (FTDCS'03), pp. 1-7.
- Psounis, “Class-based Delta-encoding: A Scalable Scheme for Caching Dynamic Web Content”, Proceedings of the 22nd International Conference on Distributed Computing Systems Workshops, 2002 IEEE (pp. 1-7).
- U.S. Appl. No. 13/752,020, Siew Yong Sim-Tang.
- U.S. Appl. No. 13/721,588, Siew Yong Sim-Tang.
- U.S. Appl. No. 13/714,829, Sim-Tang.
Type: Grant
Filed: Jun 1, 2012
Date of Patent: Feb 11, 2014
Patent Publication Number: 20120284238
Assignee: Dell Software Inc. (Aliso Viejo, CA)
Inventor: Siew Yong Sim-Tang (Saratoga, CA)
Primary Examiner: Michael Hicks
Application Number: 13/486,989
International Classification: G06F 7/00 (20060101); G06F 17/30 (20060101);