Abstract: Example apparatus, methods, and computers control configurable, variable data reduction. One example method includes identifying data reduction controlling attributes in an object to be data reduced by a configurable variable data reducer. The attributes provide information upon which decisions concerning whether and/or how to data reduce the object can be based. The example method also includes controlling a configurable variable data reducer to selectively data reduce the object based, at least in part, on the data reduction controlling attributes. The control exercised can determine whether, where, when, and/or how data reduction will proceed.

Abstract: Example apparatus, methods, and computers support cloud-based de-duplication with transport layer transparency. One example apparatus includes a processor, a memory, and an interface to connect the processor, memory, and a set of logics. The set of logics includes a hardware identification logic for identifying networking hardware used by or available to the apparatus to interact with a cloud-based computing environment and a protocol identification logic for identifying a protocol used by or available to the apparatus to communicate messages for the cloud-based computing environment through the networking hardware. The set of logics also includes a de-duplication preparation logic for preparing a message to carry data to be de-duplicated and metadata concerning the data to be de-duplicated. The message is to be provided to the cloud-based computing environment through the networking hardware according to the protocol.

Abstract: Example apparatus, methods, and computers support adaptive, collaborative deduplication. One example method includes controlling a computer to compute a correlation between a processing state associated with collaborative data deduplication performed by the computer and an environmental state associated with a collaborative data deduplication environment with which the computer interacts. With the correlation in hand, the method then includes controlling the computer to adapt how the computer participates in collaborative data deduplication based, at least in part, on the correlation. The adaptations can include changing what processing is performed on the computer and what data is stored on the computer.

Abstract: Example apparatus, methods, and computers control establishing a shared parse scope between two computers that intend to be involved in a shared de-duplication action. One example method includes, upon determining that a first de-duplication logic and a second de-duplication logic are to participate in a shared de-duplication action for an object, controlling the first de-duplication logic to establish a shared parse scope with the second de-duplication logic. Establishing the shared parse scope may include negotiations between the computers, where the negotiations transfer dialect information. The dialect information may take the form of rules. The method may also include persisting the shared parse scope.

Abstract: Example apparatus, methods, and computers support data de-duplication indexing. One example apparatus includes a processor, a memory, and an interface to connect the processor, memory, and a set of logics. The set of logics includes an establishment logic to instantiate one-to-many de-duplication data structures, a manipulation logic to update the de-dupe data structure(s), a key logic to generate a key from a block of data to be de-duplicated, and a similarity logic to make a similarity determination for the block. The similarity determination identifies the block as a unique block, a duplicate block, or a block that meets a similarity threshold with respect to a stored de-duplicated block accessible through the dedupe data structure. The similarity determination involves comparing the block to be de-duplicated to a stored block available to the apparatus using a byte-by-byte approach, a hash approach, a delta hash approach and/or a sampling sequence approach.

Abstract: Example apparatus, methods, and computers participate in collaborative, distributed, data de-duplication. One example method includes initializing a layered parser in a first node in a collaborative distributed data de-duplication (CDDD) topology with a first set of de-duplication control parameters. After transmitting some information to another node in the CDDD topology, the method includes selectively reconfiguring the layered parser in response to feedback acquired from the second node in the CDDD topology. The feedback concerns the data provided by the layered parser.

Abstract: Example apparatus, methods, and computers control processing delta chunks with delta hashes. One example method includes computing a first hash for a chunk for which a duplicate determination is to be made. The first hash is suitable for making the duplicate chunk determination. The method also includes computing a delta hash for the chunk. The delta hash is suitable for making a delta chunk determination. The method controls a de-duplication logic to process the chunk as a duplicate upon determining that the first hash matches a stored first hash. The method controls the de-duplication logic to process the chunk as a delta chunk upon determining that the first hash does not match a stored first hash and that the delta hash matches a stored delta hash. Processing a chunk as a delta chunk may include storing a reference to a stored chunk and storing delta hash information.

Abstract: Example apparatus, methods, and computers perform sampling based data de-duplication. One example method controls a data de-duplication computer to compute a sampling sequence for a sub-block of data and to use the sampling sequence to locate a stored sub-block known to the data de-duplication computer. Upon finding a stored sub-block to compare to, the method includes controlling the data de-duplication computer to determine a degree of similarity (e.g., duplicate, very similar, somewhat similar, very dissimilar, completely dissimilar, x % similar) between the sub-block and the stored sub-block and to control whether and how the sub-block is stored and/or transmitted based on the degree of similarity. The degree of similarity can also control whether and how the data de-duplication computer updates a dedupe data structure(s) that stores information for finding groups of similarity sampling sequence related sub-blocks.

Abstract: Example apparatus, methods, and computers control establishing a shared parse scope between two computers that intend to be involved in a shared de-duplication action. One example method includes, upon determining that a first de-duplication logic and a second de-duplication logic are to participate in a shared de-duplication action for an object, controlling the first de-duplication logic to establish a shared parse scope with the second de-duplication logic. Establishing the shared parse scope may include negotiations between the computers, where the negotiations transfer dialect information. The dialect information may take the form of rules. The method may also include persisting the shared parse scope.

Abstract: Example apparatus, methods, data structures, and computers control indexing to facilitate duplicate determinations. One example method includes indexing, in a global index, a unique chunk processed by a data de-duplicator. Indexing the unique chunk in the global index can include updating an expedited data structure associated with the global index. The example method can also include selectively indexing, in a temporal index, a relationship chunk processed by the data de-duplicator. The relationship chunk is a chunk that is related to another chunk processed by the data de-duplicator by sequence, storage location, and/or similarity hash value. Indexing the relationship chunk in the temporal index can also include updating one or more expedited data structures associated with the temporal index. The expedited data structures and indexes can then be consulted to resolve a duplicate determination being made by a data reducer.

Abstract: Example apparatus, methods, and computers control configurable, variable data reduction. One example method includes identifying data reduction controlling attributes in an object to be data reduced by a configurable variable data reducer. The attributes provide information upon which decisions concerning whether and/or how to data reduce the object can be based. The example method also includes controlling a configurable variable data reducer to selectively data reduce the object based, at least in part, on the data reduction controlling attributes.

Abstract: Example apparatus, methods, and computers perform sampling based data de-duplication. One example method controls a data de-duplication computer to compute a sampling sequence for a sub-block of data and to use the sampling sequence to locate a stored sub-block known to the data de-duplication computer. Upon finding a stored sub-block to compare to, the method includes controlling the data de-duplication computer to determine a degree of similarity (e.g., duplicate, very similar, somewhat similar, very dissimilar, completely dissimilar, x % similar) between the sub-block and the stored sub-block and to control whether and how the sub-block is stored and/or transmitted based on the degree of similarity. The degree of similarity can also control whether and how the data de-duplication computer updates a dedupe data structure(s) that stores information for finding groups of similarity sampling sequence related sub-blocks.

Abstract: Example apparatus, methods, and computers support cloud-based de-duplication with transport layer transparency. One example apparatus includes a processor, a memory, and an interface to connect the processor, memory, and a set of logics. The set of logics includes a hardware identification logic for identifying networking hardware used by or available to the apparatus to interact with a cloud-based computing environment and a protocol identification logic for identifying a protocol used by or available to the apparatus to communicate messages for the cloud-based computing environment through the networking hardware. The set of logics also includes a de-duplication preparation logic for preparing a message to carry data to be de-duplicated and metadata concerning the data to be de-duplicated. The message is to be provided to the cloud-based computing environment through the networking hardware according to the protocol.

Abstract: Example apparatus, methods, and computers support data de-duplication indexing. One example apparatus includes a processor, a memory, and an interface to connect the processor, memory, and a set of logics. The set of logics includes an establishment logic to instantiate one-to-many de-duplication data structures, a manipulation logic to update the de-dupe data structure(s), a key logic to generate a key from a block of data to be de-duplicated, and a similarity logic to make a similarity determination for the block. The similarity determination identifies the block as a unique block, a duplicate block, or a block that meets a similarity threshold with respect to a stored de-duplicated block accessible through the dedupe data structure. The similarity determination involves comparing the block to be de-duplicated to a stored block available to the apparatus using a byte-by-byte approach, a hash approach, a delta hash approach and/or a sampling sequence approach.

Abstract: Example apparatus, methods, and computers support adaptive, collaborative de-duplication. One example method includes controlling a computer to compute a correlation between a processing state associated with collaborative data deduplication performed by the computer and an environmental state associated with a collaborative data deduplication environment with which the computer interacts. With the correlation in hand, the method then includes controlling the computer to adapt how the computer participates in collaborative data deduplication based, at least in part, on the correlation. The adaptations can include changing what processing is performed on the computer and what data is stored on the computer.

Abstract: Example apparatus, methods, and computers control establishing a shared parse scope between two computers that intend to be involved in a shared de-duplication action. One example method includes, upon determining that a first de-duplication logic and a second de-duplication logic are to participate in a shared de-duplication action for an object, controlling the first de-duplication logic to establish a shared parse scope with the second de-duplication logic. Establishing the shared parse scope may include negotiations between the computers, where the negotiations transfer dialect information. The dialect information may take the form of rules. The method may also include persisting the shared parse scope.

Abstract: Example apparatus, methods, and computers control configurable, variable data reduction. One example method includes identifying data reduction controlling attributes in an object to be data reduced by a configurable variable data reducer. The attributes provide information upon which decisions concerning whether and/or how to data reduce the object can be based. The example method also includes controlling a configurable variable data reducer to selectively data reduce the object based, at least in part, on the data reduction controlling attributes. The control exercised can determine whether, where, when, and/or how data reduction will proceed.

Abstract: Example apparatus, methods, and computers participate in collaborative, distributed, data de-duplication. One example method includes initializing a layered parser in a first node in a collaborative distributed data de-duplication (CDDD) topology with a first set of de-duplication control parameters. After transmitting some information to another node in the CDDD topology, the method includes selectively reconfiguring the layered parser in response to feedback acquired from the second node in the CDDD topology. The feedback concerns the data provided by the layered parser.

Abstract: Example apparatus, methods, and computers control establishing a shared parse scope between two computers that intend to be involved in a shared de-duplication action. One example method includes, upon determining that a first de-duplication logic and a second de-duplication logic are to participate in a shared de-duplication action for an object, controlling the first de-duplication logic to establish a shared parse scope with the second de-duplication logic. Establishing the shared parse scope may include negotiations between the computers, where the negotiations transfer dialect information. The dialect information may take the form of rules. The method may also include persisting the shared parse scope.

Abstract: Example apparatus, methods, and computers control processing delta chunks with delta hashes. One example method includes computing a first hash for a chunk for which a duplicate determination is to be made. The first hash is suitable for making the duplicate chunk determination. The method also includes computing a delta hash for the chunk. The delta hash is suitable for making a delta chunk determination. The method controls a de-duplication logic to process the chunk as a duplicate upon determining that the first hash matches a stored first hash. The method controls the de-duplication logic to process the chunk as a delta chunk upon determining that the first hash does not match a stored first hash and that the delta hash matches a stored delta hash. Processing a chunk as a delta chunk may include storing a reference to a stored chunk and storing delta hash information.