Abstract: A deduplicated storage system storing objects receives a search term. Storage includes metadata and segments into which the objects have been split and deduplicated. The metadata includes fingerprint sequences according to which the segments should be assembled. A partial match is found when a prefix of the term is found at an end of a segment or a suffix is found at a beginning of the segment. A fingerprint of the segment having the partial match is recorded. A first sequence of fingerprints associated with a first object is read to check whether any fingerprints in the first sequence have been recorded. When a fingerprint in the sequence has been recorded, a check of a next fingerprint in the sequence is made to see if it has been recorded as having the partial match. If the next fingerprint has been recorded, the first object is reported as having the term.

Abstract: Described is a system for a verification microservice engine for generating and deploying a controller module and one or more worker nodes to detect corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. The controller module builds local versions of slice recipe names based on metadata available object recipes. The controller module verifies the accuracy of the metadata based on whether the locally built slice recipes names match slice recipe names in object storage.

Abstract: A search term is received at deduplicated storage storing data segmented into segments. Segment fingerprints are generated and metadata maintained to allow reconstruction of the segmented data. The metadata includes fingerprint listings indicating sequences according to which the segments should be reconstructed. The segments are read to determine whether there are any matches of the search term. Matches are recorded in a results table. A first fingerprint listing associated with a first object is read. The results table is queried for fingerprints in the first fingerprint listing to determine whether the first object references any matches in the results table.

Abstract: Described is a system for a verification microservice engine for generating and deploying a controller module and one or more worker nodes to detect corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. The controller module builds local versions of slice recipe names based on metadata available object recipes. The controller module verifies the accuracy of the metadata based on whether the locally built slice recipes names match slice recipe names in object storage.

Abstract: Described is a system for a providing a service (or microservice) for performing deduplication for an object storage. The service (or microservice) may be source-agnostic in that it may receive data from multiple types of source systems by providing a uniform set of functions for deduplicating and writing the data to a destination object storage. The set of functions encapsulate a previously dispersed set of functionality provided by various components. Accordingly, the service provides a single scalable and stateless component for performing deduplication. For example, the service (e.g. deduplication service) may receive object related information and perform a filtering to accelerate network transfers. Accordingly, the service provides the ability to only transfer and write data that does not already exist on the object storage.

Abstract: Techniques are provided for inline deduplication based on a number of physical blocks having common fingerprints among multiple entries of a buffer cache. One method comprises storing input/output operations in a first cache comprising a plurality of entries each corresponding to a physical storage entity comprising a plurality of physical blocks. A given entry is maintained in the first cache based on a first number of physical blocks of the given entry having a duplicate fingerprint with at least one physical block of another entry in the first cache. A second number can be determined of the physical blocks of each entry having a fingerprint in a second cache, and a first ratio is determined for two entries in the first cache using the second number and the first number. A comparison of the first ratios can be performed to sort and possibly evict entries in the first cache based on the comparison.

Abstract: A controller is provided to manage searching a storage system. A search query including a search term is received. The controller generates a set of tasks associated with conducting a search of the storage system for any objects in the storage system having the search term. These tasks are pushed onto a message queue. A set of workers are allocated to retrieve and process the tasks. A worker processes a task in parallel with another worker processing another task.

Abstract: A search term is received at deduplicated storage storing data segmented into segments. Segment fingerprints are generated and metadata maintained to allow reconstruction of the segmented data. The metadata includes fingerprint listings indicating sequences according to which the segments should be reconstructed. The segments are read to determine whether there are any matches of the search term. Matches are recorded in a results table. A first fingerprint listing associated with a first object is read. The results table is queried for fingerprints in the first fingerprint listing to determine whether the first object references any matches in the results table.

Abstract: Techniques are provided for inline deduplication based on a number of physical blocks having common fingerprints among multiple entries of a buffer cache. One method comprises storing input/output operations in a first cache comprising a plurality of entries each corresponding to a physical storage entity comprising a plurality of physical blocks. A given entry is maintained in the first cache based on a first number of physical blocks of the given entry having a duplicate fingerprint with at least one physical block of another entry in the first cache. A second number is determined of the physical blocks of each entry having a fingerprint in a second cache, and a first ratio is determined for two entries in the first cache using the second number and the first number. A comparison of the first ratios can be performed to sort and possibly evict entries in the first cache based on the comparison.

Abstract: Systems and methods for cleaning a storage system. A deduplicated storage system is cleaned by identifying structures that include dead or unreferenced segments. This includes processing recipes to identify the segments that are no longer part of a live object recipe. Then, the dead segments are removed. This is accomplished by copying forward the live segments and then deleting, as a whole, the structure that included the dead segments.

Abstract: A technique for transferring data over a network leverages a standard NAS (Network Attached Storage) protocol to augment its inherent file-copying ability with fingerprint matching, enabling the NAS protocol to limit its data copying over the network to unique data segments while avoiding copying of redundant data segments.

Abstract: Techniques for balancing data compression and read performance of data chunks of a storage system are described herein. According to one embodiment, similar data chunks are identified based on sketches of a plurality of data chunks stored in the storage system. A first portion of the similar data chunks as a first group is associated with a first storage area. The first storage area is associated with one or more data chunks that are dissimilar to the first group but are likely accessed together. The first group of the similar data chunks and its associated dissimilar data chunks are compressed and stored in the first storage area.

Abstract: A method, computer program product, and computing system for performing an entropy analysis on each of a plurality of candidate data chunks associated with a potential candidate to generate a plurality of candidate data chunk entropies; performing an entropy analysis on each of a plurality of target data chunks associated with a potential target to generate a plurality of target data chunk entropies; identifying a candidate data chunk entropy limit, chosen from the plurality of candidate data chunk entropies, and a target data chunk entropy limit, chosen from the plurality of candidate data chunk entropies; and comparing a specific candidate data chunk associated with the candidate data chunk entropy limit to a specific target data chunk associated with the target data chunk entropy limit to determine if the specific candidate data chunk and the specific target data chunk are identical.

Abstract: One example method includes receiving a write request that includes a data structure version to be written, wherein the data structure version is associated with a unique identifier, storing the data structure version in association with the unique identifier, receiving a read request for a most recent version of the data structure and, when the stored data structure version is not the most recent version of the data structure, examining respective unique identifiers of each of a group of other stored data structure versions to determine which stored data structure version is the most recent. Finally, the example method includes returning the most recent data structure version, notwithstanding that one or more other data structure versions existed at the time that the read request was received.

Abstract: Use of a survival queue to manage a container-based flash cache is disclosed. In various embodiments, a corresponding survival time is associated with each of a plurality of containers stored in a flash cache, each container comprising a plurality of data blocks. The survival time may be determined based at least in part on a calculated proportion of relatively recently accessed data blocks associated with the container is associated with the container. A container to evict from the flash cache is selected based at least in part on a determination that the corresponding survival time of the selected container has expired.

Abstract: Systems and methods for cleaning a storage system. A deduplicated storage system is cleaned by identifying structures that include dead or unreferenced segments. This includes processing recipes to identify the segments that are no longer part of a live object recipe. Then, the dead segments are removed. This is accomplished by copying forward the live segments and then deleting, as a whole, the structure that included the dead segments.

Abstract: One example method includes receiving a write request that includes a data structure version to be written, wherein the data structure version is associated with a unique identifier, storing the data structure version in association with the unique identifier, receiving a read request for a most recent version of the data structure and, when the stored data structure version is not the most recent version of the data structure, examining respective unique identifiers of each of a group of other stored data structure versions to determine which stored data structure version is the most recent. Finally, the example method includes returning the most recent data structure version, notwithstanding that one or more other data structure versions existed at the time that the read request was received.

Abstract: A deduplicated microservices-based storage system and method running in a cloud includes frontend and backend microservice nodes that process requests from a plurality of clients and interface with a master node and one or more worker nodes connected as a network for deduplicating, storing and retrieving data from object storage. A custom metric compares demand on system resources to thresholds and dynamically scales resources to add or eliminate resources according to demand. The master node partitions the range of values identifying data similarity groups across the worker nodes, reassigns ranges of values using consistent hashing as worker nodes are scaled, and maintains a mapping of ranges of values to worker IP addresses.

Abstract: A controller at a source site generates a set of tasks associated with a replication job. Each task includes one or more of copying an object from the source to destination site, or deleting an object from the destination site. The tasks are placed onto a message queue at the source site. Source worker nodes at the source site retrieve the tasks from the source site message queue for processing in conjunction with destination worker nodes at the destination site. A destination worker node, upon receiving a task from a source worker nodes, places the task onto a message queue at the destination site for retrieval by a backend worker node that handles writing to an object store at the destination site.

Abstract: A method, computer program product, and computing system for processing a data portion to divide the data portion into a plurality of data chunks; performing an entropy analysis on each of the plurality of data chunks to generate a plurality of data chunk entropies; and determining an average data chunk entropy from the plurality of data chunk entropies.