Abstract: The technique introduced here involves using a block address and a corresponding generation number as a “fingerprint” to uniquely identify a sequence of data within a given storage domain. Each block address has an associated generation number which indicates the number of times that data at that block address has been modified. This technique can be employed, for example, to determine whether a given storage server already has the data, and to avoid sending the data to that storage server over a network if it already has the data. It can also be employed to maintain cache coherency among multiple storage nodes.

Abstract: A technique for routing data for deduplication in a storage server cluster includes computing, for each node in the cluster, a value collectively representative of the data stored on the node, such as a “geometric center” of the node. New or modified data is routed to the node which has stored data identical or most similar to the new or modified data, as determined based on those values. Each node stores a plurality of chunks of data, where each chunk includes multiple deduplication segments. A content hash is computed for each deduplication segment in each node, and a similarity hash is computed for each chunk from the content hashes of all segments in the chunk. A geometric center of a node is computed from the similarity hashes of the chunks stored in the node.

Abstract: Storage systems and methods to improve space saving from data compression by providing a plurality of compression processes, and optionally, one or more parameters for controlling operation of the compression processes and selecting from the plurality of compression processes and the parameters to satisfy resource limits, such as CPU usage and memory usage. In one embodiment, the methods takes into account the content-type, such as text file or video file, and select the compression process and parameters that provide the greatest space savings for that content type while also remaining within a defined resource-usage limit.

Abstract: The technique introduced here involves using a block address and a corresponding generation number as a “fingerprint” to uniquely identify a sequence of data within a given storage domain. Each block address has an associated generation number which indicates the number of times that data at that block address has been modified. This technique can be employed, for example, to maintain cache coherency among multiple storage nodes. It can also be employed to avoid sending the data to a network node over a network if it already has the data.

Abstract: A technique for routing data for improved deduplication in a storage server cluster includes computing, for each node in the cluster, a value collectively representative of the data stored on the node, such as a “geometric center” of the node. New or modified data is routed to the node which has stored data identical or most similar to the new or modified data, as determined based on those values. Each node stores a plurality of chunks of data, where each chunk includes multiple deduplication segments. A content hash is computed for each deduplication segment in each node, and a similarity hash is computed for each chunk from the content hashes of all segments in the chunk. A geometric center of a node is computed from the similarity hashes of the chunks stored in the node.

Abstract: A technique for routing data for deduplication in a storage server cluster includes computing, for each node in the cluster, a value collectively representative of the data stored on the node, such as a “geometric center” of the node. New or modified data is routed to the node which has stored data identical or most similar to the new or modified data, as determined based on those values. Each node stores a plurality of chunks of data, where each chunk includes multiple deduplication segments. A content hash is computed for each deduplication segment in each node, and a similarity hash is computed for each chunk from the content hashes of all segments in the chunk. A geometric center of a node is computed from the similarity hashes of the chunks stored in the node.

Abstract: A persistent cache is implemented in a flash memory that includes a journal section that stores metadata and a low frequency section and a high frequency section that store data entries. Writing new metadata to the persistent cache includes sequentially advancing to a next sector containing an invalid metadata entry, saving a working copy of the sector in RAM, writing metadata corresponding to one or more new data entries in the working copy, and overwriting the sector in the flash memory containing the invalid entry with the working copy. Writes to the low frequency and high frequency sections occur sequentially in the current locations of a low frequency section pointer and a high frequency section pointer, respectively. In a persistent cache, the reconstruction of a non-persistent cache utilizes the metadata entry that has the most recent timestamp.

Abstract: A technique for routing data for improved deduplication in a storage server cluster includes computing, for each node in the cluster, a value collectively representative of the data stored on the node, such as a “geometric center” of the node. New or modified data is routed to the node which has stored data identical or most similar to the new or modified data, as determined based on those values. Each node stores a plurality of chunks of data, where each chunk includes multiple deduplication segments. A content hash is computed for each deduplication segment in each node, and a similarity hash is computed for each chunk from the content hashes of all segments in the chunk. A geometric center of a node is computed from the similarity hashes of the chunks stored in the node.