Abstract: A disclosed method for managing a RAID (redundant array of independent devices) file system memory comprises coalescing small read and write chunks into reasonably sized coalesced chunks up to 1 MB without any small size I/Os. The method also includes creating a new mapping layer configured to map the small size chunks in the coalesced chunks. The method additionally comprises flushing the coalesced chunks by allocating large chunks based on a chunk size coalescence or on a check point hit in terms of a SPA Sync. The method may also include applying a dynamic parity algorithm on the coalesced chunks to generate parities and data fragmented columns in reasonably big sized chunks. The method may yet include creating an appropriate coalesced chunk header to maintain the new mapping layer. The method may still comprise generating a chunked block pointer and a regular block pointer configured to co-exist in dual process threads.

Abstract: A process for reconstructing data stored on a failed storage drive in a RAID storage system involves performing a resilvering procedure on a first portion of the data for reconstruction and performing a recompaction procedure on the remaining portion of the data for reconstruction. Because the resilvering procedure rebuilds its data only on the replacement storage drive, but the recompaction procedure rebuilds its data on one or more, likely multiple, non-failed storage drives, the additional bandwidth provided by the non-failed drives is utilized to increase the speed of the overall data reconstruction. Determining how much of the data storage chunk processing to distribute to the resilvering and to the recompaction procedures may be based on the respective write workloads of the replacement and non-failed drives, as well as on the percentage of free storage space available from each of the chunks.

Abstract: A system and process for recompacting digital storage space involves continuously maintaining a first log of free storage space available from multiple storage regions of a storage system such as a RAID system, and based on the first log, maintaining a second log file including a bitmap identifying the free storage space available from a given storage chunk corresponding to the storage regions. Based on the bitmaps, distributions corresponding to the storage regions are generated, where the distributions represent the percentage of free space available from each chunk, and a corresponding weight is associated with each storage region. The storage region weights may then be sorted and stored in RAM, for use in quickly identifying a particular storage region that includes the maximum amount of free space available, for recompaction.

Abstract: Techniques and mechanisms described herein facilitate the replication of data between storage nodes. According to various embodiments, a request to provide a data chunk to a target storage node may be received at a source data storage node. A reference data chunk may be identified based on fingerprint information associated with the requested data chunk. The reference data chunk may be stored on the target storage node. The reference data chunk and the requested data chunk may each include a first data portion. Data chunk reconstruction information may be transmitted from the source data storage node to the target data storage node. The data chunk reconstruction information may identify the reference data chunk. The data chunk reconstruction information may include data difference information for constructing the requested data chunk at the target data storage node based on the reference data chunk.

Abstract: A disclosed method for managing a RAID (redundant array of independent devices) file system memory comprises coalescing small read and write chunks into reasonably sized coalesced chunks up to 1 MB without any small size I/Os. The method also includes creating a new mapping layer configured to map the small size chunks in the coalesced chunks. The method additionally comprises flushing the coalesced chunks by allocating large chunks based on a chunk size coalescence or on a check point hit in terms of a SPA Sync. The method may also include applying a dynamic parity algorithm on the coalesced chunks to generate parities and data fragmented columns in reasonably big sized chunks. The method may yet include creating an appropriate coalesced chunk header to maintain the new mapping layer. The method may still comprise generating a chunked block pointer and a regular block pointer configured to co-exist in dual process threads.

Abstract: A system and process for recompacting digital storage space involves continuously maintaining a first log of free storage space available from multiple storage regions of a storage system such as a RAID system, and based on the first log, maintaining a second log file including a bitmap identifying the free storage space available from a given storage chunk corresponding to the storage regions. Based on the bitmaps, distributions corresponding to the storage regions are generated, where the distributions represent the percentage of free space available from each chunk, and a corresponding weight is associated with each storage region. The storage region weights may then be sorted and stored in RAM, for use in quickly identifying a particular storage region that includes the maximum amount of free space available, for recompaction.

Abstract: A process for reconstructing data stored on a failed storage drive in a RAID storage system involves performing a resilvering procedure on a first portion of the data for reconstruction and performing a recompaction procedure on the remaining portion of the data for reconstruction. Because the resilvering procedure rebuilds its data only on the replacement storage drive, but the recompaction procedure rebuilds its data on one or more, likely multiple, non-failed storage drives, the additional bandwidth provided by the non-failed drives is utilized to increase the speed of the overall data reconstruction. Determining how much of the data storage chunk processing to distribute to the resilvering and to the recompaction procedures may be based on the respective write workloads of the replacement and non-failed drives, as well as on the percentage of free storage space available from each of the chunks.

Abstract: Techniques and mechanisms described herein facilitate the replication of data between storage nodes. According to various embodiments, a request to provide a data chunk to a target storage node may be received at a source data storage node. A reference data chunk may be identified based on fingerprint information associated with the requested data chunk. The reference data chunk may be stored on the target storage node. The reference data chunk and the requested data chunk may each include a first data portion. Data chunk reconstruction information may be transmitted from the source data storage node to the target data storage node. The data chunk reconstruction information may identify the reference data chunk. The data chunk reconstruction information may include data difference information for constructing the requested data chunk at the target data storage node based on the reference data chunk.

Abstract: Techniques and mechanisms described herein facilitate the performance of duplicate data block instruction identification. According to various embodiments, a data block update operation message may be received at a communications interface in a secondary storage node. The secondary storage node may be configured to store secondary data mirroring primary data stored on a primary storage node. The primary data and the secondary data may each include a respective plurality of data blocks. The data block update operation message may include a data block update instruction for updating a designated one of the plurality of secondary storage node data blocks. The data block update operation message may include a primary storage node data block sequence number designating an update operation status. When it is determined that the data block update instruction is not a duplicate, the data block update instruction may be performed.

Abstract: Mechanisms are provided for efficient resynchronization of replicated data. A hash value is generated for a chunk of data replicated from a source node to a target node. The chunk of data may be a file deduplicated and compressed at both a source node and a target node. A current sequence number is determined and a sequence number and hash tuple is maintained for the chunk of data at both the source node and the target node. Sequence numbers are modified whenever the data is modified. Current sequence numbers and sequence number and hash values in the sequence number hash tuples at the source node and the target node may be compared to determine whether data is still synchronized at a later point in time or whether data requires resynchronization.

Abstract: Mechanisms are provided for efficient replica cleanup during resynchronization. According to various embodiments, a plurality of deleted data segment ranges on a first storage node may be identified. The first storage node may be configured to store a plurality of data segments. Each of the plurality of data segments may have associated therewith a respective identifier. Each of the data segment ranges may designate one or more data segments that have been deleted from the first storage node. The plurality of deleted data segment ranges may be transmitted to a second storage node configured to mirror the plurality of data segments stored on the first storage node. The plurality of deleted data segment ranges may be capable of being used to identify one or more data segments to delete from the second storage node.

Abstract: Techniques and mechanisms described herein facilitate the replication of data between storage nodes. According to various embodiments, a request to provide a data chunk to a target storage node may be received at a source data storage node. A reference data chunk may be identified based on fingerprint information associated with the requested data chunk. The reference data chunk may be stored on the target storage node. The reference data chunk and the requested data chunk may each include a first data portion. Data chunk reconstruction information may be transmitted from the source data storage node to the target data storage node. The data chunk reconstruction information may identify the reference data chunk. The data chunk reconstruction information may include data difference information for constructing the requested data chunk at the target data storage node based on the reference data chunk.

Abstract: Techniques and mechanisms described herein facilitate the performance of duplicate data block instruction identification. According to various embodiments, a data block update operation message may be received at a communications interface in a secondary storage node. The secondary storage node may be configured to store secondary data mirroring primary data stored on a primary storage node. The primary data and the secondary data may each include a respective plurality of data blocks. The data block update operation message may include a data block update instruction for updating a designated one of the plurality of secondary storage node data blocks. The data block update operation message may include a primary storage node data block sequence number designating an update operation status. When it is determined that the data block update instruction is not a duplicate, the data block update instruction may be performed.

Abstract: Mechanisms are provided for efficient replica cleanup during resynchronization. According to various embodiments, a plurality of deleted data segment ranges on a first storage node may be identified. The first storage node may be configured to store a plurality of data segments. Each of the plurality of data segments may have associated therewith a respective identifier. Each of the data segment ranges may designate one or more data segments that have been deleted from the first storage node. The plurality of deleted data segment ranges may be transmitted to a second storage node configured to mirror the plurality of data segments stored on the first storage node. The plurality of deleted data segment ranges may be capable of being used to identify one or more data segments to delete from the second storage node.

Abstract: Mechanisms are provided for efficient resynchronization of replicated data. A hash value is generated for a chunk of data replicated from a source node to a target node. The chunk of data may be a file deduplicated and compressed at both a source node and a target node. A current sequence number is determined and a sequence number and hash tuple is maintained for the chunk of data at both the source node and the target node. Sequence numbers are modified whenever the data is modified. Current sequence numbers and sequence number and hash values in the sequence number hash tuples at the source node and the target node may be compared to determine whether data is still synchronized at a later point in time or whether data requires resynchronization.