Abstract: A method and system may reconstruct data in a smart storage array where upon detection of data inconsistency, an application of the system is notified and affected data strips are rebuilt. When an initiator detects stripe corruption, the initiator may report the strip corruption to storage. The storage may lock the strip for I/O operations. Initiators may determine recovery scenarios for rebuilding the data strips and send the scenarios to storage. At storage, initiator replies may be collected, and a rebuild option with the highest votes may be automatically determined for a reconstruction operation. Once the reconstruction operation is completed, the rebuilt stripe may be unlocked and data operations may recommence.

Abstract: A mechanism is provided for dispersed location-based data storage. A request is received to write a data file to a referrer memory region in a set of memory regions. For each data chunk of the data file, responsive to a comparison of a hash value for the data chunk to other hash values for other stored data chunks referenced in the referrer memory region indicating that the data chunk fails to exist in the referrer memory region, responsive to the data chunk existing in another memory region in the set of memory regions, responsive to the memory region failing to be one of the predetermined number N of owner memory regions associated with the referrer memory region, and responsive to the predetermined number N of owner memory regions failing to have been met, a reference to the data chunk is stored in the referrer memory region.

Abstract: A mechanism is provided for dispersed location-based data storage. A request is received to write a data file to a referrer memory region in a set of memory regions. For each data chunk of the data file, responsive to a comparison of a hash value for the data chunk to other hash values for other stored data chunks referenced in the referrer memory region indicating that the data chunk fails to exist in the referrer memory region, responsive to the data chunk existing in another memory region in the set of memory regions, responsive to the memory region failing to be one of the predetermined number N of owner memory regions associated with the referrer memory region, and responsive to the predetermined number N of owner memory regions failing to have been met, a reference to the data chunk is stored in the referrer memory region.

Abstract: A method and system may reconstruct data in a smart storage array where upon detection of data inconsistency, an application of the system is notified and affected data strips are rebuilt. When an initiator detects stripe corruption, the initiator may report the strip corruption to storage. The storage may lock the strip for I/O operations. Initiators may determine recovery scenarios for rebuilding the data strips and send the scenarios to storage. At storage, initiator replies may be collected, and a rebuild option with the highest votes may be automatically determined for a reconstruction operation. Once the reconstruction operation is completed, the rebuilt stripe may be unlocked and data operations may recommence.

Abstract: A computer-implemented method according to one embodiment includes receiving data, storing chunks of the data in fixed size blocks in physical storage, and deduplicating a copy of the data. The storing includes splitting some of the data chunks into portions for filling the fixed size blocks. Metadata is stored with the data chunks for chaining the associated portions of the split data chunks together. The deduplicating including creating a pointer to a first of the data chunks. A computer program product for utilizing free space in physical storage, according to another embodiment includes a computer readable storage medium having program instructions embodied therewith. The computer readable storage medium is not a transitory signal per se. The program instructions are readable and/or executable by a computer to cause the computer to perform the foregoing method.

Abstract: A computer-implemented method according to one embodiment includes determining resource usage of at least a first module in a grid storage system having multiple modules and approximately equal resource usage across the multiple modules of the grid storage system. The computer-implemented method further includes determining a garbage collection cost in the grid storage system by stopping garbage collection in a second of the modules of the grid storage system, determining a resource usage in the second module upon stopping the garbage collection, and comparing the resource usage in the second module to the resource usage of the at least the first module. The method further includes adjusting an amount of garbage collection based on both the garbage collection cost and the resource usage.

Abstract: A computer-implemented method according to one embodiment includes determining resource usage of at least a first module in a grid storage system having multiple modules and approximately equal resource usage across the multiple modules of the grid storage system. The computer-implemented method further includes determining a garbage collection cost in the grid storage system by stopping garbage collection in a second of the modules of the grid storage system, determining a resource usage in the second module upon stopping the garbage collection, and comparing the resource usage in the second module to the resource usage of the at least the first module. The method further includes adjusting an amount of garbage collection based on both the garbage collection cost and the resource usage.

Abstract: Embodiments for efficient garbage collection in a data storage environment. In a storage system comprising multiple storage devices having respective sets of storage regions, at least one respective storage fragmentation threshold used to trigger a garbage collection operation is identified. The garbage collection operation is performed to reclaim data space in the storage system according to each of a block perspective and an area perspective. The block perspective performs the garbage collection operation on individual blocks of data and the area perspective performs the garbage collection operation on a plurality of the blocks in a respective storage region. The block perspective and the area perspective portions of the garbage collection operation are executed independently of one another.

Abstract: A computer-implemented method according to one embodiment includes receiving data, storing chunks of the data in fixed size blocks in physical storage, and deduplicating a copy of the data. The storing includes splitting some of the data chunks into portions for filling the fixed size blocks. Metadata is stored with the data chunks for chaining the associated portions of the split data chunks together. The deduplicating including creating a pointer to a first of the data chunks. A computer program product for utilizing free space in physical storage, according to another embodiment includes a computer readable storage medium having program instructions embodied therewith. The computer readable storage medium is not a transitory signal per se. The program instructions are readable and/or executable by a computer to cause the computer to perform the foregoing method.

Abstract: Embodiments for efficient garbage collection in a data storage environment. In a storage system comprising multiple storage devices having respective sets of storage regions, at least one respective storage fragmentation threshold used to trigger a garbage collection operation is identified. The garbage collection operation is performed to reclaim data space in the storage system according to each of a block perspective and an area perspective. The block perspective performs the garbage collection operation on individual blocks of data and the area perspective performs the garbage collection operation on a plurality of the blocks in a respective storage region. The block perspective and the area perspective portions of the garbage collection operation are executed independently of one another.

Abstract: A mechanism is provided for dispersed location-based data storage. A request is received to write a data file to a referrer memory region in a set of memory regions. For each data chunk of the data file, responsive to a comparison of a hash value for the data chunk to other hash values for other stored data chunks referenced in the referrer memory region indicating that the data chunk fails to exist in the referrer memory region, responsive to the data chunk existing in another memory region in the set of memory regions, responsive to the memory region failing to be one of the predetermined number N of owner memory regions associated with the referrer memory region, and responsive to the predetermined number N of owner memory regions failing to have been met, a reference to the data chunk is stored in the referrer memory region.

Abstract: For deduplicated data processing hierarchical rate control in a data deduplication system in a computing storage environment, priorities in a hierarchal structure are propagated by defining a relationship between each of a plurality of maintenance tasks and using a resource distribution function for managing and dividing resources between the plurality of maintenance tasks based on a priority for each maintenance task. Each one of the plurality of maintenance tasks is defined to be one of above, below, and at a same level in the hierarchal structure as compared to another one of the plurality of maintenance tasks. The hierarchal structure comprises a tree structure having a plurality of leaf nodes that are actual task and branch nodes that are virtual tasks.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include configuring, at a first time, a clustered storage system comprising multiple modules to store and process storage requests for respective sets of data objects. While processing the storage requests, respective subsets of the data objects having one or more data errors are identified, and at a second time subsequent to the first time, respective modules storing each of the data objects having at least one data error are identified. computing, based on the identified modules, a frequency distribution of the identified data errors in the data objects over the multiple modules. Based on the frequency distribution, a failure in a given module is identified, and the identified module is removed from the storage system. In some embodiments, prior to the second time, the data objects can be redistributed among the modules of the storage system.

Abstract: For deduplicated data processing hierarchical rate control in a data deduplication system in a computing storage environment, priorities in a hierarchal structure are propagated by defining a relationship between each of a plurality of maintenance tasks and using a resource distribution function for managing and dividing resources between the plurality of maintenance tasks based on a priority for each maintenance task. Each one of the plurality of maintenance tasks is defined to be one of above, below, and at a same level in the hierarchal structure as compared to another one of the plurality of maintenance tasks. The hierarchal structure comprises a tree structure having a plurality of leaf nodes that are actual task and branch nodes that are virtual tasks.

Abstract: For deduplicated data processing hierarchical rate control in a data deduplication system in a computing storage environment, priorities in a hierarchal structure are propagated by defining a relationship between each of a plurality of maintenance tasks and using a resource distribution function for managing and dividing resources between the plurality of maintenance tasks based on a priority for each maintenance task.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include configuring, at a first time, a clustered storage system comprising multiple modules to store and process storage requests for respective sets of data objects. While processing the storage requests, respective subsets of the data objects having one or more data errors are identified, and at a second time subsequent to the first time, respective modules storing each of the data objects having at least one data error are identified. computing, based on the identified modules, a frequency distribution of the identified data errors in the data objects over the multiple modules. Based on the frequency distribution, a failure in a given module is identified, and the identified module is removed from the storage system. In some embodiments, prior to the second time, the data objects can be redistributed among the modules of the storage system.

Abstract: For deduplicated data processing hierarchical rate control in a data deduplication system in a computing storage environment, priorities in a hierarchal structure are propagated by defining a relationship between each of a plurality of maintenance tasks and using a resource distribution function for managing and dividing resources between the plurality of maintenance tasks based on a priority for each maintenance task.