Abstract: Scaling-in of a geographically diverse data storage system is disclosed. Data chunks stored in storage devices of different zones can be convolved to conserve memory. The zone storage system can be a geographically diverse storage system. Removing a zone from the geographically diverse data storage system can risk data loss without adequate replication of data from the zone to be removed. The zone to be removed can be placed in a ‘read-only’ state. Data chunks form the zone to be removed can be duplicated at other zones of the geographically diverse data storage system. A data chunk comprising convolved representations of data is duplicated to a zone in a manner that results in loss of geographical diversity, a corresponding chunk can be duplicated at another zone to restore geographical diversity. Validation of data can be performed prior to removal of a zone in a scaling-in event.

Abstract: A computer program product, system, and method for generating a test data set using object count and object size parameters, storing the generated data set in storage, subsequently regenerating the data set, and comparing the regenerated data set to the stored data set to validate the storage.

Abstract: Scaling-in of a geographically diverse data storage system is disclosed. Data chunks stored in storage devices of different zones can be convolved to conserve memory. The zone storage system can be a geographically diverse storage system. Removing a zone from the geographically diverse data storage system can risk data loss without adequate replication of data from the zone to be removed. The zone to be removed can be placed in a ‘read-only’ state. Data chunks form the zone to be removed can be duplicated at other zones of the geographically diverse data storage system. A data chunk comprising convolved representations of data is duplicated to a zone in a manner that results in loss of geographical diversity, a corresponding chunk can be duplicated at another zone to restore geographical diversity. Validation of data can be performed prior to removal of a zone in a scaling-in event.

Abstract: Method for tree management of trees under multi-version concurrency control is described herein. Method starts by detecting change in a size of a cluster. The number of nodes in the cluster is counted to obtain the changed size of the cluster. The number of trees needed for the number of nodes in the cluster is determined. The number of trees may be based on the number of nodes in the cluster and predetermined system balancing coefficient. When the number of trees needed is greater than existing number of trees, existing number of trees is doubled as a single-step operation which includes modifying and scaling hash function used to derive each tree to generate new hash function and using new hash function to partition each tree instantly into two new trees. Scaling happens on demand without service disruption. Hash function scales automatically when number of trees increases. Other embodiments are described.

Abstract: Scale out data protection with erasure coding is presented herein. Based on an initial number of storage devices determined to have been included in an initial stage of a data storage cluster, an initial protection scheme for the initial stage can determine first coding fragment(s) for data stored within the data storage cluster to facilitate a first recovery, from the initial stage, of the data using the first coding fragment(s). Further, in response to a defined number of additional storage devices being determined to have been added to the data storage cluster to generate a modified data storage cluster, the initial protection scheme can be modified to obtain a modified protection scheme that can determine, for the modified data storage cluster, second coding fragment(s) for the data to facilitate a second recovery of the data using the first coding fragment(s) and the second coding fragment(s).

Abstract: Overhead associated with data re-protection during scaling out and/or scaling up of a cloud storage system can be reduced. During initial data protection (e.g., prior to a change in the cluster size), data fragments and coding fragments can be stored as a set of adjacent data sub-fragments and a set of adjacent coding sub-fragments, respectively. In response to determining that the cluster has expanded, a portion of the set of adjacent data sub-fragments can be moved to the new nodes that have been added to the cluster. Further, the set of adjacent coding sub-fragments can be combined to generate a new coding fragment that can be stored within the new cluster. Accordingly, the amount of system resources utilized during re-protection can be significantly reduced.

Abstract: Scaling-in of a geographically diverse data storage system is disclosed. Data chunks stored in storage devices of different zones can be convolved to conserve memory. The zone storage system can be a geographically diverse storage system. Removing a zone from the geographically diverse data storage system can risk data loss without adequate replication of data from the zone to be removed. The zone to be removed can be placed in a ‘read-only’ state. Data chunks form the zone to be removed can be duplicated at other zones of the geographically diverse data storage system. A data chunk comprising convolved representations of data is duplicated to a zone in a manner that results in loss of geographical diversity, a corresponding chunk can be duplicated at another zone to restore geographical diversity. Validation of data can be performed prior to removal of a zone in a scaling-in event.

Abstract: Floating data protection is presented herein. The method comprises receiving a defined data protection policy; determining that the defined data protection policy is not susceptible to a single point of failure scenario; and in response to determining that the defined data protection policy is not susceptible to the single point of failure scenario, reducing a code fragment associated with a data portion based on the defined data protection policy.

Abstract: Scaling-in of a geographically diverse data storage system is disclosed. Data chunks stored in storage devices of different zones can be convolved to conserve memory. The zone storage system can be a geographically diverse storage system. Removing a zone from the geographically diverse data storage system can risk data loss without adequate replication of data from the zone to be removed. The zone to be removed can be placed in a ‘read-only’ state. Data chunks form the zone to be removed can be duplicated at other zones of the geographically diverse data storage system. A data chunk comprising convolved representations of data is duplicated to a zone in a manner that results in loss of geographical diversity, a corresponding chunk can be duplicated at another zone to restore geographical diversity. Validation of data can be performed prior to removal of a zone in a scaling-in event.

Abstract: Scale out data protection with erasure coding is presented herein. Based on an initial number of storage devices determined to have been included in an initial stage of a data storage cluster, an initial protection scheme for the initial stage can determine first coding fragment(s) for data stored within the data storage cluster to facilitate a first recovery, from the initial stage, of the data using the first coding fragment(s). Further, in response to a defined number of additional storage devices being determined to have been added to the data storage cluster to generate a modified data storage cluster, the initial protection scheme can be modified to obtain a modified protection scheme that can determine, for the modified data storage cluster, second coding fragment(s) for the data to facilitate a second recovery of the data using the first coding fragment(s) and the second coding fragment(s).

Abstract: One embodiment is related to a method for implementing a demoted chunk, comprising: replicating a first chunk from a first zone of a cloud storage system to a third zone of the cloud storage system; replicating a second chunk from a second zone of the cloud storage system to the third zone; creating an exclusive or (XOR) chunk at the third zone based at least in part on the replicated first chunk and the replicated second chunk; and marking the replicated first chunk and the replicated second chunk as demoted first chunk and demoted second chunk, respectively, at the third zone.

Abstract: A method for creating distributed erasure coding chunks in a distributed storage system with unbalanced load is disclosed. The operations comprise configuring the distributed storage system into at least k+m zones, wherein each zone accumulates at least l primary backup chunks of original data chunks replicated from different remote zones, and wherein l<k, and preparing the distributed storage system for recovery from a failure of 1 to m zones of the at least k+m zones including, in each zone, encoding the at least l primary backup chunks to create m coding chunks using an erasure coding having parameters k+m, wherein in each zone in which fewer than k primary backup chunks have been accumulated, at most k?l predetermined virtual chunks are used to create partial coding chunks.

Abstract: A method comprising: generating a storage chunk having a plurality of data fragments, the storage chunk stored in one or more storage devices; allocating space in a primary memory to store a plurality of coded fragments; receiving a plurality of I/O requests to write data; allocating space in the primary memory to store a next unfilled data fragment; processing a plurality of I/O requests to write data; and copying the coded fragments from the primary memory to the one or more storage devices. For each I/O request, writing the data to the next unfilled data fragment in the one or more storage devices, writing the data to the next unfilled data fragment in the primary memory, and if the data fragment in the primary memory is full, updating the coded fragments in the primary memory using the filled data fragment in the primary memory.

Abstract: A method comprising: receiving an I/O request for object data; determining one or more data fragments wherein the object data is stored; determining that one or more of the data fragments are unavailable; determining, from within the one or more unavailable data fragments, a set of slices storing the object data, each slice comprising k small data fragments and m coded fragments; for each slice, retrieving at least k small data and coded fragments within the slice from storage; and recovering a segment of the object data using the retrieved small data and coded fragments. A related system and computer program product are also described.

Abstract: Data object content verification systems and processes provide perfect reliability and low storage overhead. Object data is generated in a reproducible manner based upon object locally stored object metadata. The object data is stored to an object storage system. The stored object data is subsequently verified by retrieving the object metadata, reproducing the original object data, and comparing the stored and original object data. The data object content verification systems and processes support both mutable and immutable data object content verification.

Abstract: A method for generating coded fragments comprises receiving data to be encoded, splitting the data into a plurality of data fragments, identifying a first group of data fragments from among the plurality of data fragments using a coding matrix, summing the data fragments within the first group of data fragments to generate a first group sum, and using the first group sum to calculate at least a portion of two or more coded fragments.

Abstract: The disclosure relates to data protection management for geographically distributed storage systems. Specifically, the present disclosure provides a mechanism for load balancing by combining the techniques of XOR Star and XOR Chain to control the protection of data internally. The load balancing is done by each zone without the need for an external load balancer. In addition, the load balancing mechanism provides the ability for native (e.g. without additional configuration) support of disaster recovery.

Abstract: Method for tree management of trees under multi-version concurrency control is described herein. Method starts by detecting change in a size of a cluster. The number of nodes in the cluster is counted to obtain the changed size of the cluster. The number of trees needed for the number of nodes in the cluster is determined. The number of trees may be based on the number of nodes in the cluster and predetermined system balancing coefficient. When the number of trees needed is greater than existing number of trees, existing number of trees is doubled as a single-step operation which includes modifying and scaling hash function used to derive each tree to generate new hash function and using new hash function to partition each tree instantly into two new trees. Scaling happens on demand without service disruption. Hash function scales automatically when number of trees increases. Other embodiments are described.

Abstract: A computer program product, system, and method for generating a test data set using object count and object size parameters, storing the generated data set in storage, subsequently regenerating the data set, and comparing the regenerated data set to the stored data set to validate the storage.

Abstract: A method comprising: receiving an I/O request for object data; determining one or more data fragments wherein the object data is stored; determining that one or more of the data fragments are unavailable; determining, from within the one or more unavailable data fragments, a set of slices storing the object data, each slice comprising k small data fragments and m coded fragments; for each slice, retrieving at least k small data and coded fragments within the slice from storage; and recovering a segment of the object data using the retrieved small data and coded fragments. A related system and computer program product are also described.