Abstract: The described technology is generally directed towards stretching a mapped storage clusters by adding nodes to a mapped cluster of mapped nodes and storage devices mapped to a real cluster of nodes and storage devices. Stretching the mapped cluster can provide additional computing resources to a set of storage devices. In one implementation, one or more newly mapped nodes are added to increase the node count of an existing mapped cluster to form a stretched cluster, with the storage devices distributed among the increased number of nodes; a mapping table is updated to relate the stretched cluster nodes and storage devices to the real cluster nodes and storage devices. Also described is un-stretching a stretched cluster, or further stretching a stretched cluster.

Abstract: The described technology is generally directed towards an input/output (I/O) load balancer of a data storage system that detects an I/O overloaded (“hot”) storage unit and logically moves its hot data to a non-overloaded (“cold”) storage unit. Threshold load levels can be used to determine hot and cold storage units. In one implementation, new writes to the hot storage unit are prevented while its hot data is logically moved to a cold storage unit. To avoid reads from the hot storage unit, the hot data can be recreated from redundant data obtained via a recovery path. To avoid a capacity imbalance, once enough hot data has been moved so that the (formerly) hot storage device is no longer considered hot, cold data from the cold storage device can be written to the formerly hot storage device. New data writes to the formerly hot storage device can then resume.

Abstract: The disclosed technology generally describes a data protection scheme that for “mid-size” objects directly writes divided object data fragments, and performs erasure coding to directly write object coding fragments, to distributed storage locations in a node cluster. A storage container such as a chunk allocated for mid-size objects is distributed among the storage cluster nodes. When a mid-size object (e.g., between 24 megabytes and 128 megabytes) is to be created, the object data is divided into object data fragments and encoded into object coding fragments, with the data object fragments and object coding fragments written/appended to the distributed storage locations, without needing a preliminary protection scheme.

Abstract: The disclosed technology generally describes separating types of data chunks in a copy-on-write/MVCC B+ tree, chunk-based data storage system, and also allocating the sizes of leaf chunks to be smaller than that of other (e.g., internal and root node) chunks. By having leaf chunks separate from node chunks, the probability of having a fully reclaimable (without copying) chunk is increased. Similarly, by having smaller sized leaf chunks relative to node chunks, the probability of having a fully reclaimable (without copying) leaf chunks is increased. The technology thus facilitates more efficient garbage collection.

Abstract: The technology described herein is directed towards consolidating garbage collection of data stored in data structures such as chunks, to facilitate efficient garbage collection. Low capacity utilization chunks are detected as source chunks, and live data of an object (e.g., in segments) is copied from the source chunks to new destination chunk(s). A source chunk is deleted when it no longer contains live data. By copying the data on an object-determined basis, new chunks contain more coherent object data, which increases the possibility of future chunk deletion without data copying or with a reduced amount of copying. When data segments of an object are adjacent, the consolidating garbage collector may unite them into a united segment, which reduces an amount of system metadata per object. New chunks can be associated with a generation number (e.g., indicating the oldest previous generation) to further facilitate more efficient future chunk deletion.

Abstract: The disclosed technology generally describes a data protection scheme that for “mid-size” objects directly writes divided object data fragments, and performs erasure coding to directly write object coding fragments, to distributed storage locations in a node cluster. A storage container such as a chunk allocated for mid-size objects is distributed among the storage cluster nodes. When a mid-size object (e.g., between 24 megabytes and 128 megabytes) is to be created, the object data is divided into object data fragments and encoded into object coding fragments, with the data object fragments and object coding fragments written/appended to the distributed storage locations, without needing a preliminary protection scheme.

Abstract: The technology described herein is directed towards consolidating garbage collection of data stored in data structures such as chunks, to facilitate efficient garbage collection. Low capacity utilization chunks are detected as source chunks, and live data of an object (e.g., in segments) is copied from the source chunks to new destination chunk(s). A source chunk is deleted when it no longer contains live data. By copying the data on an object-determined basis, new chunks contain more coherent object data, which increases the possibility of future chunk deletion without data copying or with a reduced amount of copying. When data segments of an object are adjacent, the consolidating garbage collector may unite them into a united segment, which reduces an amount of system metadata per object. New chunks can be associated with a generation number (e.g., indicating the oldest previous generation) to further facilitate more efficient future chunk deletion.

Abstract: The disclosed technology generally describes a preliminary (e.g., triple mirroring) data protection scheme that operates by writing data as redundant (e.g., three) composite copies made up of copies of data fragments to different nodes of a data storage system. The data fragments are distributed such that any two nodes can fail yet a complete set of data remains among the remaining data fragments. Later, erasure encoding creates redundant coding fragments that are written to the nodes of a data storage system in a distributed manner along with one copy of the data fragments, such that any two nodes can fail but the complete data can still be recovered. Redundant data fragments are then deleted.

Abstract: Affinity sensitive storage of data corresponding to a mapped redundant array of independent nodes, e.g., mapped cluster, in a real storage system, e.g., a real cluster, is disclosed. Different mappings of mapped cluster data to real cluster storage locations can result in different levels of affinity between real nodes of the real cluster. A data storage scheme can be selected based on affinity scores, for example drawn from an affinity matrix, to provide access to stored data that can be more resilient against a real node becoming less available. Further, data recovery from a real node that has become less accessible can be improved where data is stored based on the affinity scores. Generally, data storage that provides greater diversity of data storage locations can be related to more desirable affinity scores. Further, data storage that provides less divergence of affinity scores across an affinity matrix can also be desirable.

Abstract: Hybrid intra-cluster migration of data in an elastic cloud storage (ECS) environment is disclosed herein. A system comprises a processor and a memory that stores executable instructions that, when executed by the processor, perform operations that include moving a first data chunk from a first storage device to a second storage device during a data migration, based on a first determination that a first use efficiency of the first data chunk satisfies a defined use efficiency threshold. The operations also include moving a first data segment from a second data chunk in the first storage device to a third data chunk in the second storage device during the data migration, based on a second determination that a second use efficiency of the second data chunk fails to satisfy the defined use efficiency threshold. The first data segment includes data that is open for new writes and a second data segment includes data that is not open for new writes. After data is moved, capacity of the first data chuck is recovered.

Abstract: A method, computer program product, and computer system for determining, by a computing device, that an object of a plurality of objects is an orphan. It may be determined that the object is older than a threshold age. A capacity occupied by the object may be reclaimed based upon, at least in part, determining that the object is an orphan and determining that the object is older than the threshold age.

Abstract: The technology described herein is directed towards balancing workload between cluster nodes via redistribution of metadata data structures (e.g., memory tables corresponding to directory table partitions). Workload-related information of a node and its partitions' primary memory tables usage is measured, and if sufficiently high, causes a move of a highly-accessed memory table (corresponding to high workload on a first node) from the first node to a second node that has less workload. The second node can contain a backup (e.g., shallow) memory table to the primary node, whereby the move can be a logical move that transforms the backup memory table into a new instance of the primary memory table on the second node. The first node's primary memory table can be deflated into a backup table on the first node that backs up the new instance of the primary table on the second node.

Abstract: A mapped redundant array of independent regions (mapped RAIR) for data storage is disclosed. A mapped RAIR can be allocated on top of one or more regions of a cluster storage construct or system. The cluster storage construct can be N nodes wide by M disks deep. A mapped RAIR cluster can comprise sites from real or mapped regions. A mapped region can comprise sites from two different real regions. Selection of sites comprised in a mapped region of a mapped RAIR can be based on geographic proximity, network proximity, a constraint, best practice, rule, etc., on customer preferences, etc. A mapped RAIR can provide data protection for data at a regional level.

Abstract: Inter-zone network traffic generated during deletion of a data chunk that has been replicated by employing geographically distributed (GEO) erasure coding is reduced. In one aspect, if a data chunk is to be deleted, partial coding chunks are generated by a source zone and provided to destination zones that store complete coding chunks for updating the complete coding chunks based on combining them with the received partial coding chunks. In another aspect, if a first data chunk is to be deleted and a second data chunk is to be replicated, partial coding chunks are generated by the source zone for each data chunk. Further, the partial coding chunks created for different data chunks can be combined to generate transforming chunks, which can then be transferred to the destination zones. The destination zones can then update the complete coding chunks based on combining them with the received transforming chunks.

Abstract: Facilitating multi-level data deduplication in an elastic cloud storage environment is provided herein. A system can comprise a processor and a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations. The operations can comprise performing a first deduplication on a group of data objects at a data block level of a storage device. The operations can also comprise performing a second deduplication of the group of data objects at an object level of the storage device.

Abstract: The described technology is generally directed towards proactive disk recovery that operates when a failing disk is detected in a data-protected cloud data storage system. A proactive recovery process evaluates the chunks of a failing disk one-by-one. If a system process is scheduled to handle that chunk, the chunk is skipped, with recovery delegated to the system process. For non-delegated chunks protected by mirroring, a chunk copy is read by the proactive disk recovery process from a good disk copy, and copied to a new location. For non-delegated chunks protected by erasure coding, the chunk fragment is read and validated. If a portion is consistent, the proactive recovery process stores the portion to a new location on a good disk. If a portion is inconsistent, the process initiates recovery of the portion, e.g., via a fragment recovery task, for copying to a new location on a good disk.

Abstract: The described technology is generally directed towards recovery of data segments from geographic zones (dynamic GEO recovery) by having a zone that needs the data direct the recovery process using counterpart segments. If needed data, such as to respond to a client request, is owned by another zone but is lost or corrupt and therefore unavailable from that owning zone, the owning zone instructs the requesting zone to perform recovery. The zone performs recovery by obtaining the counterpart segments, combining (XOR-ing) the counterpart recovery segments into the needed segment, and returns the data to the client. If the zone performing recovery owns one of the counterpart segments, only one of the two counterpart segments needs to be communicated over the inter-zone network, facilitating more efficient, less resource-demanding GEO recovery.

Abstract: The described technology is generally directed towards stretching a mapped storage clusters by adding nodes to a mapped cluster of mapped nodes and storage devices mapped to a real cluster of nodes and storage devices. Stretching the mapped cluster can provide additional computing resources to a set of storage devices. In one implementation, one or more newly mapped nodes are added to increase the node count of an existing mapped cluster to form a stretched cluster, with the storage devices distributed among the increased number of nodes; a mapping table is updated to relate the stretched cluster nodes and storage devices to the real cluster nodes and storage devices. Also described is un-stretching a stretched cluster, or further stretching a stretched cluster.

Abstract: Resource-efficient data protection is performed by generating meta chunks in storage systems that utilize erasure coding. During erasure coding with a k+m configuration, a data chunk can be divided into k data fragments, having indices 1 to k, that can be encoded by combining them with corresponding coefficients of a coding matrix, to generate coding fragments. Source portions that have a reduced set (e.g., less than k data fragments) of data fragments can be modified such that they are made complementary (e.g., that do not have common indices) without complete data re-protection. The complementary portions can then be combined to generate a meta chunk. The coding fragments of the complementary portions can be added to generate coding fragments for the meta chunk, which can then be utilized to recover data fragments of any of the source portions.

Abstract: Described herein, system that facilitates mapping of redundant array of independent nodes of a storage device. According to an embodiment, a system can comprise generating a first configuration of a storage cluster, wherein the storage cluster comprises a group of nodes and a group of disks, generating a second configuration of the storage cluster using the first configuration, wherein the group of nodes are divided into a first pair of nodes comprising a first node having access to a first group of disks and a second node having access to a second group of disks, and generating a third configuration of the storage cluster using the second configuration, wherein the first node comprises a first mapped node that manages the first group of disks of the first node and enables access to the second group of disks of the second node.