Abstract: The technology describes scanning tree data structures (trees) for multiple processes, at least partly in parallel. A service scans a tree from a beginning tree element to an ending tree element on behalf of a process; while scanning, another process can join in the scan at an intermediate tree element location (e.g., a key). For the subsequent process, the service scans the tree based on the intermediate location to the tree end, thereby visiting tree elements in parallel until the tree end, then continuing from the tree beginning element to the intermediate location for the subsequent process. The service basically completes a full carousel-type revolution for each process. One or more other processes can join an ongoing scan at any time, facilitating further parallel tree element visits, while still obtaining a full scan of the entire set of tree elements. The service handles changing tree versions during the scanning.

Abstract: Scaling of an ordered event stream (OES) based on a characteristic of one or more writer groups is disclosed. Scaling a portion of an OES contemporaneous to writing events into that portion can conserve computing resources in contrast to more conventional scaling techniques. Moreover, scaling an OES contemporaneously with writing events thereto can enable improved management of OES scaling for applications that can both read events from an input portion of an OES and, via interim events and interim portions of an OES, write events to an output portion of an OES. An application instance can therefore simultaneously act as both a reader group and writer group and can manage data via interim OESs, such that effects of passing the data through the interim OESs can be cascaded into a scaling of the output portion of an OES based on the writer group characteristic.

Abstract: The described technology is generally directed towards an input/output (I/O) load balancer of a data storage system that detects an I/O overloaded (“hot”) storage unit and logically moves its hot data to a non-overloaded (“cold”) storage unit. Threshold load levels can be used to determine hot and cold storage units. In one implementation, new writes to the hot storage unit are prevented while its hot data is logically moved to a cold storage unit. To avoid reads from the hot storage unit, the hot data can be recreated from redundant data obtained via a recovery path. To avoid a capacity imbalance, once enough hot data has been moved so that the (formerly) hot storage device is no longer considered hot, cold data from the cold storage device can be written to the formerly hot storage device. New data writes to the formerly hot storage device can then resume.

Abstract: Scaling of an ordered event stream (OES) based on a characteristic of one or more writer groups is disclosed. Scaling a portion of an OES contemporaneous to writing events into that portion can conserve computing resources in contrast to more conventional scaling techniques. Moreover, scaling an OES contemporaneously with writing events thereto can enable improved management of OES scaling for applications that can both read events from an input portion of an OES and, via interim events and interim portions of an OES, write events to an output portion of an OES. An application instance can therefore simultaneously act as both a reader group and writer group and can manage data via interim OESs, such that effects of passing the data through the interim OESs can be cascaded into a scaling of the output portion of an OES based on the writer group characteristic.

Abstract: The described technology is generally directed towards recovering a chunk (or similar block of data) when the chunk is erasure coded into fragments, and recovery fragments need to be obtained from geographically distributed sites. The recovery fragments needed to perform recovery of a chunk are determined, and assigned to the geographically distributed sites as subtasks. Each site that receives a subtask from the requesting site obtains XOR-related fragments needed to produce the recovery fragment, performs the XOR operations on the XOR-related fragments to produce the recovery fragment, and returns the recovery fragment to the requesting site. When finished, a site receives another subtask until no subtasks remain, such that the fastest site or sites receive the most subtasks. The requesting site recovers the chunk from the received recovery fragments. The shared participation in the chunk recovery among the distributed sites provides for efficient distribution of the recovery-related resources and work.

Abstract: The technology describes performing garbage collection while data writes are occurring, which can lead to a conflict in that a new reference to an otherwise non-referenced candidate object for garbage collection is written after the non-referenced candidate object is detected. In one example implementation, orphaned binary large objects (BLOBs) that are not referenced by a descriptor file and are beyond a certain age are detected and deleted via an object references table traversal as part of garbage collection. Before reclaiming a deleted BLOB's capacity, a background process operates to restore the deleted BLOB if a new descriptor file reference to the BLOB was written during the object references table traversal. Capacity is only reclaimed after the object references table traversal and the background processing completes, for those BLOBs that were deleted and had not been restored.

Abstract: Efficient handling of disk input/output (I/O) operations is disclosed for a storage system supporting a doubly mapped redundant array of independent nodes. Delegating I/O operation management to a mapping layer component can support adapting I/O operation performance based on a topology of a real cluster supporting doubly mapped clusters. In traditional techniques, some corresponding real node topologies can result in inefficiently moving data from a first real node to a second real node and then promptly moving it back from the second real node to the first real node. These, and other, inefficient disk I/O conditions can be mitigated or avoided based on the presently disclosed techniques that can support more direct access to mapping information. Moreover, the disclosed techniques can support proportionately adapting an order of performing I/O operations due to the inherent isolation of real nodes and real clusters in accord with the disclosed subject matter.

Abstract: A mapped redundant array of independent regions (mapped RAIR) for data storage is disclosed. A mapped RAIR can be allocated on top of one or more regions of a cluster storage construct or system. The cluster storage construct can be N nodes wide by M disks deep. A mapped RAIR cluster can comprise sites from real or mapped regions. A mapped region can comprise sites from two different real regions. Selection of sites comprised in a mapped region of a mapped RAIR can be based on geographic proximity, network proximity, a constraint, best practice, rule, etc., on customer preferences, etc. A mapped RAIR can provide data protection for data at a regional level.

Abstract: Described herein, system that facilitates mapping of redundant array of independent nodes of a storage device. According to an embodiment, a system can comprise generating a first configuration of a storage cluster, wherein the storage cluster comprises a group of nodes and a group of disks, generating a second configuration of the storage cluster using the first configuration, wherein the group of nodes are divided into a first pair of nodes comprising a first node having access to a first group of disks and a second node having access to a second group of disks, and generating a third configuration of the storage cluster using the second configuration, wherein the first node comprises a first mapped node that manages the first group of disks of the first node and enables access to the second group of disks of the second node.

Abstract: The described technology is generally directed towards recovering a chunk (or similar block of data) when the chunk is erasure coded into fragments, and recovery fragments need to be obtained from geographically distributed sites. The recovery fragments needed to perform recovery of a chunk are determined, and assigned to the geographically distributed sites as subtasks. Each site that receives a subtask from the requesting site obtains XOR-related fragments needed to produce the recovery fragment, performs the XOR operations on the XOR-related fragments to produce the recovery fragment, and returns the recovery fragment to the requesting site. When finished, a site receives another subtask until no subtasks remain, such that the fastest site or sites receive the most subtasks. The requesting site recovers the chunk from the received recovery fragments. The shared participation in the chunk recovery among the distributed sites provides for efficient distribution of the recovery-related resources and work.

Abstract: Selectively distributing fragments of a data protection set in a geographically diverse data storage system is disclosed. The data protection set can comprise fewer fragments than there are zones comprising the geographically diverse data storage system, which can result in some zones not storing a fragment of the data protection set. Control over distribution of fragments of different data protection sets in the geographically diverse data storage system can mitigate or avoid unbalanced storage of the protection sets. The distribution can be controlled in accordance with a protection set distribution scheme (PSDS). A first PSDS can generate coding fragments from randomly select data fragments of all zones. A second PSDS can generate coding fragments from determined unique zone combinations. A third PSDS can generate coding fragments based on affinity values from an affinity matrix.

Abstract: The disclosed technology generally describes a preliminary (e.g., triple mirroring) data protection scheme that operates by writing data as redundant (e.g., three) composite copies made up of copies of data fragments to different nodes of a data storage system. The data fragments are distributed such that any two nodes can fail yet a complete set of data remains among the remaining data fragments. Later, erasure encoding creates redundant coding fragments that are written to the nodes of a data storage system in a distributed manner along with one copy of the data fragments, such that any two nodes can fail but the complete data can still be recovered. Redundant data fragments are then deleted.

Abstract: Merging of portions of ordered event streams is disclosed. The disclosed merging of events can limit loss of order of events from streams in exchange for reduced computational load by grouping events according to a pseudo-epoch, wherein events are stored according to a scheme, even though the grouping can result reading events in a different order that that in which the events were written. However, by grouping the events, there can be fewer transitions between storage schemes when reading events than if they were read in the same order in which they were written, thereby reducing computational load. Moreover, restraints on the loss of order can be imposed by selecting a maximum progress window and generally restricting groups from comprising events two different storage schemes. Where events can be moved to archival storage, reducing storage scheme transitions can be of further benefit and speed access times of archived events.

Abstract: The technology describes performing garbage collection while data writes are occurring, which can lead to a conflict in that a new reference to an otherwise non-referenced candidate object for garbage collection is written after the non-referenced candidate object is detected. In one example implementation, orphaned binary large objects (BLOBs) that are not referenced by a descriptor file and are beyond a certain age are detected and deleted via an object references table traversal as part of garbage collection. Before reclaiming a deleted BLOB's capacity, a background process operates to restore the deleted BLOB if a new descriptor file reference to the BLOB was written during the object references table traversal. Capacity is only reclaimed after the object references table traversal and the background processing completes, for those BLOBs that were deleted and had not been restored.

Abstract: The technology describes scanning tree data structures (trees) for multiple processes, at least partly in parallel. A service scans a tree from a beginning tree element to an ending tree element on behalf of a process; while scanning, another process can join in the scan at an intermediate tree element location (e.g., a key). For the subsequent process, the service scans the tree based on the intermediate location to the tree end, thereby visiting tree elements in parallel until the tree end, then continuing from the tree beginning element to the intermediate location for the subsequent process. The service basically completes a full carousel-type revolution for each process. One or more other processes can join an ongoing scan at any time, facilitating further parallel tree element visits, while still obtaining a full scan of the entire set of tree elements. The service handles changing tree versions during the scanning.

Abstract: The technology described herein is directed towards processing data that is protected by a preliminarily protection scheme (e.g., triple mirroring) before re-protecting that data via erasure coding. Data of new or updated objects, which can be segmented in one or more preliminarily protected data chunks (a data inbox), is consolidated to put the object's data segments in contiguous space. The consolidated object data can be compressed, and erasure coded (possibly along with consolidated and compressed data of one or more other objects) into data fragments and coding fragments of a distributed destination data chunk. Once an object is stored via erasure coding, the source chunk or chunks no longer contain live data of that object; when a source chunk contains no live data of any object, the capacity of the source chunk (and any mirror copies) can be reclaimed.

Abstract: The described technology is generally directed towards data storage using a node cluster, and garbage collecting unused chunks (data storage units) in the cluster based on which node owns the particular unused chunks. A node determines which chunks are in use, and exchanges datasets identifying those chunks with other nodes such that the other nodes know which of the chunks that they own are in use. When a node obtains the dataset identifying the chunks in use, the node determines the chunks not in use by a difference of those owned and those in use. This difference dataset is used to garbage collect owned, unused chunks. Garbage collection via this technology is able to be performed in a single cycle.

Abstract: Merging of portions of ordered event streams is disclosed. The disclosed merging of events can limit loss of order of events from streams in exchange for reduced computational load by grouping events according to a pseudo-epoch, wherein events are stored according to a scheme, even though the grouping can result reading events in a different order that that in which the events were written. However, by grouping the events, there can be fewer transitions between storage schemes when reading events than if they were read in the same order in which they were written, thereby reducing computational load. Moreover, restraints on the loss of order can be imposed by selecting a maximum progress window and generally restricting groups from comprising events two different storage schemes. Where events can be moved to archival storage, reducing storage scheme transitions can be of further benefit and speed access times of archived events.

Abstract: The technology described herein is directed towards balancing workload between cluster nodes via redistribution of metadata data structures (e.g., memory tables corresponding to directory table partitions). Workload-related information of a node and its partitions' primary memory tables usage is measured, and if sufficiently high, causes a move of a highly-accessed memory table (corresponding to high workload on a first node) from the first node to a second node that has less workload. The second node can contain a backup (e.g., shallow) memory table to the primary node, whereby the move can be a logical move that transforms the backup memory table into a new instance of the primary memory table on the second node. The first node's primary memory table can be deflated into a backup table on the first node that backs up the new instance of the primary table on the second node.

Abstract: A method, computer program product, and computer system for receiving a change to a configuration parameter associated with a first cluster node in a cluster configuration. The configuration parameter may be stored in a shared cluster memory. The configuration parameter may be distributed from the shared cluster memory to the cluster configuration.