Abstract: Retention of events of an ordered event stream according to at least one triggered retention policy is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events can be added to a head of an OES segment, and pruning events from a tail of the OES segment can be desirable. Processing applications can predicate event retention, e.g., restricting expiration of an event, on at least one triggered retention policy. In some embodiments, an additional fixed retention policy can be combined with the triggered retention. The disclosed retention can be performed at the event-level or at less granular levels, e.g., segment-level, OES-level, etc., e.g., via batching of events.

Abstract: Data migration from a legacy data storage system into an ordered event stream (OES) data storage system is disclosed. In contrast to conventional techniques, the disclosed subject matter can provide for migrating legacy data into a first portion of a vintage OES (VOES) and new data, intended for the legacy data storage system during the migration process, into a second portion of the VOES. The first and second portions of the VOES can then be linked at completion of the migration process. Avoiding writing new data to the legacy data storage system prior to migration enables a corresponding reduction in data access disruption. Moreover, event sequencing by separating migrated legacy events from new events can preserving event order. Scaling of vintage streams can be supported. Vintage streams can support function calls to a single VOES in contrast to the more conventional use of multiple streams.

Abstract: The technology describes performing garbage collection while data writes are occurring, which can lead to a conflict in that a new reference to an otherwise non-referenced candidate object for garbage collection is written after the non-referenced candidate object is detected. In one example implementation, orphaned binary large objects (BLOBs) that are not referenced by a descriptor file and are beyond a certain age are detected and deleted via an object references table traversal as part of garbage collection. Before reclaiming a deleted BLOB's capacity, a background process operates to restore the deleted BLOB if a new descriptor file reference to the BLOB was written during the object references table traversal. Capacity is only reclaimed after the object references table traversal and the background processing completes, for those BLOBs that were deleted and had not been restored.

Abstract: The described technology is generally directed towards rebalancing fragments (components) of a protection group among storage entities (e.g., nodes or disks) based on affinity levels in response to a scale out event that expands the storage entities e.g., nodes or disks) of a system. Affinity levels (e.g., maintained in an affinity matrix) represent dependency relationships between the storage entities with respect to storing distributed protection group fragments. A protection group's components are rebalanced based on the new storage entities such that the affinity level between any pair of storage entities is approximately the same as any other pair. In the event of a storage entity failure, as a result of the affinity-based distribution of the protection group components needed for data recovery, a larger number of the other storage entities can be involved in the data recovery. The technology facilitates improved capacity load balancing and faster data recovery.

Abstract: The described technology is generally directed towards recovering a chunk (or similar block of data) when the chunk is erasure coded into fragments, and recovery fragments need to be obtained from geographically distributed sites. The recovery fragments needed to perform recovery of a chunk are determined, and assigned to the geographically distributed sites as subtasks. Each site that receives a subtask from the requesting site obtains XOR-related fragments needed to produce the recovery fragment, performs the XOR operations on the XOR-related fragments to produce the recovery fragment, and returns the recovery fragment to the requesting site. When finished, a site receives another subtask until no subtasks remain, such that the fastest site or sites receive the most subtasks. The requesting site recovers the chunk from the received recovery fragments. The shared participation in the chunk recovery among the distributed sites provides for efficient distribution of the recovery-related resources and work.

Abstract: The described technology is generally directed towards fine-grained data expiration to reduce storage capacity used in a streaming data storage system. Segments of events are scanned, with each event's payload evaluated to determine whether that event is expired or unexpired. For example, for a log event, an expiration period can be determined based on the event creation time and the event logging level, both of which are typically within the log event's payload. Unexpired events are copied to a new destination segment that replaces the original, scanned source segment; the expired events are deleted when the source segment is deleted, reclaiming storage capacity used by the expired events. Also described is maintaining and using segment and chunk metadata to bypass individual event processing when the metadata indicates that the chunk contains only expired events and can be safely deleted.

Abstract: Disk access event control for mapped nodes of a cluster storage system supporting a redundant array of independent nodes (mapped RAIN) system is disclosed. A mapped RAIN cluster can be allocated on top of one or more real data clusters. In an embodiment, disk access events can be routed via a storage service instance supporting a mapped node. In another embodiment, disk access events can be routed via another storage service instance that does not support the mapped node. Routing the disk access event via another storage service instance that does not support the mapped node can reduce the use of computing resources. Further, the routing of the disk access event can be according to a proportional disk operation value determined based on historical disk access event characteristics.

Abstract: The described technology is generally directed towards replicating metadata representing a virtual data structure corresponding to replicated legacy data instead of the actual data for the data structure. Once virtual chunks are replicated to a remote, newer storage system, the corresponding legacy data is locally read into the virtual chunks to transform the virtual chunks into real data chunks of the remote newer storage system. A checksum can be replicated for the remote newer storage system to evaluate the consistency of the data. Efficient data storage migration is thus accomplished in a replicated environment based on relatively negligible replication traffic between two remote locations, while still assuring the consistency of migrated data.

Abstract: The described technology is generally directed towards cross-stream transactions in a streaming data storage system, which allows a writer application to commit multiple events to distinct data streams in a single transaction. The system creates a cross-stream transaction for a writer application, and the writer application adds events to the cross-stream transaction, indicating which destination data stream(s) each event's data is to be appended. The system adds the event to a subordinate transaction created for each specified data stream. Upon committing the cross-stream transaction, the system coordinates the committing of the subordinate transactions to their respective data streams.

Abstract: Scaling an ordered event stream (OES) based on an information-unit (IU) metric is disclosed. The IU metric can correspond to an amount of computing resources that can be consumed to access information embodied in event data of an event of the OES. In this regard, the amount of computing resources to access the data of the stream event itself can be distinct from an amount of computing resources employed to access information embodied in the data. As such, where an external application, e.g., a reader, a writer, etc., can connect to an OES data storage system, enabling the OES to be scaled in response to burdening of computing resources accessing event information, rather than merely event data, can aid in preservation of an ordering of events accessed from the OES.

Abstract: Data migration from a legacy data storage system into an ordered event stream (OES) data storage system is disclosed. In contrast to conventional techniques, the disclosed subject matter can provide for migrating legacy data into a first portion of a vintage OES (VOES) and new data, intended for the legacy data storage system during the migration process, into a second portion of the VOES. The first and second portions of the VOES can then be linked at completion of the migration process. Avoiding writing new data to the legacy data storage system prior to migration enables a corresponding reduction in data access disruption. Moreover, event sequencing by separating migrated legacy events from new events can preserving event order. Scaling of vintage streams can be supported. Vintage streams can support function calls to a single VOES in contrast to the more conventional use of multiple streams.

Abstract: Event prioritization for an ordered event stream (OES) is disclosed. Unlike conventional prioritization techniques, the disclosed subject matter can be performed by an OES data storage system to provide direct, rather than indirect, control of prioritization. In an embodiment, a prioritized hashed key (PHK) can be determined from an event characteristic and an indicated event priority value based on a selectable priority-sensitive hashing function. As such, events with a same key characteristic but different indicated priorities can have different PHKs, events with different key characteristics but the same indicated priority can have different PHKs, and events with the same key characteristic and the same priority can have a same PHK. An event priority can be inherently comprised in the PHK without needing to explicitly store the priority value with a written event in the OES. Moreover, the disclosed prioritization for the OES can be compatible with OES scaling techniques.

Abstract: The described technology is generally directed towards fine-grained data event expiration in a streaming data storage system. An event to append is given an expiration period, and the expiration time for the events in a data stream or segment of a data stream is the largest expiration time among events in the data stream or segment. Different segments can have different expiration times for their events. In a segment comprising a group of events, a subgroup of expired events prior to a stream cut are deleted by an expiration task. For a subgroup of unexpired events prior to a stream cut, the expiration task retains (does not delete) the subgroup of events. If a scaling operation is performed on a segment, the new successor segment or segments inherit the largest expiration time of the predecessor segment or segments.

Abstract: A mapped redundant array of independent regions (mapped RAIR) for data storage is disclosed. A mapped RAIR can be allocated on top of one or more regions of a cluster storage construct or system. The cluster storage construct can be N nodes wide by M disks deep. A mapped RAIR cluster can comprise sites from real or mapped regions. A mapped region can comprise sites from two different real regions. Selection of sites comprised in a mapped region of a mapped RAIR can be based on geographic proximity, network proximity, a constraint, best practice, rule, etc., on customer preferences, etc. A mapped RAIR can provide data protection for data at a regional level.

Abstract: Efficient handling of disk input/output (I/O) operations is disclosed for a storage system supporting a doubly mapped redundant array of independent nodes. Delegating I/O operation management to a mapping layer component can support adapting I/O operation performance based on a topology of a real cluster supporting doubly mapped clusters. In traditional techniques, some corresponding real node topologies can result in inefficiently moving data from a first real node to a second real node and then promptly moving it back from the second real node to the first real node. These, and other, inefficient disk I/O conditions can be mitigated or avoided based on the presently disclosed techniques that can support more direct access to mapping information. Moreover, the disclosed techniques can support proportionately adapting an order of performing I/O operations due to the inherent isolation of real nodes and real clusters in accord with the disclosed subject matter.

Abstract: Mitigating the effects of a real node failure in a doubly mapped redundant array of independent nodes, e.g., doubly mapped cluster is disclosed. In response to a change in an accessibility to data stored on an extent of a real storage device of a real node of a real cluster, wherein the extent of the real storage device corresponds to a portion of a mapped storage device of a mapped node of a doubly mapped cluster, substituting a reserved extent of a real storage device for the extent of the real storage device. The substituting the reserved extent of the real storage device can correspond to a change in a topology of the doubly mapped cluster, wherein the change in the topology comprises replacing the portion of the mapped storage device with a substitute portion of a mapped storage device that corresponds to the replacement extent of the real storage device.

Abstract: The technology describes forking (dividing) a data stream into multiple data streams comprising appending a first new data stream and a second new data stream to the original data stream. After forking, the two new data streams have different sets of stream fragments, with one or more of the older stream fragments common to both sets. Data events before the forking operation are read from the original data stream, while data events appended after the forking operation are appended to and read from only one of the data streams, based on which stream each writer/reader is registered to. Also described is truncating the original data stream based on one stream identifier, while leaving the original data intact for another stream identifier. Only when no stream needs a portion of the original data (resulting from separate truncation operations) is the data portion deleted.

Abstract: Scaling out of a geographically diverse storage system is disclosed. A first chunk can be selected to be moved in response to scaling out of the geographically diverse storage system. In some embodiments, the first chunk can be convolved, combined, etc., with a second chunk prior to moving a representation of information comprised in the first chunk, e.g., via copy and delete operations. The moving can be in accord with a geographically diverse storage system schema and can be based on a criteria associated with the geographically diverse storage system. In an embodiment the convolved chunk can be copied to a new zone and, subsequently, the first chunk can be deleted from the old zone. In another embodiment the combined chunk can be retained at the old zone and the first chunk can be copied to the new zone prior to deleting the first chunk from the old zone.

Abstract: A mapped redundant array of independent nodes (mapped RAIN) for data storage is disclosed. A mapped RAIN cluster can be allocated on top of one or more real data clusters, wherein the real clusters can comprise storage devices of different storage capacities. Mapping of data storage locations for a mapped RAIN cluster to real storage devices can be based on an affinity value determined for pairs of real nodes of the real data clusters. A normalized affinity can be employed to enable allocation of real storage to mapped nodes of mapped clusters that can be based on the heterogeneous capacities of the storage devices. This can provide improved data availability and data recovery over other techniques where heterogeneity of hardware can make efficient resource allocation a non-trivial task. The disclosed subject matter can facilitate more efficient allocation of Mapped RAINs in a heterogeneous cluster storage construct.

Abstract: Selective instantiation of a mapped cluster storage service (MCSS) is disclosed for storage system supporting a doubly mapped redundant array of independent nodes environment. An instance of a MCSS can be run on computing resources of the storage system supporting a doubly mapped redundant array of independent nodes environment, wherein a real node of a real cluster comprises a real storage device that comprises real extents, and wherein extents of the storage system map to a mapped node of a mapped cluster. The instance of the MCSS can be run near to the highest number of real extents participating in a mapped node. This can reduce computing resource consumption associated with running the MCSS farther from the more concentrated real extents.