Abstract: Data retention for a tiered ordered event stream (OES) data storage system is disclosed. Data retention can be applied to demoted data, e.g., data demoted from a first tier to a second tier of data storage devices. The data retention disclosed can avoid demoted events, generally aggregated via a chunk object, file object, etc., from being subject to both a stream-cut operation and a retention period. In embodiments, where a stream-cut instance corresponds to events occurring later in time than a retention end time (RET), all demoted events, e.g., as aggregated by a chunk, file, etc., can be affiliated with the RET. However, where the stream-cut instance corresponds to events occurring earlier in time than the RET, then a portion of the events, e.g., those that will be truncated by performing the stream-cut instance, can be demoted and affiliated with the RET.

Abstract: Scaling an ordered event stream (OES) based on an information-unit (IU) metric is disclosed. The IU metric can correspond to an amount of computing resources that can be consumed to access information embodied in event data of an event of the OES. In this regard, the amount of computing resources to access the data of the stream event itself can be distinct from an amount of computing resources employed to access information embodied in the data. As such, where an external application, e.g., a reader, a writer, etc., can connect to an OES data storage system, enabling the OES to be scaled in response to burdening of computing resources accessing event information, rather than merely event data, can aid in preservation of an ordering of events accessed from the OES.

Abstract: Storage of data according to mapped reliability groups (MRGs) of mapped redundant arrays of independent nodes (mapped clusters) is disclosed. Real reliability groups (RRGs) of a real cluster can provide compartmentalization that can reduce a probability of a data loss event occurring in a cluster storage system. MRGs for a mapped cluster employing the RRGs of the real cluster can provide additional compartmentalization that can further reduce a probability of a data loss event occurring. A data protection set can be stored according to a first group of MRGs and a redundant representation of the data protection set can be stored according to a second group of MRGs, wherein a network distance between the first and second group of MRGs is greater than a network distance between constituent MRGs of either the first or second group of MRGs.

Abstract: The described technology is generally directed towards fine-grained data event expiration in a streaming data storage system. An event to append is given an expiration period, and the expiration time for the events in a data stream or segment of a data stream is the largest expiration time among events in the data stream or segment. Different segments can have different expiration times for their events. In a segment comprising a group of events, a subgroup of expired events prior to a stream cut are deleted by an expiration task. For a subgroup of unexpired events prior to a stream cut, the expiration task retains (does not delete) the subgroup of events. If a scaling operation is performed on a segment, the new successor segment or segments inherit the largest expiration time of the predecessor segment or segments.

Abstract: A geographically diverse data storage system that can protect data via replication of data among relevant zones according to a determined replication topology is disclosed. The replication topology can be determined based on replication times between the relevant zones. In an aspect, a tree topology can provide advantages over a star topography. In an embodiment, a tree topology can be generated, or an existing topology can be modified, via selection of a next replication task(s) based on the replication times. In an aspect, the replication times can be determined from measurable characteristics of the geographically diverse data storage system. In some embodiments, the replications times can be based on historical measurements, time limited historical measurements, inferences from machine learning, etc. A determined topology can be ranked relative to other viable topologies based on criteria such as speed, monetary cost, computing resource usage, etc.

Abstract: Data retention for a tiered ordered event stream (OES) data storage system is disclosed. Data retention can be applied to demoted data, e.g., data demoted from a first tier to a second tier of data storage devices. The data retention disclosed can avoid demoted events, generally aggregated via a chunk object, file object, etc., from being subject to both a stream-cut operation and a retention period. In embodiments, where a stream-cut instance corresponds to events occurring later in time than a retention end time (RET), all demoted events, e.g., as aggregated by a chunk , file, etc., can be affiliated with the RET. However, where the stream-cut instance corresponds to events occurring earlier in time than the RET, then a portion of the events, e.g., those that will be truncated by performing the stream-cut instance, can be demoted and affiliated with the RET.

Abstract: The described technology is generally directed towards cross-stream transactions in a streaming data storage system, which allows a writer application to commit multiple events to distinct data streams in a single transaction. The system creates a cross-stream transaction for a writer application, and the writer application adds events to the cross-stream transaction, indicating which destination data stream(s) each event's data is to be appended. The system adds the event to a subordinate transaction created for each specified data stream. Upon committing the cross-stream transaction, the system coordinates the committing of the subordinate transactions to their respective data streams.

Abstract: Framed event access in an ordered event stream (OES) storage system is disclosed. Events can be written to one or more segments of an OES and can have an inherent write sequence. Segments can be parallel segments. Reading events from parallel segments can result in a read sequence that does not match the write sequence. This mismatch can be more severe as segment length increases, as event density disparities increase, as access times diverge for different segments, or for numerous other reasons. Event framing can compartmentalize divergence between the write and read sequence. In an aspect, readers in the several segments of the OES can be constrained to read within a frame defined by frame boundaries until all readers have reached the frame boundary, then can advance to a next frame. The restriction can act as a pseudo-synchronization of readers that can mitigate difference between write and read sequences.

Abstract: Disk access event control for mapped nodes of a cluster storage system supporting a redundant array of independent nodes (mapped RAIN) system is disclosed. A mapped RAIN cluster can be allocated on top of one or more real data clusters. In an embodiment, disk access events can be routed via a storage service instance supporting a mapped node. In another embodiment, disk access events can be routed via another storage service instance that does not support the mapped node. Routing the disk access event via another storage service instance that does not support the mapped node can reduce the use of computing resources. Further, the routing of the disk access event can be according to a proportional disk operation value determined based on historical disk access event characteristics.

Abstract: The described technology is generally directed towards replicating metadata representing a virtual data structure corresponding to replicated legacy data instead of the actual data for the data structure. Once virtual chunks are replicated to a remote, newer storage system, the corresponding legacy data is locally read into the virtual chunks to transform the virtual chunks into real data chunks of the remote newer storage system. A checksum can be replicated for the remote newer storage system to evaluate the consistency of the data. Efficient data storage migration is thus accomplished in a replicated environment based on relatively negligible replication traffic between two remote locations, while still assuring the consistency of migrated data.

Abstract: The described technology is generally directed towards cross-stream transactions in a streaming data storage system, which allows a writer application to commit multiple events to distinct data streams in a single transaction. The system creates a cross-stream transaction for a writer application, and the writer application adds events to the cross-stream transaction, indicating which destination data stream(s) each event's data is to be appended. The system adds the event to a subordinate transaction created for each specified data stream. Upon committing the cross-stream transaction, the system coordinates the committing of the subordinate transactions to their respective data streams.

Abstract: Scaling an ordered event stream (OES) based on an information-unit (IU) metric is disclosed. The IU metric can correspond to an amount of computing resources that can be consumed to access information embodied in event data of an event of the OES. In this regard, the amount of computing resources to access the data of the stream event itself can be distinct from an amount of computing resources employed to access information embodied in the data. As such, where an external application, e.g., a reader, a writer, etc., can connect to an OES data storage system, enabling the OES to be scaled in response to burdening of computing resources accessing event information, rather than merely event data, can aid in preservation of an ordering of events accessed from the OES.

Abstract: The described technology is generally directed towards fine-grained data event expiration in a streaming data storage system. An event to append is given an expiration period, and the expiration time for the events in a data stream or segment of a data stream is the largest expiration time among events in the data stream or segment. Different segments can have different expiration times for their events. In a segment comprising a group of events, a subgroup of expired events prior to a stream cut are deleted by an expiration task. For a subgroup of unexpired events prior to a stream cut, the expiration task retains (does not delete) the subgroup of events. If a scaling operation is performed on a segment, the new successor segment or segments inherit the largest expiration time of the predecessor segment or segments.

Abstract: Mitigating the effects of a real node failure in a doubly mapped redundant array of independent nodes, e.g., doubly mapped cluster is disclosed. In response to a change in an accessibility to data stored on an extent of a real storage device of a real node of a real cluster, wherein the extent of the real storage device corresponds to a portion of a mapped storage device of a mapped node of a doubly mapped cluster, substituting a reserved extent of a real storage device for the extent of the real storage device. The substituting the reserved extent of the real storage device can correspond to a change in a topology of the doubly mapped cluster, wherein the change in the topology comprises replacing the portion of the mapped storage device with a substitute portion of a mapped storage device that corresponds to the replacement extent of the real storage device.

Abstract: Selective instantiation of a mapped cluster storage service (MCSS) is disclosed for storage system supporting a doubly mapped redundant array of independent nodes environment. An instance of a MCSS can be run on computing resources of the storage system supporting a doubly mapped redundant array of independent nodes environment, wherein a real node of a real cluster comprises a real storage device that comprises real extents, and wherein extents of the storage system map to a mapped node of a mapped cluster. The instance of the MCSS can be run near to the highest number of real extents participating in a mapped node. This can reduce computing resource consumption associated with running the MCSS farther from the more concentrated real extents.

Abstract: A mapped redundant array of independent nodes (mapped RAIN) for data storage is disclosed. A mapped RAIN cluster can be allocated on top of one or more real data clusters, wherein the real clusters can comprise storage devices of different storage capacities. Mapping of data storage locations for a mapped RAIN cluster to real storage devices can be based on an affinity value determined for pairs of real nodes of the real data clusters. A normalized affinity can be employed to enable allocation of real storage to mapped nodes of mapped clusters that can be based on the heterogeneous capacities of the storage devices. This can provide improved data availability and data recovery over other techniques where heterogeneity of hardware can make efficient resource allocation a non-trivial task. The disclosed subject matter can facilitate more efficient allocation of Mapped RAINs in a heterogeneous cluster storage construct.

Abstract: Storage of data according to mapped reliability groups (MRGs) of mapped redundant arrays of independent nodes (mapped clusters) is disclosed. Real reliability groups (RRGs) of a real cluster can provide compartmentalization that can reduce a probability of a data loss event occurring in a cluster storage system. MRGs for a mapped cluster employing the RRGs of the real cluster can provide additional compartmentalization that can further reduce a probability of a data loss event occurring. A data protection set can be stored according to a first group of MRGs and a redundant representation of the data protection set can be stored according to a second group of MRGs, wherein a network distance between the first and second group of MRGs is greater than a network distance between constituent MRGs of either the first or second group of MRGs.

Abstract: Framed event access in an ordered event stream (OES) storage system is disclosed. Events can be written to one or more segments of an OES and can have an inherent write sequence. Segments can be parallel segments. Reading events from parallel segments can result in a read sequence that does not match the write sequence. This mismatch can be more severe as segment length increases, as event density disparities increase, as access times diverge for different segments, or for numerous other reasons. Event framing can compartmentalize divergence between the write and read sequence. In an aspect, readers in the several segments of the OES can be constrained to read within a frame defined by frame boundaries until all readers have reached the frame boundary, then can advance to a next frame. The restriction can act as a pseudo-synchronization of readers that can mitigate difference between write and read sequences.

Abstract: The described technology is generally directed towards rebalancing fragments (components) of a protection group among storage entities (e.g., nodes or disks) based on affinity levels in response to a scale out event that expands the storage entities e.g., nodes or disks) of a system. Affinity levels (e.g., maintained in an affinity matrix) represent dependency relationships between the storage entities with respect to storing distributed protection group fragments. A protection group's components are rebalanced based on the new storage entities such that the affinity level between any pair of storage entities is approximately the same as any other pair. In the event of a storage entity failure, as a result of the affinity-based distribution of the protection group components needed for data recovery, a larger number of the other storage entities can be involved in the data recovery. The technology facilitates improved capacity load balancing and faster data recovery.

Abstract: Watermarking of an ordered event stream (OES) is disclosed. Writing events to an OES can advance populating of the OES, which can be correspond to progress values. Progress values can enable a reader to know where it is in an OES. A reported progress value can be determined from the progress values based on an event being committed to the OES. Moreover, an effective progress value can be determined from the progress values and monitoring the OES for events with that defer being committed to the OES. A watermark for the OES can be determined from the effective progress value such that reading beyond the effective progress value is restricted. This can provide confidence that the reader has read out properly committed events and has not omitted any events having deferred commitment.