Abstract: The technology described herein is directed towards executing continuous query tasks for monitoring a data storage system to obtain real-time metrics and store query results, in which the continuous query tasks are resource-demanding yet need to execute on limited system resources. A continuous query tasks scheduler levels the load produced by scheduling continuous query tasks with starting time offsets based on the period of continuous query tasks and load descriptions for the tasks. Further, a continuous query task that produces too high of a workload is divided into continuous query subtasks, which are scheduled at subtask starting time offsets.

Abstract: Storage of data according to mapped reliability groups (MRGs) of mapped redundant arrays of independent nodes (mapped clusters) is disclosed. Real reliability groups (RRGs) of a real cluster can provide compartmentalization that can reduce a probability of a data loss event occurring in a cluster storage system. MRGs for a mapped cluster employing the RRGs of the real cluster can provide additional compartmentalization that can further reduce a probability of a data loss event occurring. A data protection set can be stored according to a first group of MRGs and a redundant representation of the data protection set can be stored according to a second group of MRGs, wherein a network distance between the first and second group of MRGs is greater than a network distance between constituent MRGs of either the first or second group of MRGs.

Abstract: Annulment of events written into an ordered event stream (OES) storage system is disclosed. Events can be written to one or more segments of an OES. An event can be invalid and overwriting the event can be desirable. A stream-cut event can be employed to facilitate annulment of the event via altering a portion of a segment of an OES. The stream-cut event can be durable. Annulment can avoid rewriting an entire stream and/or complicating a stream by adding mitigating events without overwriting invalid events. In an aspect, annulment can comprise moving a write cursor according to a stream-cut event. The annulment can be lossy or lossless. In an embodiment, annulment can be performed in single tier OES storage system. In another embodiment, multi-tier OES storage systems can facilitate annulment.

Abstract: Allocation of parity data storage extents for a virtual RAID is disclosed. A virtual RAID can be enabled by a virtual storage device pool (VSDP) mapped to a physical storage device pool (PSDP) that can comprise physical extents of physical disks. In an aspect, the physical disks can be co-located, remotely located, or combinations thereof. Mapping of the VSDP can enable allocation of virtual extents in a virtual RAID that can, for example, emulate a RAID4 while still providing parity data storage diversity that can, for example, emulate RAID5, RAID6, etc. Moreover, the disclosed subject matter can support proactive wear leveling, for example, based on historical storage of parity data via an extent, e.g., extents previously used to store parity data. Furthermore, the disclosed subject matter can support active wear leveling.

Abstract: Framed event access in an ordered event stream (OES) storage system is disclosed. Events can be written to one or more segments of an OES and can have an inherent write sequence. Segments can be parallel segments. Reading events from parallel segments can result in a read sequence that does not match the write sequence. This mismatch can be more severe as segment length increases, as event density disparities increase, as access times diverge for different segments, or for numerous other reasons. Event framing can compartmentalize divergence between the write and read sequence. In an aspect, readers in the several segments of the OES can be constrained to read within a frame defined by frame boundaries until all readers have reached the frame boundary, then can advance to a next frame. The restriction can act as a pseudo-synchronization of readers that can mitigate difference between write and read sequences.

Abstract: The technology describes forking (dividing) a data stream into multiple data streams comprising appending a first new data stream and a second new data stream to the original data stream. After forking, the two new data streams have different sets of stream fragments, with one or more of the older stream fragments common to both sets. Data events before the forking operation are read from the original data stream, while data events appended after the forking operation are appended to and read from only one of the data streams, based on which stream each writer/reader is registered to. Also described is truncating the original data stream based on one stream identifier, while leaving the original data intact for another stream identifier. Only when no stream needs a portion of the original data (resulting from separate truncation operations) is the data portion deleted.

Abstract: The described technology is generally directed towards fine-grained data retention management in stream-based data storage systems, which facilitates specifying retention policy at the event level. A fixed retention period corresponding to a retention end time is associated with each event, and the system keeps the longest of the retention end times for each segment of a data stream. Stream cuts are generated which divide the data stream into fragments, and are the basis for truncating the data stream. Each stream cut contains a stream cut retention end time obtained from the longest retention end time of the retention end times of the active segments. When an attempt to truncate a data stream from a stream cut is made, the attempt is blocked if the stream cut's retention end time is in the future, otherwise truncation from the stream cut point is allowed to proceed.

Abstract: The described technology is generally directed towards rebalancing fragments (components) of a protection group among storage entities (e.g., nodes or disks) based on affinity levels in response to a scale out event that expands the storage entities e.g., nodes or disks) of a system. Affinity levels (e.g., maintained in an affinity matrix) represent dependency relationships between the storage entities with respect to storing distributed protection group fragments. A protection group's components are rebalanced based on the new storage entities such that the affinity level between any pair of storage entities is approximately the same as any other pair. In the event of a storage entity failure, as a result of the affinity-based distribution of the protection group components needed for data recovery, a larger number of the other storage entities can be involved in the data recovery. The technology facilitates improved capacity load balancing and faster data recovery.

Abstract: The described technology is generally directed towards recovering a chunk (or similar block of data) when the chunk is erasure coded into fragments, and recovery fragments need to be obtained from geographically distributed sites. The recovery fragments needed to perform recovery of a chunk are determined, and assigned to the geographically distributed sites as subtasks. Each site that receives a subtask from the requesting site obtains XOR-related fragments needed to produce the recovery fragment, performs the XOR operations on the XOR-related fragments to produce the recovery fragment, and returns the recovery fragment to the requesting site. When finished, a site receives another subtask until no subtasks remain, such that the fastest site or sites receive the most subtasks. The requesting site recovers the chunk from the received recovery fragments. The shared participation in the chunk recovery among the distributed sites provides for efficient distribution of the recovery-related resources and work.

Abstract: Watermarking of an ordered event stream (OES) is disclosed. Writing events to an OES can advance populating of the OES, which can be correspond to progress values. Progress values can enable a reader to know where it is in an OES. A reported progress value can be determined from the progress values based on an event being committed to the OES. Moreover, an effective progress value can be determined from the progress values and monitoring the OES for events with that defer being committed to the OES. A watermark for the OES can be determined from the effective progress value such that reading beyond the effective progress value is restricted. This can provide confidence that the reader has read out properly committed events and has not omitted any events having deferred commitment.

Abstract: Compaction of an ordered event stream (OES) is disclosed. An OES storage system can employing multiple tiers of storage devices, wherein the different tiers each can provide certain advantages and disadvantages that can be used to balance OES event storage costs, both monetarily and in terms of computing resource burden. Compaction can be facilitated by storing, in a stream map, a reference to an event stored via a second tier of storage in contrast to storing in the stream map actual events of a first tier of storage. A reference can be enabled by storing events in a chunk body and storing a location relationships in a header of the chunk. The stream map can then search headers for references, which can often be smaller than the events themselves, which references can then be communicated, stored, and updated in the stream map to facilitate OES compaction.

Abstract: Access control for an ordered event stream (OES) storage system is disclosed. Access to a portion of an OES can be controlled at a key-level in relation to a key space of the OES. An application instance can be identified to enable determining a correspondence to one or more keys. The correspondence can be embodied in stored data, for example, via an advanced access control list (AACL) that can be in the form of a list, a table, etc. Application instance access to the portion of the OES can be controlled by determining if an access rule is satisfied, e.g., determining if the key space the application instance wants to access comports with the one or more keys corresponding to the application instance identity. In an aspect, screening data corresponding to the AACL can enable preliminary access screening external to the OES storage system.

Abstract: Selectively distributing fragments of a data protection set in a geographically diverse data storage system is disclosed. The data protection set can comprise fewer fragments than there are zones comprising the geographically diverse data storage system, which can result in some zones not storing a fragment of the data protection set. Control over distribution of fragments of different data protection sets in the geographically diverse data storage system can mitigate or avoid unbalanced storage of the protection sets. The distribution can be controlled in accordance with a protection set distribution scheme (PSDS). A first PSDS can generate coding fragments from randomly select data fragments of all zones. A second PSDS can generate coding fragments from determined unique zone combinations. A third PSDS can generate coding fragments based on affinity values from an affinity matrix.

Abstract: The technology describes synchronization of writer and reader applications in a streaming data storage system, such as to facilitate a mode switch in which event writers can change event contents being appended to stream segments and event readers can process the event contents according to the new mode. A signal event is generated by a signaler for stream segments to which data event writes are being written, which changes the data writers' writing mode. The data storage system rejects appends of events after the signal that do not correspond to the new writing mode. The data storage system also writes a signal indicator to the segments being read at a location in each segment that is between the events written before the signal and the events written after the signal. Reader applications are synchronized based on encountering the signal indicator so as to appropriately switch to a new processing mode.

Abstract: Efficient rolling transactions for a data storage system is described herein. A method as described herein can include structuring, by a device operatively coupled to a processor, a data stream according to a first structure, the first structure comprising respective stream segments that are associated with respective first key ranges; writing, by the device, a transaction to the data stream, wherein the transaction comprises events arranged in a second structure that is distinct from the first structure, and wherein the second structure comprises respective transaction segments that are associated with respective second key ranges; and merging, by the device, respective ones of the transaction segments into respective ones of the stream segments, resulting in merged transaction segments, in response to the second key ranges of the respective merged transaction segments at least partially matching the first key ranges of the respective ones of the stream segments.

Abstract: Drive health information is collected for each one of the data storage drives in one or more RAID groups of data storage drives, and is used to calculate a faultiness level of each data storage drive in the RAID group(s). A suitable RAID level for configuration of at least one RAID group may be generated based on the faultiness levels of the data storage drives contained in the RAID group. A faultiness-balanced distribution of the data storage drives across multiple RAID groups may be generated based on the faultiness levels of individual data storage drives. The data storage drives may be automatically redistributed across the multiple RAID groups according to the faultiness-balanced distribution of the data storage drives.

Abstract: The disclosed technology generally describes a preliminary (e.g., triple mirroring) data protection scheme that operates by writing data as redundant (e.g., three) composite copies made up of copies of data fragments to different nodes of a data storage system. The data fragments are distributed such that any two nodes can fail yet a complete set of data remains among the remaining data fragments. Later, erasure encoding creates redundant coding fragments that are written to the nodes of a data storage system in a distributed manner along with one copy of the data fragments, such that any two nodes can fail but the complete data can still be recovered. Redundant data fragments are then deleted.

Abstract: A streaming data storage system facilitates appending of large events (e.g., up to one gigabyte) to a data segment of a streaming data storage system in a multipart upload operation. A micro-transaction data structure is created for a multipart upload of a large event, to which subparts of the multipart upload are appended during write operations. Order of the subparts is preserved, including when not appended in order, to provide for reading of the subparts in order. An event reference to the micro-transaction data structure is maintained in a data segment corresponding to the large event, and when the event reference encountered during reading, results in reading from the micro-transaction data structure when the multipart upload is complete. The reading from the micro-transaction data structure maintains the order of the large event's subparts, such that raw data is returned to an upstream reader application as the large event.

Abstract: The described technology is generally directed towards a streaming data storage system that compacts older stream data by aggregating that data based on an aggregation function, and replacing the older data with the aggregated data. The technology works with a data stream in different storage tiers (e.g., Tier-1 and Tier-2), provides consistent data stream compaction of data streams having multiple segments, works with different reader applications at different event positons in the data stream, and allows the use of multiple, different aggregation functions. The technology facilitates more efficient compaction for a data stream with a lesser number of routing keys by maintaining per-chunk aggregated data for such data streams.

Abstract: Multistep recovery of chunk fragments of a peer group employing hierarchical erasure coding for geographically diverse data storage protection is disclosed. A peer group of chunks can employ zone-level erasure coding of chunks that can each employ chunk-level erasure coding. In a first iteration, fragment recovery can be performed across peer group chunks based on the zone-level erasure coding. Subsequently, the first iteration can perform recovery of other fragments within a chunk based on the chunk-level erasure coding. Where additional fragments are to be recovered, subsequent iterations can be performed. The disclosed multistep recovery can enable recovery of fragments that would typically have been considered unrecoverable via conventional techniques. Additionally, multistep recovery can enable recovery of fragments across a peer group of chunks that can be more computing resource efficient than recovery of chunks across the peer group of chunks.