Abstract: Multistep recovery of chunk fragments of a peer group employing hierarchical erasure coding for geographically diverse data storage protection is disclosed. A peer group of chunks can employ zone-level erasure coding of chunks that can each employ chunk-level erasure coding. In a first iteration, fragment recovery can be performed across peer group chunks based on the zone-level erasure coding. Subsequently, the first iteration can perform recovery of other fragments within a chunk based on the chunk-level erasure coding. Where additional fragments are to be recovered, subsequent iterations can be performed. The disclosed multistep recovery can enable recovery of fragments that would typically have been considered unrecoverable via conventional techniques. Additionally, multistep recovery can enable recovery of fragments across a peer group of chunks that can be more computing resource efficient than recovery of chunks across the peer group of chunks.

Abstract: The described technology is generally directed towards distributing data fragments and coding fragments of a protection group among storage entities (e.g., nodes or disks) based on affinity levels (e.g., maintained in an affinity matrix) that represent dependency relationships between the storage entities with respect to storing protection groups. The technology operates to distribute a protection group's components such that the affinity level between any pair of storage entities is approximately the same as any other pair. In the event of a storage entity failure, as a result of the affinity-based distribution of the protection group components needed for data recovery, a larger number of the other storage entities can be involved in the data recovery (relative to the number likely involved without affinity-based distribution). This tends to assure a better load balance and faster data recovery.

Abstract: The technology describes synchronization of writer and reader applications in a streaming data storage system, such as to facilitate a mode switch in which event writers can change event contents being appended to stream segments and event readers can process the event contents according to the new mode. A signal event is generated by a signaler for stream segments to which data event writes are being written, which changes the data writers' writing mode. The data storage system rejects appends of events after the signal that do not correspond to the new writing mode. The data storage system also writes a signal indicator to the segments being read at a location in each segment that is between the events written before the signal and the events written after the signal. Reader applications are synchronized based on encountering the signal indicator so as to appropriately switch to a new processing mode.

Abstract: Data convolution for geographically diverse storage is disclosed. Data and corresponding convolutions of data can employ erasure coding to improve robustness of access to information represented in the data. For a peer group of chunks employing a given erasure coding scheme, access to the information represented in the data can be via accessible chunks and/or recovery of a less-accessible chunk, e.g., via a deconvolution operation, via a decoding operation, via a mix of deconvolution and decoding operations. The mix of deconvolution and decoding operations can enable recovery of a less-accessible chunk that cannot be recovered by either a deconvolution or decoding operation alone. This can improve access to information represented in less-available data.

Abstract: Data convolution for geographically diverse storage is disclosed, wherein the data convolution is based on a valency metric, determination, parameter, etc. Valency can be correlated to a data object and can be inherited by a chunk comprising the data object. Valency can indicate permitted levels of convolution for a chunk. A chunk resulting from convolving at least two other contributing/participating chunks can be determined to have a valency that can be less than either of the two contributing/participating chunks. A chunk with a zero valence can be restricted from contributing to a convolution. Where the at least two other chunks both have infinite valency, the resulting convolved chunk can also have infinite valency. A chunk resulting from convolving participating chunks having mixed valency values can be reconvolved. Valency can enable orchestrating convolution of chunks, for example, to conserving storage space, mitigate computing resource consumption, etc.

Abstract: Improved scaling of an ordered event stream (OES) is disclosed. In contrast to conventional scaling of an OES that, in immediate response to loading exceeding a given processor performance level, merely divides a segment into segments having similar key space size, and then determines an alternate OES topology. The alternate OES topology can be selected from among ranked alternate OES topologies. The alternate OES topology can be implemented where the expected performance will meet a threshold level of improvement over an existing OES topology. Moreover, the alternate OES topology of the disclosed subject matter can comprise two or more two new segments that can have dissimilar key space sizes. Additionally, the two or more two new segments of the alternate OES topology can provide the same, or similar, loading relative to performance levels of corresponding processing instances, even where the performance levels of corresponding processing instances are also dissimilar.

Abstract: Merging of portions of ordered event streams is disclosed. The disclosed merging of events can limit loss of order of events from streams in exchange for reduced computational load by grouping events according to a pseudo-epoch, wherein events are stored according to a scheme, even though the grouping can result reading events in a different order that that in which the events were written. However, by grouping the events, there can be fewer transitions between storage schemes when reading events than if they were read in the same order in which they were written, thereby reducing computational load. Moreover, restraints on the loss of order can be imposed by selecting a maximum progress window and generally restricting groups from comprising events two different storage schemes. Where events can be moved to archival storage, reducing storage scheme transitions can be of further benefit and speed access times of archived events.

Abstract: A mapped RAID system with a protection group of physical data storage drives calculates an affinity value for each pair of physical data storage drives in the protection group that is equal to a total number of RAID extents to which at least one drive extent is allocated from each of the physical data storage drives in the pair. A change from a previous level of RAID protection to a new level of RAID protection provided by one of the RAID extents is detected. The new level of RAID protection provided by the RAID extent uses fewer drive extents than the previous level of RAID protection provided by the RAID extent. The disclosed technology selects one of the drive extents previously allocated to the RAID extent for deallocation such that the range of the affinity values is minimized after deallocation of the drive extent.

Abstract: A size of a metadata-based system snapshot is determined based on performing incremental calculations in background mode to assure accurate size reporting without producing considerable workload. Snapshot sizes are relative to each other. For example, a size of a first snapshot is set when a second snapshot is generated. Moreover, a size of the latest snapshot is relative to a current tree version and a size of an older snapshot is relative to a snapshot that follows it. An aggregated size of all system snapshots can be determined by adding the individual snapshot sizes. This provides simplicity during snapshot management.

Abstract: Systems and methods are described for providing storage of encrypted data sets, deduplication of such data sets, and control of the redundancy of those data sets. A form of modified convergent encryption can be employed, whereby an encryption key for a data set is selected based on a combination of the plaintext of the data set and a salt value, with the salt value being selected from a number of permutations corresponding to a desired redundancy of the data set in a storage system. Accordingly, a given data set can result in a number of ciphertexts equal to the desired redundancy, and deduplication can occur by removing duplicative instances of individual ciphertexts. Salt values can be selected according to a variety of criteria, including user-based, time-based, and location-based criteria.

Abstract: The described technology is generally directed towards a streaming data storage system that provides different views of a data stream to different applications based on a per-application data expiration policy. Applications that have different needs related to data lifetime management can be registered to read the same data stream, with each application individually assigned a data expiration time period. When one registered application reads the data stream, that application is presented with a view of the data stream that is subject to that application's expiration time period. When another registered application with a different expiration time period reads the same data stream, the other application is presented with a different view, according to that other application's expiration time period. The time periods for a registered application can be set between a minimum expiration period and a maximum expiration period, or set to a default value between the minimum and maximum.

Abstract: Commutative coding in a geographically diverse data storage system is disclosed. Commutative coding can achieve a same result as more conventional hierarchical erasure coding of data, but can be more efficient. Commutative coding can employ Galois Field (GF) based bit-matrix operations. The bit-matrix operations can employ a reduced GF order in associated with expanding elements of input matrixes. A reduced GF order can perform matrix operations at a lower complexity, e.g., employing AND operations for a GF(2) in contrast to XOR operations for a GF(2w), where w=4, 8, 16, etc. In an aspect, commutative coding can comprise generating a second-tier coding fragment based on applying a second erasure coding scheme, via bit-matrix operations, to a first-tier encoded fragment, wherein the first-tier encoded fragment is based on an input data fragment and a first erasure coding scheme.

Abstract: Compaction of an ordered event stream (OES) is disclosed. An OES storage system can employing multiple tiers of storage devices, wherein the different tiers each can provide certain advantages and disadvantages that can be used to balance OES event storage costs, both monetarily and in terms of computing resource burden. Compaction can be facilitated by storing, in a stream map, a reference to an event stored via a second tier of storage in contrast to storing in the stream map actual events of a first tier of storage. A reference can be enabled by storing events in a chunk body and storing a location relationships in a header of the chunk. The stream map can then search headers for references, which can often be smaller than the events themselves, which references can then be communicated, stored, and updated in the stream map to facilitate OES compaction.

Abstract: The described technology is generally directed towards a streaming data storage system that provides different views of a data stream to different applications based on a per-application data expiration policy. Applications that have different needs related to data lifetime management can be registered to read the same data stream, with each application individually assigned a data expiration time period. When one registered application reads the data stream, that application is presented with a view of the data stream that is subject to that application's expiration time period. When another registered application with a different expiration time period reads the same data stream, the other application is presented with a different view, according to that other application's expiration time period. The time periods for a registered application can be set between a minimum expiration period and a maximum expiration period, or set to a default value between the minimum and maximum.

Abstract: The disclosed technology generally describes efficient garbage collection operations based on chunk filters (e.g., Bloom filters) maintained in association with object table partitions (e.g., trees). When an object update impacts a chunk, a corresponding chunk filter is updated, using the chunk's identifier as a chunk filter key by hashing the chunk identifier into chunk filter locations and setting bits at those locations. When garbage collection occurs, a garbage collection candidate chunk's identifier is similarly hashed to read the bit values at those locations in the chunk filters for the partitions. If any bit is zero, the chunk is definitely not in use in a partition; if no chunk filter for any partition indicates the candidate chunk is possibly in use, the chunk is garbage collected without tree scanning. If tree scanning is needed, only the partition(s) with an associated chunk filter that indicates “possibly in use” need to be scanned.

Abstract: Access control for an ordered event stream (OES) storage system is disclosed. Access to a portion of an OES can be controlled at a key-level in relation to a key space of the OES. An application instance can be identified to enable determining a correspondence to one or more keys. The correspondence can be embodied in stored data, for example, via an advanced access control list (AACL) that can be in the form of a list, a table, etc. Application instance access to the portion of the OES can be controlled by determining if an access rule is satisfied, e.g., determining if the key space the application instance wants to access comports with the one or more keys corresponding to the application instance identity. In an aspect, screening data corresponding to the AACL can enable preliminary access screening external to the OES storage system.

Abstract: Retention of events of an ordered event stream according to at least one triggered retention policy is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events can be added to a head of an OES segment, and pruning events from a tail of the OES segment can be desirable. Processing applications can predicate event retention, e.g., restricting expiration of an event, on at least one triggered retention policy. In some embodiments, an additional fixed retention policy can be combined with the triggered retention. The disclosed retention can be performed at the event-level or at less granular levels, e.g., segment-level, OES-level, etc., e.g., via batching of events. Triggers can be affirmative or negative triggers, triggered/fixed retention windows can be combined, etc., facilitating retention policies that can enable event pruning that can be more nuanced than conventional retention techniques.

Abstract: Systems and methods are described for providing secure storage of data sets while enabling efficient deduplication of data. Each data set can be divided into fixed-length blocks. The plaintext of each block can be convergently encrypted, such as by using a hash of the plaintext as an encryption key, to result in block-level ciphertext that can be stored. If two data sets share blocks, the resulting block-level ciphertext can be expected to overlap, and thus duplicative block-level ciphertexts need not be stored. A manifest can be created to facilitate re-creation of the data set, which manifest identifies the block-level ciphertexts of the data set and a key by which each block-level ciphertext was encrypted. By use of block-level encryption, nearly identical data sets can be largely deduplicated, even if they are not perfectly identical.

Abstract: Systems and methods are described for providing storage of encrypted data sets, deduplication of such data sets, and control of the redundancy of those data sets. A form of modified convergent encryption can be employed, whereby an encryption key for a data set is selected based on a combination of the plaintext of the data set and a salt value, with the salt value being selected from a number of permutations corresponding to a desired redundancy of the data set in a storage system. Accordingly, a given data set can result in a number of ciphertexts equal to the desired redundancy, and deduplication can occur by removing duplicative instances of individual ciphertexts. Salt values can be selected according to a variety of criteria, including user-based, time-based, and location-based criteria.

Abstract: Verifiable intra-cluster migration (VICM) for a chunk storage system is disclosed. VICM can migrate data from a first portion of a cluster to a second portion of a cluster. VICM can comprise locking a first portion of a cluster and locking a corresponding first cluster table during a preparation phase. Chunks of the first portion can then be migrated, during a migration phase, to the second portion and a second cluster table, corresponding to the second portion, can be updated accordingly. Garbage management operations, including recovery operations, can be performed via the second cluster table and the second portion during the migration phase. Upon completion of the migration phase, a reconciliation phase can comprise verifying chunk relationships of the second cluster table and the second portion based on the first cluster table. Exceptions to the verification can be reported via an exception report.