Abstract: The described technology is generally directed towards bulk-mode replication of data objects to a destination replication site. When replicating a group of objects, and there is enough object data to fill one or more chunk data structures, bulk-mode data replication is performed, with the destination site replicating the objects into chunk data structures and performing erasure coding to protect the data. This avoids the need for inefficient preliminary data protection scheme at the replication destination. When there are multiple objects to be replicated but not enough object data to fill a chunk data structure, bulk-mode data replication transfers the objects to the destination site for replicating as individual objects. Standard object-level replication can be used for single object replication. The technology thus facilitates high-performance data transfer to destination replication sites and efficient handling of data at the replication destination site.

Abstract: Data retention for a tiered ordered event stream (OES) data storage system is disclosed. Data retention can be applied to demoted data, e.g., data demoted from a first tier to a second tier of data storage devices. The data retention disclosed can avoid demoted events, generally aggregated via a chunk object, file object, etc., from being subject to both a stream-cut operation and a retention period. In embodiments, where a stream-cut instance corresponds to events occurring later in time than a retention end time (RET), all demoted events, e.g., as aggregated by a chunk , file, etc., can be affiliated with the RET. However, where the stream-cut instance corresponds to events occurring earlier in time than the RET, then a portion of the events, e.g., those that will be truncated by performing the stream-cut instance, can be demoted and affiliated with the RET.

Abstract: Allocation of parity data storage extents for a virtual RAID is disclosed. A virtual RAID can be enabled by a virtual storage device pool (VSDP) mapped to a physical storage device pool (PSDP) that can comprise physical extents of physical disks. In an aspect, the physical disks can be co-located, remotely located, or combinations thereof. Mapping of the VSDP can enable allocation of virtual extents in a virtual RAID that can, for example, emulate a RAID4 while still providing parity data storage diversity that can, for example, emulate RAID5, RAID6, etc. Moreover, the disclosed subject matter can support proactive wear leveling, for example, based on historical storage of parity data via an extent, e.g., extents previously used to store parity data. Furthermore, the disclosed subject matter can support active wear leveling.

Abstract: The described technology is generally directed towards managing data retention policy for stream data stored in a streaming storage system. When a request to truncate a data stream from a certain position (e.g., from a request-specified stream cut) is received, an evaluation is made to determine whether the requested position is within a data retention period as specified by data retention policy. If any data prior to the stream cut position (corresponding to a stream cut time) is within the data retention period, the truncation request is blocked. Otherwise truncation from the stream cut point is allowed to proceed/is performed. Also described is handling automated (e.g., sized based) stream truncation requests with respect to data retention.

Abstract: Systems and methods are provided for efficiently configuring an execution environment for an on-demand code execution system to handle a single request (or session) for a single user. Once the session or request is complete, the execution environment is reset, such as by having the hardware processor state, memory, and storage reset. In particular, prior to the execution of code, state of the execution environment of the host computing device is retrieved, such as hardware processor(s), memory, and/or storage state. Moreover, during execution of the code instructions, intermediate state can be gathered. Following the execution of the code, the execution environment is reset based on the saved state related to the hardware processor(s), memory, and/or storage. A subsequent code execution securely occurs in the execution environment and the execution environment is reset again, and so forth.

Abstract: Stream retention in a data storage system is described herein. A method as described herein can include receiving, by a device operatively coupled to a processor, a hold for a portion of a data stream, wherein the data stream comprises respective events indexed by time, and wherein the portion of the data stream is bounded by a first event of the data stream that is associated with a first time and a second event of the data stream that is associated with a second time; generating, by the device, a synthetic stream comprising the portion of the data stream; and preventing, by the device, events of the data stream that are included in the synthetic stream from alteration.

Abstract: The described technology is generally directed towards a stream browser that provides a human-readable (e.g., hierarchical) view of event data for a data stream of a data streaming storage service. A stream browser associated with a data stream maintains routing key and event data for the data stream, e.g., in a primary index. The routing keys can correspond to container structures, e.g., folders and subfolders, with the events associated with each routing key corresponding to hierarchical children of the routing key parent container structures (e.g., like files). A secondary index can be used to evict the oldest events from the primary index to limit the number of data stream events returnable by the stream browser to a practical number.

Abstract: Retention of events of an ordered event stream is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events are added to a head of an OES segment, and pruning events from a tail of the OES segment can be valuable. Processing applications can register a processing scheme for a segment, e.g., at-least-once processing, exactly-once processing, etc., and can generate checkpoints indicating a degree of advancement in processing events of the segment. The ordered event stream can determine a cut point indicative of a progress point, that before which, events of an OES can be marked as ready for expiration. However, events that are marked for expiration can be retained to allow processing based on a checkpoint, e.g., expiration of the event can be refused until there is an assurance the event was read by the processing application.

Abstract: The described technology is generally directed towards cross-stream transactions in a streaming data storage system, which allows a writer application to commit multiple events to distinct data streams in a single transaction. The system creates a cross-stream transaction for a writer application, and the writer application adds events to the cross-stream transaction, indicating which destination data stream(s) each event's data is to be appended. The system adds the event to a subordinate transaction created for each specified data stream. Upon committing the cross-stream transaction, the system coordinates the committing of the subordinate transactions to their respective data streams.

Abstract: Establishing data reliability groups within a geographically distributed data storage environment is presented herein. A system can comprise a processor; and a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising: partitioning geographically distributed data storage zones into reliability groups, in which a reliability group of the reliability groups comprises a group of storage zones comprising a defined amount of the geographically distributed data storage zones; and facilitating a replication of data of the group of storage zones using a portion of remaining data of the group of storage zones.

Abstract: Retention of events of an ordered event stream according to at least one triggered retention policy is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events can be added to a head of an OES segment, and pruning events from a tail of the OES segment can be desirable. Processing applications can predicate event retention, e.g., restricting expiration of an event, on at least one triggered retention policy. In some embodiments, an additional fixed retention policy can be combined with the triggered retention. The disclosed retention can be performed at the event-level or at less granular levels, e.g., segment-level, OES-level, etc., e.g., via batching of events.

Abstract: Systems and methods are described for providing secure storage of data sets while enabling efficient deduplication of data. Each data set can be divided into fixed-length blocks. The plaintext of each block can be convergently encrypted, such as by using a hash of the plaintext as an encryption key, to result in block-level ciphertext that can be stored. If two data sets share blocks, the resulting block-level ciphertext can be expected to overlap, and thus duplicative block-level ciphertexts need not be stored. A manifest can be created to facilitate re-creation of the data set, which manifest identifies the block-level ciphertexts of the data set and a key by which each block-level ciphertext was encrypted. By use of block-level encryption, nearly identical data sets can be largely deduplicated, even if they are not perfectly identical.

Abstract: The described technology is generally directed towards an input/output (I/O) load balancer of a data storage system that detects an I/O overloaded (“hot”) storage unit and logically moves its hot data to a non-overloaded (“cold”) storage unit. Threshold load levels can be used to determine hot and cold storage units. In one implementation, new writes to the hot storage unit are prevented while its hot data is logically moved to a cold storage unit. To avoid reads from the hot storage unit, the hot data can be recreated from redundant data obtained via a recovery path. To avoid a capacity imbalance, once enough hot data has been moved so that the (formerly) hot storage device is no longer considered hot, cold data from the cold storage device can be written to the formerly hot storage device. New data writes to the formerly hot storage device can then resume.

Abstract: Data verification technology for ordered event stream (OES) events written into an ordered event stream storage system is disclosed. The verification technology provides perfect reliability. The verification technology further requires low storage overhead in comparison to typical checksums, storing replicated data, etc. Test event data can be generated in a reproducible manner based upon determined OES metadata. OES metadata can be determined from input received via a user interface, via characteristics of an OES storage system, etc., and can be stored for later use in data verification. The test event data can be stored to a portion of an OES storage system under test. The stored test event data can subsequently be verified by using the stored OES metadata to regenerate test event data for comparison to the stored test event data. The test event ordering can be verified via sequence information included in the stored test event data.

Abstract: The described technology is generally directed towards quasi-listing table entries that facilitate the efficient return of an object list in response to a query for relevant objects in a data storage system. A quasi-listing table entry is created when an object is created, and subsequently converted to a normal listing table entry. A normal listing table entry is converted to a quasi-listing table entry when an object is deleted, with the quasi-listing table entry subsequently deleted. When an object list request is received, a normal listing table entry can be used directly to add to the object list; only if a quasi-listing table entry is encountered (before it can be converted or deleted) does the object table need to be accessed to determine if the object actually exists. This eliminates most object table accesses, including those in remote geographic zones, thereby making object listing significantly more efficient.

Abstract: Systems and methods are provided for scoped credentials within secure execution environments executing within virtual machines instances in an on-demand code execution system. In the on-demand code execution system, the execution environments are reset after every request or session. By resetting the single execution environment after each request or session, security issues are addressed, such as side-channel attacks and persistent malware. Additionally, the use of scoped credentials improves security by limiting the access rights for each code execution request or session to the smallest atomic level for the request or session. Following the request or session, the scoped credential is invalidated.

Abstract: Systems and methods are described for management of garbage collection processes in an on-demand code execution system. An on-demand code execution system may execute user-submitted code on virtual machine instances. As each virtual machine instance executes code, garbage (e.g., memory that is allocated and then not properly de-allocated) may accumulate and may reduce the computing resources that the virtual machine instance makes available for allocation to further code executions. A garbage collection schedule manager may be used to generate a system-wide garbage collection schedule and manage garbage collection processes in accordance with the schedule.

Abstract: Framed event access in an ordered event stream (OES) storage system is disclosed. Events can be written to one or more segments of an OES and can have an inherent write sequence. Segments can be parallel segments. Reading events from parallel segments can result in a read sequence that does not match the write sequence. This mismatch can be more severe as segment length increases, as event density disparities increase, as access times diverge for different segments, or for numerous other reasons. Event framing can compartmentalize divergence between the write and read sequence. In an aspect, readers in the several segments of the OES can be constrained to read within a frame defined by frame boundaries until all readers have reached the frame boundary, then can advance to a next frame. The restriction can act as a pseudo-synchronization of readers that can mitigate difference between write and read sequences.

Abstract: Scaling of an ordered event stream (OES) based on a characteristic of one or more writer groups is disclosed. Scaling a portion of an OES contemporaneous to writing events into that portion can conserve computing resources in contrast to more conventional scaling techniques. Moreover, scaling an OES contemporaneously with writing events thereto can enable improved management of OES scaling for applications that can both read events from an input portion of an OES and, via interim events and interim portions of an OES, write events to an output portion of an OES. An application instance can therefore simultaneously act as both a reader group and writer group and can manage data via interim OESs, such that effects of passing the data through the interim OESs can be cascaded into a scaling of the output portion of an OES based on the writer group characteristic.

Abstract: A method, computer program product, and computer system for identifying, by a computing device, information associated with a relationship between a physical layer block and a virtual logic block for RAID storage. The information associated with the relationship between the physical layer block and the virtual logic block may be written within the RAID storage. The physical layer block within the RAID storage may be rebuilt only when the physical layer block includes the information associated with the relationship between the physical layer block and the virtual logic block.