Abstract: Data verification technology for ordered event stream (OES) events written into an ordered event stream storage system is disclosed. The verification technology provides perfect reliability. The verification technology further requires low storage overhead in comparison to typical checksums, storing replicated data, etc. Test event data can be generated in a reproducible manner based upon determined OES metadata. OES metadata can be determined from input received via a user interface, via characteristics of an OES storage system, etc., and can be stored for later use in data verification. The test event data can be stored to a portion of an OES storage system under test. The stored test event data can subsequently be verified by using the stored OES metadata to regenerate test event data for comparison to the stored test event data. The test event ordering can be verified via sequence information included in the stored test event data.

Abstract: Data verification technology for ordered event stream (OES) events written into an ordered event stream storage system is disclosed. The verification technology provides perfect reliability. The verification technology further requires low storage overhead in comparison to typical checksums, storing replicated data, etc. Test event data can be generated in a reproducible manner based upon determined OES metadata. OES metadata can be determined from input received via a user interface, via characteristics of an OES storage system, etc., and can be stored for later use in data verification. The test event data can be stored to a portion of an OES storage system under test. The stored test event data can subsequently be verified by using the stored OES metadata to regenerate test event data for comparison to the stored test event data. The test event ordering can be verified via sequence information included in the stored test event data.

Abstract: Data migration from a legacy data storage system into an ordered event stream (OES) data storage system is disclosed. In contrast to conventional techniques, the disclosed subject matter can provide for migrating legacy data into a first portion of a vintage OES (VOES) and new data, intended for the legacy data storage system during the migration process, into a second portion of the VOES. The first and second portions of the VOES can then be linked at completion of the migration process. Avoiding writing new data to the legacy data storage system prior to migration enables a corresponding reduction in data access disruption. Moreover, event sequencing by separating migrated legacy events from new events can preserving event order. Scaling of vintage streams can be supported. Vintage streams can support function calls to a single VOES in contrast to the more conventional use of multiple streams.

Abstract: Data migration from a legacy data storage system into an ordered event stream (OES) data storage system is disclosed. In contrast to conventional techniques, the disclosed subject matter can provide for migrating legacy data into a first portion of a vintage OES (VOES) and new data, intended for the legacy data storage system during the migration process, into a second portion of the VOES. The first and second portions of the VOES can then be linked at completion of the migration process. Avoiding writing new data to the legacy data storage system prior to migration enables a corresponding reduction in data access disruption. Moreover, event sequencing by separating migrated legacy events from new events can preserving event order. Scaling of vintage streams can be supported. Vintage streams can support function calls to a single VOES in contrast to the more conventional use of multiple streams.

Abstract: Event prioritization for an ordered event stream (OES) is disclosed. Unlike conventional prioritization techniques, the disclosed subject matter can be performed by an OES data storage system to provide direct, rather than indirect, control of prioritization. In an embodiment, a prioritized hashed key (PHK) can be determined from an event characteristic and an indicated event priority value based on a selectable priority-sensitive hashing function. As such, events with a same key characteristic but different indicated priorities can have different PHKs, events with different key characteristics but the same indicated priority can have different PHKs, and events with the same key characteristic and the same priority can have a same PHK. An event priority can be inherently comprised in the PHK without needing to explicitly store the priority value with a written event in the OES. Moreover, the disclosed prioritization for the OES can be compatible with OES scaling techniques.

Abstract: The technology described herein is directed towards executing continuous query tasks for monitoring a data storage system to obtain real-time metrics and store query results, in which the continuous query tasks are resource-demanding yet need to execute on limited system resources. A continuous query tasks scheduler levels the load produced by scheduling continuous query tasks with starting time offsets based on the period of continuous query tasks and load descriptions for the tasks. Further, a continuous query task that produces too high of a workload is divided into continuous query subtasks, which are scheduled at subtask starting time offsets.

Abstract: Annulment of events written into an ordered event stream (OES) storage system is disclosed. Events can be written to one or more segments of an OES. An event can be invalid and overwriting the event can be desirable. A stream-cut event can be employed to facilitate annulment of the event via altering a portion of a segment of an OES. The stream-cut event can be durable. Annulment can avoid rewriting an entire stream and/or complicating a stream by adding mitigating events without overwriting invalid events. In an aspect, annulment can comprise moving a write cursor according to a stream-cut event. The annulment can be lossy or lossless. In an embodiment, annulment can be performed in single tier OES storage system. In another embodiment, multi-tier OES storage systems can facilitate annulment.

Abstract: The technology described herein is directed towards executing continuous query tasks for monitoring a data storage system to obtain real-time metrics and store query results, in which the continuous query tasks are resource-demanding yet need to execute on limited system resources. A continuous query tasks scheduler levels the load produced by scheduling continuous query tasks with starting time offsets based on the period of continuous query tasks and load descriptions for the tasks. Further, a continuous query task that produces too high of a workload is divided into continuous query subtasks, which are scheduled at subtask starting time offsets.

Abstract: Techniques are provided for a monitoring subsystem for computer systems. In an example, a plurality of time series databases (TSDBs) can determine monitoring information for a plurality of computing nodes. A metrics reporting server can maintain an availability history for each TSDB that it communicates with. The metrics reporting server can implement a greedy heuristic to determine which TSDBs to query for a given time window. The metrics reporting server can use the responses from these queries to assemble monitoring information for the time window.

Abstract: Embodiments of the present disclosure relate to a system, a computer program product and a method for synchronizing data between a source disk and a target disk in a cluster by performing synchronization between a source disk and a target disk, the synchronization being performed while a plurality of application I/Os on a plurality of nodes in a cluster are configured to access the source disk; and wherein a coordinator and a plurality of workers in the cluster are configured to manage copying data from the source disk to the target disk.

Abstract: Processing load and incoming inter-zone network traffic for remote zones during geographically distributed (GEO) erasure coding can be reduced. In one aspect, GEO erasure coding can be performed in multiple phases, wherein each phase can be distributed between zones. Moreover, during a first phase, partially-finished coding chunks can be generated by frontend zones. Further, during an intermediate phase, partially-finished coding chunks that are associated with a common coding chunk can be transferred between the frontend zones and combined to generate consolidated partial coding chunks. Furthermore, during a final phase, the remote zones can receive and combine the consolidated partial coding chunks to generate a complete coding chunk that can be utilized for data recovery.

Abstract: Data lifetime-aware migration is presented herein. A data sorting component can generate a list of data portions that have been sorted in ascending order of respective determined probabilities that the data portions will be deleted from a source storage device during a migration of the data portions from the source storage device to a destination storage device—the migration comprising copying at least a part of the data portions from the source storage device to the destination storage device. Further, a data lifetime-aware component can iteratively migrate, based on the list in the ascending order of the respective determined probabilities, at least the part of the data portions from the source storage device to the destination storage device.

Abstract: Data lifetime-aware migration is presented herein. A data sorting component can generate a list of data portions that have been sorted in ascending order of respective determined probabilities that the data portions will be deleted from a source storage device during a migration of the data portions from the source storage device to a destination storage device—the migration comprising copying at least a part of the data portions from the source storage device to the destination storage device. Further, a data lifetime-aware component can iteratively migrate, based on the list in the ascending order of the respective determined probabilities, at least the part of the data portions from the source storage device to the destination storage device.

Abstract: Processing load and incoming inter-zone network traffic for remote zones during geographically distributed (GEO) erasure coding can be reduced. In one aspect, GEO erasure coding can be performed in multiple phases, wherein each phase can be distributed between zones. Moreover, during a first phase, partially-finished coding chunks can be generated by frontend zones. Further, during an intermediate phase, partially-finished coding chunks that are associated with a common coding chunk can be transferred between the frontend zones and combined to generate consolidated partial coding chunks. Furthermore, during a final phase, the remote zones can receive and combine the consolidated partial coding chunks to generate a complete coding chunk that can be utilized for data recovery.

Abstract: A method, computer program product, and computer system for performing, via a first computing device, a copy sweep operation to a first range of data on a source storage device. It may be determined that the copy sweep operation has failed. A message may be sent to a second computing device to suspend I/O operations to the first range of data. The copy sweep operation may be retried based upon, at least in part, determining that the copy sweep operation has failed, wherein the copy sweep operation may be retried without the first computing device receiving acknowledgement that the second computing device is suspending the I/O operations to the first range of data.

Abstract: Embodiments of the present disclosure relate to a system, a computer program product and a method for synchronizing data between a source disk and a target disk in a cluster by performing synchronization between a source disk and a target disk, the synchronization being performed while a plurality of application I/Os on a plurality of nodes in a cluster are configured to access the source disk; and wherein a coordinator and a plurality of workers in the cluster are configured to manage copying data from the source disk to the target disk.

Abstract: Embodiments of the present disclosure relate to a system, a computer program product and a method for synchronizing data between a source disk and a target disk in a cluster by performing synchronization between a source disk and a target disk, the synchronization being performed while a plurality of application I/Os on a plurality of nodes in a cluster are configured to access the source disk; and wherein a coordinator and a plurality of workers in the cluster are configured to manage copying data from the source disk to the target disk.

Abstract: In one aspect, a method, includes designating a range of memory in a source disk to synchronize with a target disk, receiving blocks of data that have changed in the source disk and were cloned to a target disk, marking the blocks of data as synchronized and copying data in portions of the range that are not marked as synchronized in the source disk to the target disk.

Abstract: A method, computer program product, and computer system for performing, via a first computing device, a copy sweep operation to a first range of data on a source storage device. It may be determined that the copy sweep operation has failed. A message may be sent to a second computing device to suspend I/O operations to the first range of data. The copy sweep operation may be retried based upon, at least in part, determining that the copy sweep operation has failed, wherein the copy sweep operation may be retried without the first computing device receiving acknowledgement that the second computing device is suspending the I/O operations to the first range of data.

Abstract: Embodiments of the present disclosure relate to a system, a computer program product and a method for synchronizing data between a source disk and a target disk in a cluster by performing synchronization between a source disk and a target disk, the synchronization being performed while a plurality of application I/Os on a plurality of nodes in a cluster are configured to access the source disk; and wherein a coordinator and a plurality of workers in the cluster are configured to manage copying data from the source disk to the target disk.