Abstract: The described technology is generally directed towards managing data retention policy for stream data stored in a streaming storage system. When a request to truncate a data stream from a certain position (e.g., from a request-specified stream cut) is received, an evaluation is made to determine whether the requested position is within a data retention period as specified by data retention policy. If any data prior to the stream cut position (corresponding to a stream cut time) is within the data retention period, the truncation request is blocked. Otherwise truncation from the stream cut point is allowed to proceed/is performed. Also described is handling automated (e.g., sized based) stream truncation requests with respect to data retention.

Abstract: Retention of events of an ordered event stream according to at least one triggered retention policy is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events can be added to a head of an OES segment, and pruning events from a tail of the OES segment can be desirable. Processing applications can predicate event retention, e.g., restricting expiration of an event, on at least one triggered retention policy. In some embodiments, an additional fixed retention policy can be combined with the triggered retention. The disclosed retention can be performed at the event-level or at less granular levels, e.g., segment-level, OES-level, etc., e.g., via batching of events.

Abstract: Various aspects of the technology described herein are generally directed to systems, methods, and computer storage media for, among other things, providing robotic system services including implementing an enhanced robotics framework. The enhanced robotics framework includes a visual feedback, a skills library, and minting and awarding a fungible token for activities associated with a robot.

Abstract: Described herein, system that facilitates replication of data in a geographically distributed storage environment. According to an embodiment, a system can comprise determining whether a data storage system is in an active data storage mode or a passive data storage mode. In active data storage mode, the data storage system can accept connections from an application, permit the application to store new data objects, replicate the new data objects to other data storage systems, and accept replicated data. In passive data storage mode, the data storage system can block storage of new data objects by the application while accepting replicated data.

Abstract: Commutative coding in a geographically diverse data storage system is disclosed. Commutative coding can achieve a same result as more conventional hierarchical erasure coding of data, but can be more efficient. Commutative coding can employ Galois Field (GF) based bit-matrix operations. The bit-matrix operations can employ a reduced GF order in associated with expanding elements of input matrixes. A reduced GF order can perform matrix operations at a lower complexity, e.g., employing AND operations for a GF(2) in contrast to XOR operations for a GF(2w), where w=4, 8, 16, etc. In an aspect, commutative coding can comprise generating a second-tier coding fragment based on applying a second erasure coding scheme, via bit-matrix operations, to a first-tier encoded fragment, wherein the first-tier encoded fragment is based on an input data fragment and a first erasure coding scheme.

Abstract: The described technology is generally directed towards managing data retention policy for stream data stored in a streaming storage system. When a request to truncate a data stream from a certain position (e.g., from a request-specified stream cut) is received, an evaluation is made to determine whether the requested position is within a data retention period as specified by data retention policy. If any data prior to the stream cut position (corresponding to a stream cut time) is within the data retention period, the truncation request is blocked. Otherwise truncation from the stream cut point is allowed to proceed/is performed. Also described is handling automated (e.g., sized based) stream truncation requests with respect to data retention.

Abstract: The described technology is generally directed towards managing data retention policy for stream data stored in a streaming storage system. When a request to truncate a data stream from a certain position (e.g., from a request-specified stream cut) is received, an evaluation is made to determine whether the requested position is within a data retention period as specified by data retention policy. If any data prior to the stream cut position (corresponding to a stream cut time) is within the data retention period, the truncation request is blocked. Otherwise truncation from the stream cut point is allowed to proceed/is performed. Also described is handling automated (e.g., sized based) stream truncation requests with respect to data retention.

Abstract: The described technology is generally directed towards consistently replicating and reconstructing a data stream made up of a dynamic set of (ordered) segments into a different location (e.g., cluster) from the one in which the stream was created. The technology facilitates consistently and generally continuously and replicating a stream of events ingested in a source cluster to a target cluster for consumption (reading). As stream data segments are replicated to a target cluster by a replicator which is not guaranteed to keep the replicated data consistent, a target controller reconstructs the replicated data stream up to a stream cut point at which the replicated data has been sufficiently replicated so as to be consistent. Reading of the replicated data stream is limited to a view up to the stream cut point; as more data is replicated, additional data up to a later stream cut point becomes available for reading.

Abstract: Retention of events of an ordered event stream is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events are added to a head of an OES segment, and pruning events from a tail of the OES segment can be valuable. Processing applications can register a processing scheme for a segment, e.g., at-least-once processing, exactly-once processing, etc., and can generate checkpoints indicating a degree of advancement in processing events of the segment. The ordered event stream can determine a cut point indicative of a progress point, that before which, events of an OES can be marked as ready for expiration. However, events that are marked for expiration can be retained to allow processing based on a checkpoint, e.g., expiration of the event can be refused until there is an assurance the event was read by the processing application.

Abstract: Stream retention in a data storage system is described herein. A method as described herein can include receiving, by a device operatively coupled to a processor, a hold for a portion of a data stream, wherein the data stream comprises respective events indexed by time, and wherein the portion of the data stream is bounded by a first event of the data stream that is associated with a first time and a second event of the data stream that is associated with a second time; generating, by the device, a synthetic stream comprising the portion of the data stream; and preventing, by the device, events of the data stream that are included in the synthetic stream from alteration.

Abstract: The described technology is generally directed towards a stream browser that provides a human-readable (e.g., hierarchical) view of event data for a data stream of a data streaming storage service. A stream browser associated with a data stream maintains routing key and event data for the data stream, e.g., in a primary index. The routing keys can correspond to container structures, e.g., folders and subfolders, with the events associated with each routing key corresponding to hierarchical children of the routing key parent container structures (e.g., like files). A secondary index can be used to evict the oldest events from the primary index to limit the number of data stream events returnable by the stream browser to a practical number.

Abstract: Retention of events of an ordered event stream according to at least one triggered retention policy is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events can be added to a head of an OES segment, and pruning events from a tail of the OES segment can be desirable. Processing applications can predicate event retention, e.g., restricting expiration of an event, on at least one triggered retention policy. In some embodiments, an additional fixed retention policy can be combined with the triggered retention. The disclosed retention can be performed at the event-level or at less granular levels, e.g., segment-level, OES-level, etc., e.g., via batching of events.

Abstract: Retention of events of an ordered event stream according to at least one triggered retention policy is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events can be added to a head of an OES segment, and pruning events from a tail of the OES segment can be desirable. Processing applications can predicate event retention, e.g., restricting expiration of an event, on at least one triggered retention policy. In some embodiments, an additional fixed retention policy can be combined with the triggered retention. The disclosed retention can be performed at the event-level or at less granular levels, e.g., segment-level, OES-level, etc., e.g., via batching of events.

Abstract: Commutative coding in a geographically diverse data storage system is disclosed. Commutative coding can achieve a same result as more conventional hierarchical erasure coding of data, but can be more efficient. Commutative coding can employ Galois Field (GF) based bit-matrix operations. The bit-matrix operations can employ a reduced GF order in associated with expanding elements of input matrixes. A reduced GF order can perform matrix operations at a lower complexity, e.g., employing AND operations for a GF(2) in contrast to XOR operations for a GF(2w), where w=4, 8, 16, etc. In an aspect, commutative coding can comprise generating a second-tier coding fragment based on applying a second erasure coding scheme, via bit-matrix operations, to a first-tier encoded fragment, wherein the first-tier encoded fragment is based on an input data fragment and a first erasure coding scheme.

Abstract: The described technology is generally directed towards consistently replicating and reconstructing a data stream made up of a dynamic set of (ordered) segments into a different location (e.g., cluster) from the one in which the stream was created. The technology facilitates consistently and generally continuously and replicating a stream of events ingested in a source cluster to a target cluster for consumption (reading). As stream data segments are replicated to a target cluster by a replicator which is not guaranteed to keep the replicated data consistent, a target controller reconstructs the replicated data stream up to a stream cut point at which the replicated data has been sufficiently replicated so as to be consistent. Reading of the replicated data stream is limited to a view up to the stream cut point; as more data is replicated, additional data up to a later stream cut point becomes available for reading.

Abstract: Retention of events of an ordered event stream according to at least one triggered retention policy is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events can be added to a head of an OES segment, and pruning events from a tail of the OES segment can be desirable. Processing applications can predicate event retention, e.g., restricting expiration of an event, on at least one triggered retention policy. In some embodiments, an additional fixed retention policy can be combined with the triggered retention. The disclosed retention can be performed at the event-level or at less granular levels, e.g., segment-level, OES-level, etc., e.g., via batching of events. Triggers can be affirmative or negative triggers, triggered/fixed retention windows can be combined, etc., facilitating retention policies that can enable event pruning that can be more nuanced than conventional retention techniques.

Abstract: Retention of events of an ordered event stream is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events are added to a head of an OES segment, and pruning events from a tail of the OES segment can be valuable. Processing applications can register a processing scheme for a segment, e.g., at-least-once processing, exact1y-once processing, etc., and can generate checkpoints indicating a degree of advancement in processing events of the segment. The ordered event stream can determine a cut point indicative of a progress point, that before which, events of an OES can be marked as ready for expiration. However, events that are marked for expiration can be retained to allow processing based on a checkpoint, e.g., expiration of the event can be refused until there is an assurance the event was read by the processing application.

Abstract: The described technology is generally directed towards managing data retention policy for stream data stored in a streaming storage system. When a request to truncate a data stream from a certain position (e.g., from a request-specified stream cut) is received, an evaluation is made to determine whether the requested position is within a data retention period as specified by data retention policy. If any data prior to the stream cut position (corresponding to a stream cut time) is within the data retention period, the truncation request is blocked. Otherwise truncation from the stream cut point is allowed to proceed/is performed. Also described is handling automated (e.g., sized based) stream truncation requests with respect to data retention.

Abstract: The described technology is generally directed towards a stream browser that provides a human-readable (e.g., hierarchical) view of event data for a data stream of a data streaming storage service. A stream browser associated with a data stream maintains routing key and event data for the data stream, e.g., in a primary index. The routing keys can correspond to container structures, e.g., folders and subfolders, with the events associated with each routing key corresponding to hierarchical children of the routing key parent container structures (e.g., like files). A secondary index can be used to evict the oldest events from the primary index to limit the number of data stream events returnable by the stream browser to a practical number.

Abstract: Stream retention in a data storage system is described herein. A method as described herein can include receiving, by a device operatively coupled to a processor, a hold for a portion of a data stream, wherein the data stream comprises respective events indexed by time, and wherein the portion of the data stream is bounded by a first event of the data stream that is associated with a first time and a second event of the data stream that is associated with a second time; generating, by the device, a synthetic stream comprising the portion of the data stream; and preventing, by the device, events of the data stream that are included in the synthetic stream from alteration.