Abstract: The technology describes synchronization of writer and reader applications in a streaming data storage system, such as to facilitate a mode switch in which event writers can change event contents being appended to stream segments and event readers can process the event contents according to the new mode. A signal event is generated by a signaler for stream segments to which data event writes are being written, which changes the data writers' writing mode. The data storage system rejects appends of events after the signal that do not correspond to the new writing mode. The data storage system also writes a signal indicator to the segments being read at a location in each segment that is between the events written before the signal and the events written after the signal. Reader applications are synchronized based on encountering the signal indicator so as to appropriately switch to a new processing mode.

Abstract: The described technology is generally directed towards a streaming data storage system that maintains streamed events, including in a manner that facilitates the ordering of those events based on external ordering (position) data. An event is appended to an event segment, and an index segment maintains offset information to the event relative to the offsets of other previously appended events. The index segment is ordered by position data provided with the event, such that the events (e.g., for a given event routing key) can be read back in position order instead of append order. An ongoing read that returns an event (that matches ongoing read criteria) as the event arrives can also be handled. Also described is the use of a tail index and an indexing queue for more efficient handling of events prior to writing the offset information into the index segment.

Abstract: A streaming data storage system facilitates appending of large events (e.g., up to one gigabyte) to a data segment of a streaming data storage system in a multipart upload operation. A micro-transaction data structure is created for a multipart upload of a large event, to which subparts of the multipart upload are appended during write operations. Order of the subparts is preserved, including when not appended in order, to provide for reading of the subparts in order. An event reference to the micro-transaction data structure is maintained in a data segment corresponding to the large event, and when the event reference encountered during reading, results in reading from the micro-transaction data structure when the multipart upload is complete. The reading from the micro-transaction data structure maintains the order of the large event's subparts, such that raw data is returned to an upstream reader application as the large event.

Abstract: A streaming data storage system maintains streamed events based on external ordering (position) data. Each segment of the stream has minimum position maximum position metadata values based on the events' positions therein. When an ordered event is appended to an ordered segment, an index segment maintains offset information that tracks the event's position order relative to offsets of other previously appended events. When a segment is scaled (split or merged), the predecessor segment set's maximum position is the basis for the successor segment set's minimum position. An event is written to any predecessor segment or successor segment based on the event's routing key and position with respect to a matching segment's routing key and minimum and maximum positions. A data stream can be truncated based on a specified position. Reads can be for events between a specified range of positions, including a range that spans predecessor and successor segments.

Abstract: The technology describes synchronization of writer and reader applications in a streaming data storage system, such as to facilitate a mode switch in which event writers can change event contents being appended to stream segments and event readers can process the event contents according to the new mode. A signal event is generated by a signaler for stream segments to which data event writes are being written, which changes the data writers' writing mode. The data storage system rejects appends of events after the signal that do not correspond to the new writing mode. The data storage system also writes a signal indicator to the segments being read at a location in each segment that is between the events written before the signal and the events written after the signal. Reader applications are synchronized based on encountering the signal indicator so as to appropriately switch to a new processing mode.

Abstract: A streaming data storage system provides for segment groups that facilitate the distribution of event traffic among storage entities. When too much event traffic with the same routing key (or with a small number of routing keys) is being ingested by the system into one segment of a data stream, the system subdivides the segment into a segment group of sub-segments, which can be distributed among storage resources of the streaming data storage system. Writer components of the system append events received from upstream applications to the sub-segments along with a writer identifier and a sequence number for each event. Reader components of the system use writer identifiers and the sequence numbers to serve events from the segment group to upstream code in the proper order, on a per-writer basis.

Abstract: Conditionally appending an event in an ordered event stream (OES) is disclosed. An OES storage system can comprise an OES that can have a dynamic topology, e.g., comprising changing numbers of segments, segments that have changing routing key ranges, mixes of local and remote storage devices and processors, employing different types of storage, or combinations thereof. A conditional append operation can be predicated on determining an appropriate segment, for example, based on a routing key. Conditions of the conditional appends can be designated by a user. Failing conditions can trigger a corrective action, throwing of an error. Satisfied conditions can result in permitting the conditional append. Moreover, the conditional append can be dependent on populating a group of attributes for the correct segment based on attributes that can be determined for a different epoch of the OES, again for example, based on a routing key.

Abstract: The described technology is generally directed towards a streaming data storage system that provides different views of a data stream to different applications based on a per-application data expiration policy. Applications that have different needs related to data lifetime management can be registered to read the same data stream, with each application individually assigned a data expiration time period. When one registered application reads the data stream, that application is presented with a view of the data stream that is subject to that application's expiration time period. When another registered application with a different expiration time period reads the same data stream, the other application is presented with a different view, according to that other application's expiration time period. The time periods for a registered application can be set between a minimum expiration period and a maximum expiration period, or set to a default value between the minimum and maximum.

Abstract: Compaction of an ordered event stream (OES) is disclosed. An OES storage system can employing multiple tiers of storage devices, wherein the different tiers each can provide certain advantages and disadvantages that can be used to balance OES event storage costs, both monetarily and in terms of computing resource burden. Compaction can be facilitated by storing, in a stream map, a reference to an event stored via a second tier of storage in contrast to storing in the stream map actual events of a first tier of storage. A reference can be enabled by storing events in a chunk body and storing a location relationships in a header of the chunk. The stream map can then search headers for references, which can often be smaller than the events themselves, which references can then be communicated, stored, and updated in the stream map to facilitate OES compaction.

Abstract: The described technology is generally directed towards consistently replicating and reconstructing a data stream made up of a dynamic set of (ordered) segments into a different location (e.g., cluster) from the one in which the stream was created. The technology facilitates consistently and generally continuously and replicating a stream of events ingested in a source cluster to a target cluster for consumption (reading). As stream data segments are replicated to a target cluster by a replicator which is not guaranteed to keep the replicated data consistent, a target controller reconstructs the replicated data stream up to a stream cut point at which the replicated data has been sufficiently replicated so as to be consistent. Reading of the replicated data stream is limited to a view up to the stream cut point; as more data is replicated, additional data up to a later stream cut point becomes available for reading.

Abstract: The described technology is generally directed towards a streaming data storage system that provides different views of a data stream to different applications based on a per-application data expiration policy. Applications that have different needs related to data lifetime management can be registered to read the same data stream, with each application individually assigned a data expiration time period. When one registered application reads the data stream, that application is presented with a view of the data stream that is subject to that application's expiration time period. When another registered application with a different expiration time period reads the same data stream, the other application is presented with a different view, according to that other application's expiration time period. The time periods for a registered application can be set between a minimum expiration period and a maximum expiration period, or set to a default value between the minimum and maximum.

Abstract: The described technology is generally directed towards efficiently organizing data in a single, non-distributed database as an ordered associated array of key-value pairs implemented on top of a storage medium that allows only data appends. A table segment in an append-only storage medium is accessed by table keys. The table keys are hashed into key values used to access information in an attribute index (a search tree) that contains offsets to the table segment entries. Hashing the table keys can result in a series of hash parts, including a primary hash part corresponding to a node in the attribute index, and as needed in the event of hash collisions, secondary hash parts that map to child nodes in the attribute index. The table keys are additionally organized in a sorted data structure which is stored within the table segment along with other keys.

Abstract: Aspects of the present disclosure relate to managing data storage resources. In embodiments, one or more data streams are received. Each data stream can include one or more data portions. Further, one or more storage parameters are monitored. Each storage parameter can include data input load of each data stream, data write rate of each stream, number of events in each stream, and data ingestion rates of one or more storage devices. Data storage resources are elastically scales based on any changes to the at least one monitored storage parameter.

Abstract: Described herein are systems and techniques to efficiently cache data for streaming applications. A cache can be organized to include multiple cache segments, and each cache segment can include multiple cache blocks. A cache entry can be created for streaming data, and the streaming data can be streamed directly into a first cache block. When the first cache block is full, a next cache block can be identified, in a same cache segment or in a new cache segment. The streaming data can be streamed directly into the next cache block, and into any further cache blocks as needed.

Abstract: The described technology is generally directed towards efficiently organizing data in a single, non-distributed database as an associated array of key-value pairs implemented on top of a storage medium that allows only data appends. A table segment in an append-only storage medium is accessed by table keys. The table keys are hashed into key values used to access information in an attribute index (a search tree) that contains offsets to the table segment entries. Hashing the table keys can result in a series of hash parts, including a primary hash part corresponding to a node in the attribute index, and as needed in the event of hash collisions, secondary hash parts that map to child nodes in the attribute index.

Abstract: The described technology is generally directed towards efficiently organizing data in a single, non-distributed database as an associated array of key-value pairs implemented on top of a storage medium that allows only data appends. A table segment in an append-only storage medium is accessed by table keys. The table keys are hashed into key values used to access information in an attribute index (a search tree) that contains offsets to the table segment entries. Hashing the table keys can result in a series of hash parts, including a primary hash part corresponding to a node in the attribute index, and as needed in the event of hash collisions, secondary hash parts that map to child nodes in the attribute index.

Abstract: Implementations are provided herein for atomically committing related stream data across multiple, distributed resources. Transactions can be established that are distributed across multiple hosts, and their data can be made to appear atomic to an observing process. A master status for the transaction can be used to flag to other processes that the transaction is being committed. A stream to which the transaction is being appended to can be locked until the transaction data is committed in full. It can be appreciated that one an event to a transaction is acknowledged, and the transaction is merged, its commitment is guaranteed.

Abstract: Described herein are systems and techniques to efficiently cache data for streaming applications. A cache can be organized to include multiple cache segments, and each cache segment can include multiple cache blocks. A cache entry can be created for streaming data, and the streaming data can be streamed directly into a first cache block. When the first cache block is full, a next cache block can be identified, in a same cache segment or in a new cache segment. The streaming data can be streamed directly into the next cache block, and into any further cache blocks as needed.

Abstract: Implementations are provided herein for organizing present and future reads from a tiered streaming data storage layer. Implementations allow for access to multi-tiered streaming data organized in different append-only segments, some of which may be related to each other. Streaming data can be read from fast local tier 1 storage, streaming data can be retrieved from fold tier 2 storage, and registrations can be made to read streaming data that has not yet been written to the storage layer.

Abstract: The described technology is generally directed towards efficiently organizing data in a single, non-distributed database as an associated array of key-value pairs implemented on top of a storage medium that allows only data appends. A table segment in an append-only storage medium is accessed by table keys. The table keys are hashed into key values used to access information in an attribute index (a search tree) that contains offsets to the table segment entries. Hashing the table keys can result in a series of hash parts, including a primary hash part corresponding to a node in the attribute index, and as needed in the event of hash collisions, secondary hash parts that map to child nodes in the attribute index.