Abstract: A method for managing a data stream (DS) includes: receiving a first request from an administrator; obtaining the DS from a streaming storage system (SSS); performing a first analysis of the DS to extract relevant data; making a first determination that the relevant data is worth indexing; generating a set of indexes corresponding at least a portion of the DS; receiving a query about a data item in the DS from a user; sending a second request to a database engine comprising the query; performing a second analysis of the query; making a second determination that a re-indexing of the portion of the DS is not required; identifying an index from the set of indexes that is associated with the query; sending position information of the data item to an API service; retrieving the data item from the SSS; and initiating displaying of the data item to the user.

Abstract: A first user-initiated data write request including stream data is received from an edge device executing in a first zone. The stream data is partitioned into at least one segment and an event included in the stream data is written into the at least one segment based on an event's routing key. A segment container of a segment store hosts the at least one segment. The stream data is transmitted to a tier-1 storage of the first zone in order to provide a low-latency access to the stream data for the first user. Thereafter, the stream data is moved to a tier-2 storage of a second zone for permanent storage of the stream data. When requested by a second user operating in the second zone, a read-only version of the segment container retrieves the stream data from the tier-2 storage.

Abstract: A method for managing stream data includes: receiving, from a first edge device (ED) executing in a first zone, a first user initiated data write request comprising stream data, wherein the stream data is partitioned into at least one segment, wherein an event included in the stream data is written into the at least one segment based on an event's routing key, wherein a first segment container (SC) of a first segment store (SS) hosts the at least one segment, transmitting the stream data to a tier-1 storage; initiating a display notification of the first user about a completion of the request; after the notification has been initiated: moving the stream data to a tier-2 storage.

Abstract: Implementations are provided herein for executive streaming data writes without duplication or loss. A client application and a pravega node can work to track where write data is, how much data has been written, and what specific data has been acknowledged by the pravega node as successfully written. In the event of an error or connection disruption, the client application can reconnect and determine how much data has been written and resend what data still needs to be written. The data can be written exactly once, and once written and acknowledged, will no longer be subject to data loss.

Abstract: Implementations are provided herein for encoding and transmitting streaming data from a client application to a server for storage without prior knowledge of the size of the streaming data. A header can be sent that includes a batch size chunk size. Raw streaming data can be packaged into the chunk. Chunks can be packaged and sent at any time prior to filling up with streaming data, by padding the chunk and including a footer that delineates the amount of raw stream data in the chunk. Chunks that are full can have a footer that delineates the entire chunk is raw stream data. It can be appreciated that you do not need to buffer data on the client side as chunks do not need to be full to send. Latency on processing streaming data can also be reduced by limited or eliminated buffering.

Abstract: Implementations are provided herein for atomically committing related stream data across multiple, distributed resources. Transactions can be established that are distributed across multiple hosts, and their data can be made to appear atomic to an observing process. A master status for the transaction can be used to flag to other processes that the transaction is being committed. A stream to which the transaction is being appended to can be locked until the transaction data is committed in full. It can be appreciated that one an event to a transaction is acknowledged, and the transaction is merged, its commitment is guaranteed.

Abstract: Embodiments enable clients to have read-write access to a stream where all other clients are aware of all changes being made to the stream. A pravega node can track the data that has been written to the stream. The clients can dynamically read the stream. A client among the clients can update the stream by sending a request to the pravega node that includes the update and a total length of the stream that was written to the stream at the time of the last read update by the client. If the total length of the stream received from the client matches the actual length of the stream maintained by the pravega node, the pravega node will update the stream. If not, a failure message can be sent to the client and the client can process more reads to the stream before making another attempt to update the stream.

Abstract: Implementations are provided herein for encoding and transmitting streaming data from a client application to a server for storage without prior knowledge of the size of the streaming data. A header can be sent that includes a batch size chunk size. Raw streaming data can be packaged into the chunk. Chunks can be packaged and sent at any time prior to filling up with streaming data, by padding the chunk and including a footer that delineates the amount of raw stream data in the chunk. Chunks that are full can have a footer that delineates the entire chunk is raw stream data. It can be appreciated that you do not need to buffer data on the client side as chunks do not need to be full to send. Latency on processing streaming data can also be reduced by limited or eliminated buffering.

Abstract: Implementations are provided herein for auto-scaling a set of stream segments to a stream. In one implementation, the amount of stream segments can be scaled up and down depending on the amount of data ingested from writers. As the number of stream segments change, writers can have their streaming data transition to a newly merged stream segment or a newly split stream segment. The defined ordering of data as written by the writer is preserved as stream segments are scaled. It can be appreciated that a dynamically scaled stream can offer more capacity than any individual host can provide, while still preserving data order.

Abstract: Implementations are provided herein for a set of clients to concurrently have read-write access to a stream where all other clients are aware of all changes being made to the stream. A pravega node can track the data that has been written to the stream. The set of clients can dynamically read the stream. A client among the set of clients can update the stream by sending a request to the pravega node that includes the update and a total length of the stream that was written to the stream at the time of the last read update by the client. If the total length of the stream received from the client matches the actual length of the stream maintained by the pravega node, the pravega node will update the stream. If not, a failure message can be sent to the client and the client can process more reads to the stream before making another attempt to update the stream.

Abstract: Implementations are provided herein for executive streaming data writes without duplication or loss. A client application and a pravega node can work to track where write data is, how much data has been written, and what specific data has been acknowledged by the pravega node as successfully written. In the event of an error or connection disruption, the client application can reconnect and determine how much data has been written and resend what data still needs to be written. The data can be written exactly once, and once written and acknowledged, will no longer be subject to data loss.

Abstract: Implementations are provided herein for atomically committing related stream data across multiple, distributed resources. Transactions can be established that are distributed across multiple hosts, and their data can be made to appear atomic to an observing process. A master status for the transaction can be used to flag to other processes that the transaction is being committed. A stream to which the transaction is being appended to can be locked until the transaction data is committed in full. It can be appreciated that one an event to a transaction is acknowledged, and the transaction is merged, its commitment is guaranteed.