Abstract: A system of managing data, comprising a streaming data storage system operating on a processor that causes the processor to receive streaming data and to process the streaming data in response to an append command, a long-term storage system operating on a processor that causes the processor to receive the processed streaming data from the streaming data storage system and to generate a chunk of data, an object storage system operating on a processor that causes the processor to receive the processed streaming data from the streaming data storage system and to generate an object of data and a tables system operating on a processor that causes the processor to receive the processed streaming data from the streaming data storage system and to generate a table of data.

Abstract: Facilitation of reclaiming of storage space is enabled relative to one or more data streams employing the storage space. A system can comprise a processor, and a memory that stores computer executable instructions that, when executed by the processor, facilitate performance of operations. The operations can comprise determining posting or non-posting of one or more specified cut positions from readers of events thus far appended to a stream of a stream storage system. The operations can further comprise, in response to the one or more specified cut positions being posted, truncating the stream based on the specified cut positions of the stream, and, in response to no specified cut positions being posted, truncating the stream based on a time limit or a space limit relative to a respective time quantity or a respective space quantity of the stream.

Abstract: Facilitation of reclaiming of storage space is enabled relative to one or more data streams employing the storage space. A system can comprise a processor, and a memory that stores computer executable instructions that, when executed by the processor, facilitate performance of operations. The operations can comprise determining posting or non-posting of one or more specified cut positions from readers of events thus far appended to a stream of a stream storage system. The operations can further comprise, in response to the one or more specified cut positions being posted, truncating the stream based on the specified cut positions of the stream, and, in response to no specified cut positions being posted, truncating the stream based on a time limit or a space limit relative to a respective time quantity or a respective space quantity of the stream.

Abstract: Systems and methods are disclosed for efficiently indexing stream data to facilitate full-text search of the stream data. A stream comprises large amount of data, only some of which is deemed useful for full-text search indexing. An administrator can specify an indexing specification for a stream. The indexing specification can specify one or more sub-streams within the stream for indexing, and/or specify one or more time intervals of stream data for indexing. A query against the stream can specify the indexing specification to use to index the stream before returning results for the query. The query can alternatively specify an indexing specification to apply to a previously indexed stream. Full-text search indexes generated using an indexing specification can return results that are more relevant to a user because the results are more narrowly focused than an index of, e.g., the entire stream.

Abstract: The present invention extends to methods, systems, and computer program products for increasing coordination service reliability. A coordination service maintains state (e.g., using replication) for one or more software components (e.g., applications). Tokens can be used to identify incarnations of a member set within the coordination service. When a member starts and has no token, the member attempts to learn the token from a majority other members. If no such token exists, the member requests a new token. Aspects of the invention can be used to detect and compensate for lost state within the coordination service, including state lost due to storage device failures (which may be referred to as “silent data loss”). Detecting and compensating for silent data loss makes the coordination service more reliable and can essentially guarantee that the coordination service notifies clients when data is lost and ceases processing when incorrect state may exist.

Abstract: The present invention extends to methods, systems, and computer program products for increasing coordination service reliability. A coordination service maintains state (e.g., using replication) for one or more software components (e.g., applications). Tokens can be used to identify incarnations of a member set within the coordination service. When a member starts and has no token, the member attempts to learn the token from a majority other members. If no such token exists, the member requests a new token. Aspects of the invention can be used to detect and compensate for lost state within the coordination service, including state lost due to storage device failures (which may be referred to as “silent data loss”). Detecting and compensating for silent data loss makes the coordination service more reliable and can essentially guarantee that the coordination service notifies clients when data is lost and ceases processing when incorrect state may exist.