Abstract: Implementations are provided herein for managing streaming data that appended and multiplexed into a durable write ahead replicated log. By writing data in the durable log, large amounts of small writes can be processed quickly. Data in the durable log can be de-multiplexed and packaged into segment containers. Segment containers can serialize stream segment specific data and can be stored in long term storage. Data that has been stored in long term storage can be truncated from the durable write ahead log making room for new data.

Abstract: Implementations are provided herein for managing streaming data that appended and multiplexed into a durable write ahead replicated log. By writing data in the durable log, large amounts of small writes can be processed quickly. Data in the durable log can be de-multiplexed and packaged into segment containers. Segment containers can serialize stream segment specific data and can be stored in long term storage. Data that has been stored in long term storage can be truncated from the durable write ahead log making room for new data.

Abstract: Implementations are provided herein for accepting operations asynchronously in a particular order and efficiently committing them into an append-only log while preserving relative order. Operations that are dependent on one or more operations prior to it in the log will be guaranteed to fail, and not accepted, if any of the prior operations failed. If an operation succeeds, it is guaranteed that all operations it depended on are also successful.

Abstract: Implementations are provided herein for atomically committing related stream data across multiple, distributed resources. Transactions can be established that are distributed across multiple hosts, and their data can be made to appear atomic to an observing process. A master status for the transaction can be used to flag to other processes that the transaction is being committed. A stream to which the transaction is being appended to can be locked until the transaction data is committed in full. It can be appreciated that one an event to a transaction is acknowledged, and the transaction is merged, its commitment is guaranteed.

Abstract: Implementations are provided herein for accepting operations asynchronously in a particular order and efficiently committing them into an append-only log while preserving relative order. Operations that are dependent on one or more operations prior to it in the log will be guaranteed to fail, and not accepted, if any of the prior operations failed. If an operation succeeds, it is guaranteed that all operations it depended on are also successful.

Abstract: Implementations are provided herein for executive streaming data writes without duplication or loss. A client application and a pravega node can work to track where write data is, how much data has been written, and what specific data has been acknowledged by the pravega node as successfully written. In the event of an error or connection disruption, the client application can reconnect and determine how much data has been written and resend what data still needs to be written. The data can be written exactly once, and once written and acknowledged, will no longer be subject to data loss.

Abstract: Implementations are provided herein for organizing present and future reads from a tiered streaming data storage layer. Implementations allow for access to multi-tiered streaming data organized in different append-only segments, some of which may be related to each other. Streaming data can be read from fast local tier 1 storage, streaming data can be retrieved from fold tier 2 storage, and registrations can be made to read streaming data that has not yet been written to the storage layer.

Abstract: Implementations are provided herein for auto-scaling a set of stream segments to a stream. In one implementation, the amount of stream segments can be scaled up and down depending on the amount of data ingested from writers. As the number of stream segments change, writers can have their streaming data transition to a newly merged stream segment or a newly split stream segment. The defined ordering of data as written by the writer is preserved as stream segments are scaled. It can be appreciated that a dynamically scaled stream can offer more capacity than any individual host can provide, while still preserving data order.

Abstract: Implementations are provided herein for managing streaming data that appended and multiplexed into a durable write ahead replicated log. By writing data in the durable log, large amounts of small writes can be processed quickly. Data in the durable log can be de-multiplexed and packaged into segment containers. Segment containers can serialize stream segment specific data and can be stored in long term storage. Data that has been stored in long term storage can be truncated form the durable write ahead log making room for new data.

Abstract: A configurable web crawler allows a user configure a web crawl by specifying one or more of thread throttling rules, domain restriction rules, and crawling rules. The configurable web crawler crawls the web beginning with a seed uniform resource locator and according to the crawl configuration.

Abstract: Methods, systems, and computer media implement a statistical spell checker for extracting suggested spell-check candidates for a query containing an unrecognized word. Vocabulary statistics are maintained, including recording a plurality of adjacent word sequences found in a document corpus. When a user query is received that contains a word not in the vocabulary database, i.e., an unrecognized word, the vocabulary statistics are consulted to find word sequences containing the same preceding word and/or succeeding word. The found word sequences may be returned in order based upon the conditional probability that given the recognized preceding and/or succeeding word(s), the unrecognized word is meant to be the suggested spell-checked word.

Abstract: A configurable web crawler allows a user configure a web crawl by specifying one or more of thread throttling rules, domain restriction rules, and crawling rules. The configurable web crawler crawls the web beginning with a seed uniform resource locator and according to the crawl configuration.