Abstract: Processing of continuously generated data using a rolling transaction procedure is described. For instance, a system can process a data stream comprising a first segment and a second segment. A transaction associated with the data stream can be initiated and in response to the transaction being initiated, a first transaction segment for the first segment and a second transaction segment for the second segment are generated. Further, a scaling event that modifies the second segment into a third segment and a fourth segment can be detected, and a data stream transaction procedure is executed to end the transaction.

Abstract: Event loading is individually tracked for each of the segments of a stream. Event loading trends of each of the segments of the stream based on event loading changes are identified and reported. Selected segments are automatically scaled-up or scaled-down based on the event loading trends. Individual segment event loading is measured by computing an exponential weighted moving average of data, measured either in events or bytes, received per fixed interval of time. Exponential functions are used to assign exponentially decreasing weights over time. Multiple different running averages are computed for different time intervals, each with a different weight that represents a decay factor.

Abstract: Event loading is individually tracked for each of the segments of a stream. Event loading trends of each of the segments of the stream based on event loading changes are identified and reported. Selected segments are automatically scaled-up or scaled-down based on the event loading trends. Individual segment event loading is measured by computing an exponential weighted moving average of data, measured either in events or bytes, received per fixed interval of time. Exponential functions are used to assign exponentially decreasing weights over time. Multiple different running averages are computed for different time intervals, each with a different weight that represents a decay factor.

Abstract: Reporting time progress for written events on a stream is disclosed. The system comprises operations for obtaining time report information comprising respective timestamp information and respective positional offset information for respective writers with respect to events written to a stream by the respective writers. The operations further comprise aggregating the time report information and determining a lowerbound, e.g., a minimum timestamp before which new events to be written by the stream storage system will no longer arrive, and determining an upperbound, e.g., a stream offset representative of an overall offset of the events written thus far in the stream. The operations further comprise storing the minimum timestamp and the stream offset in a data structure for the events written thus far.

Abstract: Described is a reliable and efficient design for a transactional status oracle (“ReTSO”), which supports large-scale storage systems. ReTSO uses a centralized scheme and implements snapshot isolation, a property that guarantees that read operations read a consistent snapshot of the data store. The centralized scheme of ReTSO enables a lock-free commit algorithm that prevents unreleased locks of a failed transaction from blocking other transactions. Each transaction submits the identifiers of modified rows to the ReTSO, the transaction is committed only if none of the modified rows is committed by a concurrent transaction, and a log of the committed transaction is cached. To recover from failures, before a client is notified of a change, the change cached is first persisted into a write-ahead log, such that the memory state can be fully reconstructed by reading from the write-ahead log. Furthermore, ReTSO implements a lazy, auto-garbage collector hash map that reduces the number of memory accesses per request.

Abstract: A search engine for finding objects that correspond to a search request, including an input module for receiving a keyword query from a user, and a search module being configured to map the keyword query to the identifiers of objects that semantically match the keyword or the plurality of keywords contained in the keyword query, and to generate a search result that contains a listing of matching object identifiers, is characterized in that the search module is further configured to generate the search result by considering network layer information about the user within the process of mapping the keyword query to identifiers of matching objects, wherein the network layer information include sophisticated information the search module receives from a dedicated entity.

Abstract: Briefly, embodiments of methods or systems to replicate indexes are described. According to an embodiment, a method may include executing instructions by one or more processors to bring about generating a first replication threshold of documents to be replicated at a local computing site and a second replication threshold of document entries to be stored in a posting list at the local computing site.

Abstract: Described is a reliable and efficient design for a transactional status oracle (“ReTSO”), which supports large-scale storage systems. ReTSO uses a centralized scheme and implements snapshot isolation, a property that guarantees that read operations read a consistent snapshot of the data store. The centralized scheme of ReTSO enables a lock-free commit algorithm that prevents unreleased locks of a failed transaction from blocking other transactions. Each transaction submits the identifiers of modified rows to the ReTSO, the transaction is committed only if none of the modified rows is committed by a concurrent transaction, and a log of the committed transaction is cached. To recover from failures, before a client is notified of a change, the change cached is first persisted into a write-ahead log, such that the memory state can be fully reconstructed by reading from the write-ahead log. Furthermore, ReTSO implements a lazy, auto-garbage collector hash map that reduces the number of memory accesses per request.

Abstract: Described is a reliable and efficient design for a transactional status oracle (“ReTSO”), which supports large-scale storage systems. ReTSO uses a centralized scheme and implements snapshot isolation, a property that guarantees that read operations read a consistent snapshot of the data store. The centralized scheme of ReTSO enables a lock-free commit algorithm that prevents unreleased locks of a failed transaction from blocking other transactions. Each transaction submits the identifiers of modified rows to the ReTSO, the transaction is committed only if none of the modified rows is committed by a concurrent transaction, and a log of the committed transaction is cached. To recover from failures, before a client is notified of a change, the change cached is first persisted into a write-ahead log, such that the memory state can be fully reconstructed by reading from the write-ahead log. Furthermore, ReTSO implements a lazy, auto-garbage collector hash map that reduces the number of memory accesses per request.

Abstract: Disclosed herein is parallel processing of a query, which uses inter-query parallelism in posting list intersections. A plurality of tasks, e.g., posting list intersection tasks, are identified for processing in parallel by a plurality of processing units, e.g., a plurality of processing cores of a multi-core system.

Abstract: A search engine for finding objects that correspond to a search request, including an input module for receiving a keyword query from a user, and a search module being configured to map the keyword query to the identifiers of objects that semantically match the keyword or the plurality of keywords contained in the keyword query, and to generate a search result that contains a listing of matching object identifiers, is characterized in that the search module is further configured to generate the search result by considering network layer information about the user within the process of mapping the keyword query to identifiers of matching objects, wherein the network layer information include sophisticated information the search module receives from a dedicated entity.

Abstract: In a system for storing and retrieving a plurality of records, the plurality of records associated with a ledger, a client issues read and write requests associated with one of the plurality of records, a plurality of record servers responds to the requests received from the client, and a management server maintains and coordinates, between the client and the record servers, information associated with the ledger, records, and record servers.

Abstract: Disclosed herein are a system, method and architecture for optimizing a manner in which queries and updates are issued to data stores servicing an application such as a social networking application. A social graph modeling a social network may be used to generate a data store request schedule that optimizes, e.g., minimizes, data store accesses, such as without limitation queries and updates to the data store.

Abstract: Briefly, embodiments of methods or systems to replicate indexes in distributed search engines are described.

Abstract: Described is a reliable and efficient design for a transactional status oracle (“ReTSO”), which supports large-scale storage systems. ReTSO uses a centralized scheme and implements snapshot isolation, a property that guarantees that read operations read a consistent snapshot of the data store. The centralized scheme of ReTSO enables a lock-free commit algorithm that prevents unreleased locks of a failed transaction from blocking other transactions. Each transaction submits the identifiers of modified rows to the ReTSO, the transaction is committed only if none of the modified rows is committed by a concurrent transaction, and a log of the committed transaction is cached. To recover from failures, before a client is notified of a change, the change cached is first persisted into a write-ahead log, such that the memory state can be fully reconstructed by reading from the write-ahead log. Furthermore, ReTSO implements a lazy, auto-garbage collector hash map that reduces the number of memory accesses per request.

Abstract: Updated queries are maintained in a cache. A search engine receives a query from a user through a query entry field. The search engine determines search results corresponding to the user query. A new entry mapping the user query to the search results is generated in a cache of results. A web crawler retrieves a new batch of documents for a particular document collection. A search index associated with a search engine is updated to reflect new documents in the document collection. A search engine of queries receives documents from the new batch of documents as inputs. Based on the received documents, the search engine of queries determines which of the queries would have returned the documents as relevant in a search. These queries are determined to be stale and invalidated.

Abstract: Disclosed herein is parallel processing of a query, which uses inter-query parallelism in posting list intersections. A plurality of tasks, e.g., posting list intersection tasks, are identified for processing in parallel by a plurality of processing units, e.g., a plurality of processing cores of a multi-core system.

Abstract: A method of caching posting lists to a search engine cache calculates the ratios between the frequencies of the query terms in a past query log and the sizes of the posting lists for each term, and uses these ratios to determine which posting lists should be cached by sorting the ratios in decreasing order and storing to the cache those posting lists corresponding to the highest ratio values. Further, a method of finding an optimal allocation between two parts of a search engine cache evaluates a past query stream based on a relationship between various properties of the stream and the total size of the cache, and uses this information to determine the respective sizes of both parts of the cache.

Abstract: Embodiments of methods, apparatuses, devices and systems associated with updating stored internet search query result values are disclosed.

Abstract: A computer implemented system for search engine facility architecting and design. The system estimates the costs of power and networking based on system parameters, such as average CPU utilization, connection time, and bytes transferred over the network. Regional distribution of facilities may be evaluated to take into account the various parameters and optimize the cost and speed of the systems being designed. The parameters used in analyzing and formulating an architecture are independent of a particular indexing or query processing technique.