Abstract: Methods and computer storage media are provided for generating an algorithm used to provide preliminary rankings to candidate documents. A final ranking function that provides final rankings for documents is analyzed to identify potential preliminary ranking features, such as static ranking features that are query independent and dynamic atom-isolated components that are related to a single atom. Preliminary ranking features are selected from the potential preliminary ranking features based on many factors. Using these selected features, an algorithm is generated to provide a preliminary ranking to the candidate documents before the most relevant documents are passed to the final ranking stage.

Abstract: A priority hash index provides efficient lookup of posting lists for search query terms. The priority hash index is a data structure in which hash values for terms are distributed across multiple storage devices based on importance of the terms and access speeds of the storage devices. Terms are grouped into search lists with each search list including a storage location on each storage device. When a search query is received, a term is identified and hashed to a location on the first storage device and to generate a unique hash value for the term. The locations on the storage device for the term's search list are sequentially read until the hash value for the term is located to access the posting list for the term.

Abstract: Methods and systems are provided for using a hybrid-distribution system to identify relevant documents based on a search query. A group of documents is assigned to a particular segment. The group of documents is indexed both by atom and by document to form a reverse index and a forward index. Both indexes are divided amongst each node in that segment so that each node is responsible for storing and accessing a different portion of both the reverse and forward indexes. The reverse index portion is accessed on each of a first set of nodes to identify a first set of documents that is relevant to a particular search query. Document identifications associated with the first set of documents are used to identify a second set of nodes that access their forward index portions to limit the number of relevant documents to a second set of documents.

Abstract: Methods and computer storage media are provided for generating an algorithm used to provide preliminary rankings to candidate documents. A final ranking function that provides final rankings for documents is analyzed to identify potential preliminary ranking features, such as static ranking features that are query independent and dynamic atom-isolated components that are related to a single atom. Preliminary ranking features are selected from the potential preliminary ranking features based on many factors. Using these selected features, an algorithm is generated to provide a preliminary ranking to the candidate documents before the most relevant documents are passed to the final ranking stage.

Abstract: Methods are provided for populating search indexes with atoms identified in documents. Documents that are to be indexed are identified, and for each document, atoms are identified and are categorized as unigrams, n-grams, and n-tuples. A list of atom/document pairs is generated such that an information metric can be computed for each pair. An information metric represents a ranking of the atom in relation to the particular document. Based on the information metric, some atom/document pairs are discarded and others are indexed.

Abstract: Search results are identified and returned in response to search queries by evaluating and pruning candidate documents in multiple stages. The process employs a search index that indexes atoms found in documents and pre-computed scores for document/atom pairs. When a search query is received, atoms are identified from the search query and a reformulated query is generated based on the identified atoms. The reformulated query is used to identify matching documents, and a preliminary score is generated for matching documents using a simplified scoring function and pre-computed scores in the search index. Documents are pruned based on preliminary scores, and the remaining documents are evaluated using a final ranking algorithm that provides a final set of ranked documents, which is used to generate search results to return in response to the search query.

Abstract: A preliminary segment root and a final segment root are selected for each segment. Each time a search query is received, a set of nodes in each segment that will be used to resolve the search query is identified. A preliminary segment root is selected from the set of nodes. Based on statistical data from each node in the set of nodes indicating each node's capability to act as a final segment root that assembles query-execution data, the preliminary segment root algorithmically selects the final segment root. The other nodes in the set of nodes are notified regarding the identity of the final segment root.

Abstract: A search index includes tiered posting lists. Each posting list in the search index corresponds with a different atom and includes a list of documents containing the particular document. Additionally, a rank is stored with each document listed in a posting list for a given atom representing the relevance of the atom to the context of each document. At least some of the posting lists in the search index are tiered. A tiered posting list is divided into a number of tiers with the tiers being ordered by document while each tier is internally ordered by document. Employing tiered posting lists within the search index allows a search engine to evaluate search queries in a manner that allows for a number of efficiencies and precise stopping.

Abstract: Methods and computer storage media are provided for generating entries for documents in a forward index. A document and its document identification are received, in addition to static features that are query-independent. The document is parsed into tokens to form a token stream corresponding to the document. Relevant data used to calculate rankings of document is identified and a position of the data is determined. The entry is then generated from the document identification, the token stream of the document, the static features, and the positional information of the relevant data. The entry is stored in the forward index.

Abstract: In a search engine with two-dimensional scalable architecture for searching of a collection of documents, the search engine comprises data processing units which forms set of nodes connected in a network, a first set of nodes comprising dispatch nodes, a second set of nodes search nodes and a third set of nodes indexing nodes. The search nodes are grouped in columns, which via the network are connected in parallel between the dispatch nodes and an indexing node. The dispatch nodes are adapted for processing search queries and search answers, the search nodes are adapted to contain search software, at least some of the search nodes additionally including at least one search processor module and the indexing nodes are adapted for generally generating indexes for the search software. Optionally, acquisition nodes provided in a fourth set of nodes and adapted for processing the search answers, thus relieving the dispatch nodes of this task.

Abstract: A search engine comprising a crawler which crawls the WWW and stores pages found on the WWW in a database. An indexer indexes the pages in the database to produce a primary index. A document mapping section maps pages in the database into a plurality of tiers based on a ranking of the pages. The ranking may be based on portions of the pages which have a relatively higher value context. A processor produces a plurality of sub-indices from the primary index based on the mapping. The sub-indices are stored in a search node cluster. The cluster is a matrix of search nodes logically arranged in a plurality of rows and columns. Search nodes in the same column include the same sub-index. Search nodes in the same row include distinct sub-indices. A search query received by a user is sent to a dispatcher which, in turn, forwards the query to the first tier of search nodes. A fall through algorithm is disclosed which indicates when the dispatcher should forward the search query to other tiers of search nodes.

Abstract: A method for providing searching and alerting capabilities in traffic content at access points in data networks is disclosed. Typical access points for Internet, intranet and wireless traffic are described. Traffic flow through an Internet Service Provider is used as a preferred embodiment to exemplify the data traffic used as the input source in the invention. The invention teaches how proper privacy and content filters can be applied to the traffic source. The filtered data stream from the traffic flow can be used to improve the quality of existing searching and alerting services. The invention also teaches how a cache can be developed optimized for holding fresh searchable information captured in the traffic flow. It is further disclosed how the said cache can be converted to a searchable index and either separately or in cooperation with external search indexes be used as a basis for improved search services.

Abstract: A search system for information retrieval includes a data structure in the form of a non-evenly spaced sparse suffix tree for storing suffixes of words and/or symbols, or sequences thereof, in a text T, a metric M including combined edit distance metrics for an approximate degree of matching respectively between words and/or symbols, or between sequences thereof, in the text T and a query Q, the latter distance metric including weighting cost functions for edit operations which transform a sequence S of the text into a sequence P of the query Q, and search algorithms for determining the degree of matching respectively between words and/or symbols, or between sequences thereof, in respectively the text T and the query Q, such that information R is retrieved with a specified degree of matching with the query Q. Optionally the search system also includes algorithms for determining exact matching such that information R may be retrieved with an exact degree of matching with the query Q.

Abstract: A method for providing searching and alerting capabilities in traffic content at access points in data networks is disclosed. Typical access points for Internet, intranet and wireless traffic are described. Traffic flow through an Internet Service Provider is used as a preferred embodiment to exemplify the data traffic used as the input source in the invention. The invention teaches how proper privacy and content filters can be applied to the traffic source. The filtered data stream from the traffic flow can be used to improve the quality of existing searching and alerting services. The invention also teaches how a cache can be developed optimized for holding fresh searchable information captured in the traffic flow. It is further disclosed how the said cache can be converted to a searchable index and either separately or in cooperation with external search indexes be used as a basis for improved search services.