Abstract: Described is a data-centric web search engine technology/architecture, in which document metadata, including offline-extracted metadata, is used as part of a search indexing and ranking pipeline. A web data management component receives crawled documents and extracts document metadata from the documents. An indexing component uses the document metadata to build an index for the documents. A serving component uses the index and the document metadata to serve content, e.g., search results. Also described is the use of query metadata extracted from queries of a query log for use in the pipeline.

Abstract: Described is the running of search-related experiments on a full (or partial) offline snapshot copy of the search engine documents of an actual production system. A snapshot experimentation subsystem runs experimental code related to web searches on the offline data, including to run experimental index building code to build an experimental index (e.g., to test a new document feature), and/or to run experimental search-related code, such as to rank search results according to experimental ranking code, to implement an experimental search strategy, and/or to generate experimental captions.

Abstract: Described is a flexible framework for index building and document retrieval in a search environment that allows different search scenario applications to reuse index building and document retrieval code for non-scenario-specific functionality. Interfaces to various functionality of an index builder and retrieval engine are defined. An application calls the interfaces to specify custom code to perform a search scenario when needed, or use default code when non-scenario-specific functionality may be used.

Abstract: A method and system for determining relevance of a document to a query based on identifier match distance is provided. The relevance system analyzes a training set of queries and documents to determine the relationship between identifier match distance and relevance of a document to a query. The identifier match distance indicates the distance from the end of an identifier of a document to an identifier term that matches a query term. The relevance system generates a prior relevance probability that a document with a certain identifier match distance is relevant to a query. The relevance system uses the prior relevance probabilities to determine relevance of documents to queries based on identifier match distance.

Abstract: Categorizing page block functionality to improve document layout for browsing is described. In one aspect, document content is analyzed with respect to multiple block function criteria. Results of this analysis are used to assign a respective block function to blocks of the document content. These block function assignments are used to generate one or more customized document layouts for browsing by a user.

Abstract: A method and system for determining relevance of a document to a query based on identifier match distance is provided. The relevance system analyzes a training set of queries and documents to determine the relationship between identifier match distance and relevance of a document to a query. The identifier match distance indicates the distance from the end of an identifier of a document to an identifier term that matches a query term. The relevance system generates a prior relevance probability that a document with a certain identifier match distance is relevant to a query. The relevance system uses the prior relevance probabilities to determine relevance of documents to queries based on identifier match distance.

Abstract: Categorizing page block functionality to improve document layout for browsing is described. In one aspect, document content is analyzed with respect to multiple block function criteria. Results of this analysis are used to assign a respective block function to blocks of the document content. These block function assignments are used to generate one or more customized document layouts for browsing by a user.