Abstract: The technology described herein provides for identifying matching documents for a search query using a bit vector search index. When a search query is received, a term is identified from the search index, and a number of bit vectors corresponding to the term are identified. Each bit vector comprises an array of bits in which at least one bit in each bit vector indicates that a corresponding document includes the term. Each bit vector also includes other bits indicating other documents include other terms. A determination is made that an unacceptable number of possible matching documents is likely to be returned. In response to the determination, a strengthening row bit vector is selected to reduce the number of possible matching documents. The identified bit vectors and the selected strengthening row are intersected to identify matching documents that contain the term.

Abstract: The technology described herein provides for identifying matching documents for a search query using a bit vector search index. When a search query is received, a term is identified from the search index, and a number of bit vectors corresponding to the term are identified. Each bit vector comprises an array of bits in which at least one bit in each bit vector indicates that a corresponding document includes the term. Each bit vector also includes other bits indicating other documents include other terms. A determination is made that an unacceptable number of possible matching documents is likely to be returned. In response to the determination, a strengthening row bit vector is selected to reduce the number of possible matching documents. The identified bit vectors and the selected strengthening row are intersected to identify matching documents that contain the term.

Abstract: The technology described herein provides for identifying matching documents for a search query using a bit vector search index. When a search query is received, a term is identified from the search index, and a number of bit vectors corresponding to the term are identified. Each bit vector comprises an array of bits in which at least one bit in each bit vector indicates that a corresponding document includes the term. Each bit vector also includes other bits indicating other documents include other terms. A determination is made that an unacceptable number of possible matching documents is likely to be returned. In response to the determination, a strengthening row bit vector is selected to reduce the number of possible matching documents. The identified bit vectors and the selected strengthening row are intersected to identify matching documents that contain the term.

Abstract: The technology described herein provides for a match fix-up stage that removes matching documents identified for a search query that don't actually contain terms from the search query. A representation of each document (e.g., a forward index storing a list of terms for each document) is used to identify valid matching documents (i.e., documents containing terms from the search query) and invalid matching documents (i.e., documents that don't contain terms from the search query). Any invalid matching documents are removed from further processing and ranking for the search query.

Abstract: The technology described herein provides for preliminary ranking of matching documents for a search query. A preliminary ranker uses score tables for scoring each matching document based on its relevant to a search query. The score table for a document stores pre-computed data used to derive a frequency of terms and other information in the document. The preliminary ranker uses the score table for each matching document and the terms from the search query to determine a score for each matching document. The lowest scoring documents are removed from further consideration by a final ranker.

Abstract: The technology described herein provides for indexing information in a bit vector search index. The bit vector search index comprises a data structure for indexing data about terms from a corpus of documents. The data structure includes a number of bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Bits in the bit vector are used to represent whether at least one document corresponding to the bit includes at least one term from the set of terms corresponding to the bit vector. The bit vector search index is stored by first indexing information about documents using bit vectors on a first accumulation buffer storage device. When a threshold is satisfied, the information is transferred to bit vectors on a subsequent storage device.

Abstract: The technology described herein provides a bit vector search index for a search system that uses shards. The bit vector search index comprises a data structure for indexing data about terms from a corpus of documents. The data structure includes a number of bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Bits in the bit vector are used to represent whether at least one document corresponding to the bit includes at least one term from the set of terms corresponding to the bit vector. The search index is provided in a number of shards. Each shard corresponds to a subset of documents having documents lengths within particular a range of document lengths.

Abstract: The technology described herein provides for identifying matching documents for a search query using a bit vector search index. When a search query is received, a term is identified from the search index, and a number of bit vectors corresponding to the term are identified. Each bit vector comprises an array of bits in which at least one bit in each bit vector indicates that a corresponding document includes the term. Each bit vector also includes other bits indicating other documents include other terms. The identified bit vectors are intersected to identify matching documents that contain the term.

Abstract: The technology described herein provides for preliminary ranking of matching documents for a search query. A preliminary ranker uses score tables for scoring each matching document based on its relevant to a search query. The score table for a document stores pre-computed data used to derive a frequency of terms and other information in the document. The preliminary ranker uses the score table for each matching document and the terms from the search query to determine a score for each matching document. The lowest scoring documents are removed from further consideration by a final ranker.

Abstract: The technology described herein provides for preliminary ranking of matching documents for a search query. A preliminary ranker uses score tables for scoring each matching document based on its relevant to a search query. The score table for a document stores pre-computed data used to derive a frequency of terms and other information in the document. The preliminary ranker uses the score table for each matching document and the terms form the search query to determine a score for each matching document. The lowest scoring documents are removed from further consideration by a final ranker.

Abstract: The technology described herein provides for storing and retrieving data in a bit vector search index. The bit vector search index stores data about terms from documents using bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Each bit in the bit vector is used to represent whether a document includes at least one term from the set of terms. A band table is used to store bit vector configurations for bands of terms having similar term characteristics. Each term is indexed in the bit vector search index according to a bit vector configuration for a band to which it belongs. When identifying bit vector storage locations for terms, explicit mappings are used for some terms and ad hoc approaches used for other terms. Explicit mappings provide specific locations for terms, while ad hoc approaches use mapping algorithms assigned to bands.

Abstract: The technology described herein provides for a match fix-up stage that removes matching documents identified for a search query that don't actually contain terms from the search query. A representation of each document (e.g., a forward index storing a list of terms for each document) is used to identify valid matching documents (i.e., documents containing terms from the search query) and invalid matching documents (i.e., documents that don't contain terms from the search query). Any invalid matching documents are removed from further processing and ranking for the search query.

Abstract: The technology described herein provides a bit vector search index for a search system. The bit vector search index comprises a data structure for indexing data about terms from a corpus of documents. The data structure includes a number of bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Bits in the bit vector are used to represent whether at least one document corresponding to the bit includes at least one term from the set of terms corresponding to the bit vector.

Abstract: The technology described herein provides for identifying matching documents for a search query using a bit vector search index. When a search query is received, a term is identified from the search index, and a number of bit vectors corresponding to the term are identified. Each bit vector comprises an array of bits in which at least one bit in each bit vector indicates that a corresponding document includes the term. Each bit vector also includes other bits indicating other documents include other terms. The identified bit vectors are intersected to identify matching documents that contain the term.

Abstract: The technology described herein provides for storing and retrieving data in a bit vector search index. The bit vector search index stores data about terms from documents using bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Each bit in the bit vector is used to represent whether a document includes at least one term from the set of terms. A band table is used to store bit vector configurations for bands of terms having similar term characteristics. Each term is indexed in the bit vector search index according to a bit vector configuration for a band to which it belongs. When identifying bit vector storage locations for terms, explicit mappings are used for some terms and ad hoc approaches used for other terms. Explicit mappings provide specific locations for terms, while ad hoc approaches use mapping algorithms assigned to bands.

Abstract: The technology described herein provides for identifying matching documents for a search query using a bit vector search index. When a search query is received, a term is identified from the search index, and a number of bit vectors corresponding to the term are identified. Each bit vector comprises an array of bits in which at least one bit in each bit vector indicates that a corresponding document includes the term. Each bit vector also includes other bits indicating other documents include other terms. A determination is made regarding which bit vectors for the term to use for the matching process. The selected bit vectors are intersected to identify matching documents that contain the term.

Abstract: The technology described herein provides for identifying matching documents for a search query using a bit vector search index. When a search query is received, a term is identified from the search index, and a number of bit vectors corresponding to the term are identified. Each bit vector comprises an array of bits in which at least one bit in each bit vector indicates that a corresponding document includes the term. Each bit vector also includes other bits indicating other documents include other terms. A determination is made that an unacceptable number of possible matching documents is likely to be returned. In response to the determination, a strengthening row bit vector is selected to reduce the number of possible matching documents. The identified bit vectors and the selected strengthening row are intersected to identify matching documents that contain the term.

Abstract: The technology described herein provides for indexing information in a bit vector search index. The bit vector search index comprises a data structure for indexing data about terms from a corpus of documents. The data structure includes a number of bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Bits in the bit vector are used to represent whether at least one document corresponding to the bit includes at least one term from the set of terms corresponding to the bit vector. The bit vector search index is stored by first indexing information about documents using bit vectors on a first accumulation buffer storage device. When a threshold is satisfied, the information is transferred to bit vectors on a subsequent storage device.

Abstract: The technology described herein provides a bit vector search index for a search system that uses shards. The bit vector search index comprises a data structure for indexing data about terms from a corpus of documents. The data structure includes a number of bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Bits in the bit vector are used to represent whether at least one document corresponding to the bit includes at least one term from the set of terms corresponding to the bit vector. The search index is provided in a number of shards. Each shard corresponds to a subset of documents having documents lengths within particular a range of document lengths.

Abstract: The technology described herein provides for preliminary ranking of matching documents for a search query. A preliminary ranker uses score tables for scoring each matching document based on its relevant to a search query. The score table for a document stores pre-computed data used to derive a frequency of terms and other information in the document. The preliminary ranker uses the score table for each matching document and the terms form the search query to determine a score for each matching document. The lowest scoring documents are removed from further consideration by a final ranker.