Abstract: A method for estimating recall in database workflows involves assigning unique keys to documents in a database, generating a random permutation of the database, accessing coding decisions for documents, determining the longest prefix of the permutation with corresponding coding decisions, calculating the number of relevant documents, applying a confidence sequence generating algorithm to the prefix to establish a confidence interval on the relevant documents, and computing a confidence interval estimate on recall for the database. This method provides a systematic approach to assess recall in database workflows, enabling accurate evaluation of the retrieval performance of relevant documents within the database.

Abstract: A query phrase may be automatically classified to one or more topics of interest (e.g., categories) to assist in routing the query phrase to one or more appropriate backend databases. A selectional preference query classification technique may be used to classify the query phrase based on a comparison between the query phrase and patterns of query phrases. Additionally, or alternatively, a combination of query classification techniques may be used to classify the query phrase. Topical classification of a query phrase also may be used to assist a search system in delivering auxiliary information to a user who entered the query phrase. Advertisements, for instance, may be tailored based on classification rather than query keywords.

Abstract: An example method for reviewing documents includes scoring documents using an artificial intelligence model, and selecting a subset of highest scoring documents. The method further includes inserting a number of randomly-selected documents into the subset of highest scoring documents to form a set of documents for review, wherein a reviewer cannot differentiate between the randomly-selected documents and the subset of highest scoring documents included in the set of documents for review, and presenting the set of documents for review by the reviewer.

Abstract: An example method for reviewing documents includes scoring documents using an artificial intelligence model, and selecting a subset of highest scoring documents. The method further includes inserting a number of randomly-selected documents into the subset of highest scoring documents to form a set of documents for review, wherein a reviewer cannot differentiate between the randomly-selected documents and the subset of highest scoring documents included in the set of documents for review, and presenting the set of documents for review by the reviewer.

Abstract: A query phrase may be automatically classified to one or more topics of interest (e.g., categories) to assist in routing the query phrase to one or more appropriate backend databases. A selectional preference query classification technique may be used to classify the query phrase based on a comparison between the query phrase and patterns of query phrases. Additionally, or alternatively, a combination of query classification techniques may be used to classify the query phrase. Topical classification of a query phrase also may be used to assist a search system in delivering auxiliary information to a user who entered the query phrase. Advertisements, for instance, may be tailored based on classification rather than query keywords.

Abstract: A query phrase may be automatically classified to one or more topics of interest (e.g., categories) to assist in routing the query phrase to one or more appropriate backend databases. A selectional preference query classification technique may be used to classify the query phrase based on a comparison between the query phrase and patterns of query phrases. Additionally, or alternatively, a combination of query classification techniques may be used to classify the query phrase. Topical classification of a query phrase also may be used to assist a search system in delivering auxiliary information to a user who entered the query phrase. Advertisements, for instance, may be tailored based on classification rather than query keywords.

Abstract: A query phrase may be automatically classified to one or more topics of interest (e.g., categories) to assist in routing the query phrase to one or more appropriate backend databases. A selectional preference query classification technique may be used to classify the query phrase based on a comparison between the query phrase and patterns of query phrases. Additionally, or alternatively, a combination of query classification techniques may be used to classify the query phrase. Topical classification of a query phrase also may be used to assist a search system in delivering auxiliary information to a user who entered the query phrase. Advertisements, for instance, may be tailored based on classification rather than query keywords.

Abstract: A query phrase may be automatically classified to one or more topics of interest (e.g., categories) to assist in routing the query phrase to one or more appropriate backend databases. A selectional preference query classification technique may be used to classify the query phrase based on a comparison between the query phrase and patterns of query phrases. Additionally, or alternatively, a combination of query classification techniques may be used to classify the query phrase. Topical classification of a query phrase also may be used to assist a search system in delivering auxiliary information to a user who entered the query phrase. Advertisements, for instance, may be tailored based on classification rather than query keywords.

Abstract: An information retrieval method wherein users may submit a query via a graphical bitmapping technique. The user provides an information retrieval system with a bitmap of a printed, written, or graphical query by either scanning the query with a graphical scanner, or employing a standard facsimile transmission machine. The information retrieval system then performs an optical image/character recognition process upon the received bitmap to determine the content of the query, information is then retried based upon the recognized characters and images. In a particular method of the invention, the user is provided with a bitmap of the retrieved information.

Abstract: The invention is an improved retrieval system and method. Many pattern recognition tasks, including estimation, classification, and the finding of similar objects, make use of linear models. For example, many text retrieval systems represent queries as linear functions, and retrieve documents whose vector representation has a high dot product with the query. The fundamental operation in such tasks is the computation of the dot product between a query vector and a large database of instance vectors. Often instance vectors which have high dot products with the query are of interest. The invention relates to a random sampling based retrieval system that can identify, for any given query vector, those instance vectors which have large dot products, while avoiding explicit computation of all dot products.

Abstract: Current tools for processing e-mail and other messages do not adequately recognize and manipulate threads, i.e., conversations among two or more people carried out by exchange of messages. The present invention utilizes the textual context and characteristics of messages in order to provide a more reliable and effective way to construct message threads. In accordance with the present invention, statistical information retrieval techniques are used in conjunction with textual material obtained by "filtering" of messages to achieve a significant level of accuracy at identifying when one message is a reply to another.

Abstract: A method and apparatus for training a text classifier is disclosed. A supervised learning system and an annotation system are operated cooperatively to produce a classification vector which can be used to classify documents with respect to a defined class. The annotation system automatically annotates documents with a degree of relevance annotation to produce machine annotated data. The degree of relevance annotation represents the degree to which the document belongs to the defined class. This machine annotated data is used as input to the supervised learning system. In addition to the machine annotated data, the supervised learning system can also receive manually annotated data and/or a user request. The machine annotated data, along with the manually annotated data and/or the user request, are used by the supervised learning system to produce a classification vector. In one embodiment, the supervised learning system comprises a relevance feedback mechanism.

Abstract: Apparatus and methods for training classifiers. The apparatus includes a degree of certainty classifier which classifies examples in categories and indicates a degree of certainty regarding the classification and an annotating classifier which receives classified examples with a low degree of certainty from the degree of certainty classifier and annotates them to indicate whether their classifications are correct. The annotated examples are then used to train another classifier. In one version of the invention, the other classifier is a new version of the degree of certainty classifier, and training continues until the degree of certainty classifier has satisfactory performance. The degree of certainty classifier of the embodiment is a probabilistic binary classifier which is trained using relevance feedback. The annotating classifier may include an interactive interface which permits a human user of the system to examine an example and indicate whether it was properly classified.