Abstract: A query difficulty prediction unit includes a query difficulty predictor to determine the extent of overlap between query documents received from a search engine operating on an input query and sub-query documents received from the search engine operating on sub-queries of the input query. The unit generates a query difficulty prediction from the extent of overlap.

Abstract: A method for Automatic Query Refinement, the method including extracting at least one lexical affinity from at least one document in a search result set of documents, the set resulting from the application of a first query to a document search engine, adding the lexical affinity to the first query, thereby forming a new query, and applying the new query to the document search engine, resulting in a new search result set.

Abstract: Disclosed is a system architecture, components and a searching technique for an Unstructured Information Management System (UIMS). The UIMS may be provided as middleware for the effective management and interchange of unstructured information over a wide array of information sources. The architecture generally includes a search engine, data storage, analysis engines containing pipelined document annotators and various adapters. The searching technique makes use of a two-level searching technique. Also disclosed is system, method and computer program product to process document data. The method includes inputting a document and operating at least one text analysis engine that comprises a plurality of coupled annotators for tokenizing document data for identifying and annotating a particular type of semantic content. Operating the at least one text analysis engine generates a plurality of views of a document, where each of the plurality of views are derived from a different tokenization of the document.

Abstract: Disclosed is a system architecture, components and a searching technique for an Unstructured Information Management System (UIMS). The UIMS may be provided as middleware for the effective management and interchange of unstructured information over a wide array of information sources. The architecture generally includes a search engine, data storage, analysis engines containing pipelined document annotators and various adapters. The searching technique makes use of a two-level searching technique. A search query includes a search operator containing of a plurality of search sub-expressions each having an associated weight value. The search engine returns a document or documents having a weight value sum that exceeds a threshold weight value sum. The search operator is implemented as a Boolean predicate that functions as a Weighted AND (WAND).

Abstract: An apparatus is provided for performing a method (FIG. 2) for pruning an index of a corpus of text documents, wherein the method includes steps for ranking (50) the postings in the index and pruning (48) from the index the postings below a given level in the ranking. The pruning methods of the invention are lossy, since some document postings are removed from the full index; however, the user cannot differentiate the lossy index from the full index.

Abstract: A method for indexing a semi-structured document, the method including arranging at least one structure entity of a semi-structured document into at least one node of a context structure tree, associating a unique context identifier with any of the structure entities, creating, for any value of any of the structure entities, a context-modified value by appending a context delimiter and the context identifier to the value, and inserting the context-modified value into a free-text tree.

Abstract: A method for gathering and recording temporal information for a linked entity, the method including identifying a link related activity within a linked source entity, and recording a time stamp in association with the link related activity.

Abstract: A method for Automatic Query Refinement, the method including extracting at least one lexical affinity from at least one document in a search result set of documents, the set resulting from the application of a first query to a document search engine, adding the lexical affinity to the first query, thereby forming a new query, and applying the new query to the document search engine, resulting in a new search result set.

Abstract: A method for searching a corpus of documents, such as the World Wide Web, includes defining a knowledge domain and identifying a set of reference documents in the corpus pertinent to the domain. Upon inputting a query, the corpus is searched using the set of reference documents to find one or more of the documents in the corpus that contain information in the domain relevant to the query. The set of reference documents is updated with the found documents that are most relevant to the domain. The updated set is used in searching the corpus for information in the domain relevant to subsequent queries.

Abstract: A method for searching a corpus of documents, such as the World Wide Web, includes defining a knowledge domain and identifying a set of reference documents in the corpus pertinent to the domain. Upon inputting a query, the corpus is searched using the set of reference documents to find one or more of the documents in the corpus that contain information in the domain relevant to the query. The set of reference documents is updated with the found documents that are most relevant to the domain. The updated set is used in searching the corpus for information in the domain relevant to subsequent queries.