Abstract: A method and apparatus for identifying important terms in a sample text. A frequency of occurrence of terms in (sample frequency) is compared to a frequency of occurrence of those terms in a reference text (reference frequency). Terms occurring with higher frequency in the sample text than in the reference text are considered important to the sample text. A difference between the respective sample and reference frequencies of a term may be used to determine an importance score. Terms can be ranked and/or added to an affinity set as a function of importance score or rank. When there are insufficient terms for determining a sample frequency, those terms may be used in a search query to identify documents for use as sample text to determine sample frequencies. The important terms may be used for document summarization, query refinement, cross-language translation, and cross-language query expansion.

Abstract: A web crawler identifies and characterizes an expression of a topic of general interest (such as cryptography) entered and generates an affinity set which comprises a set of related words. This affinity set is related to the expression of a topic of general interest. Using a common search engine, seed documents are found. The seed documents along with the affinity set and other search data will provide training to a classifier to create classifier output for the web crawler to search the web based on multiple criteria, including a content-based rating provided by the trained classifier. The web crawler can perform it's search topic focused, rather than “link” focused. The found relevant content will be ranked and results displayed or saved for a specialty search.