Abstract: A method, system, and computer program product for discriminating boilerplate text in documents, such as web pages. An example method includes: receiving documents structured as labelled text elements; generating a local language model for each labelled text element of the received documents; comparing local language models for different labelled text elements that have the same label; for each comparison of local language models, deriving a similarity indicator, and using the similarity indicators of all the comparisons to derive a similarity score for that label; using the similarity scores to determine labels associated with text elements comprising boilerplate text; and providing the textual content of the labelled text elements to a receiving computer system; and identifying the textual content of labelled text elements that include boilerplate text.

Abstract: A method, system, and computer program product for discriminating boilerplate text in documents, such as web pages. An example method includes: receiving documents structured as labelled text elements; generating a local language model for each labelled text element of the received documents; comparing local language models for different labelled text elements that have the same label; for each comparison of local language models, deriving a similarity indicator, and using the similarity indicators of all the comparisons to derive a similarity score for that label; using the similarity scores to determine labels associated with text elements comprising boilerplate text; and providing the textual content of the labelled text elements to a receiving computer system; and identifying the textual content of labelled text elements that include boilerplate text.

Abstract: Large websites employ internal search engines to assist visitors of the site to access pages relevant to the visitor's needs. Such internal search engines generally use a specialist database containing information relevant to the website. Internet search engines generally do not have access to the specialise databases and so the results they produce are frequently less useful than the results produced from the same query addressed to an internal search engine. The effectiveness of the internet search engine is improved by the use of a store (11) of information for each page (1) of the website, the store (11) comprising a record of each query (Q1, Q2 . . . Qn) directed to the page, the frequency (f1, f2 . . . fn) of the query and the relevance (r1, r2 . . . rn) of the query as calculated by a relevance calculator (5). Also included is a mechanism to generate, for each relevant query, an intent page (16) which contains the relevant query and a link to the webpage that it accessed.

Abstract: Conventional electronic information retrieval systems using specialist databases assume that the database being searched is appropriate to the search being conducted. The invention solves the problem that the above assumption may be incorrect. It does this by providing mechanisms (4, 4A and 4B) for identifying concepts respectively in: the search input question, a specialist database and a non-specialist database. A comparator 8 and associated mechanisms are able to analyze how the specialist database differs from the non-specialist database and to use this information to derive an indication of the ability of the specialist database to answer the input question. This indication can be fed to the user via interface 1 or in an alternative arrangement could be used to block inappropriate information.