Abstract: Disclosed is a technique for building an index in which global analysis computations and index creation are pipelined, wherein the global analysis computations share intermediate results.

Abstract: Disclosed is a method, system, and program for handling redirects in documents. At least one equivalence class that includes documents that are connected through a redirect. Cycles for each equivalence class are detected, wherein documents in a cycle are marked so that they are not indexed. Incomplete chains for each equivalence class are detected, wherein documents in an incomplete chain are marked so that they are not indexed. A representative for each equivalence class is selected.

Abstract: A collaborative focused crawler crawls documents on a network locating documents that match multiple focus topics. The collaborative crawler comprises a fetcher and a focus engine. The fetcher prioritizes which documents to crawl based on a set of rules, obtains documents from the network, and outputs crawled documents to the focus engine. The focus engine determines whether a fetched document is relevant to any of the multiple focus topics. The focus engine determines whether fetched documents are disallowed. If a fetched document is disallowed, the present system may place the URL for that web document in a blacklist, a list of URLs that may not be crawled. URLs may be disallowed if they match a disallowed topic or if they fail a set of rules designed for a web space focus, for example, domain rules, IP address rules, and prefix rules.

Abstract: A search-routing database is used to direct database search requests to only those databases that may contain results to the search request. The search-routing database contains a subset of the data fields from the other databases in the network, along with a field indicating the database to which the search request should be routed. When a database search request is received, a modified search request is generated based on the overlap between the type of information contained in the search request and the type of information contained in the data-fields of the search routing database. This modified search request is submitted to the search-routing database, which returns a list of database candidates which could produce results to the original search request. The original search request is then submitted to those database candidates, and the results are returned to the user.

Abstract: A search-routing database is used to direct database search requests to only those databases that may contain results to the search request. The search-routing database contains a subset of the data fields from the other databases in the network, along with a field indicating the database to which the search request should be routed. When a database search request is received, a modified search request is generated based on the overlap between the type of information contained in the search request and the type of information contained in the data-fields of the search routing database. This modified search request is submitted to the search-routing database, which returns a list of database candidates which could produce results to the original search request. The original search request is then submitted to those database candidates, and the results are returned to the user.