Abstract: A rewrite component automatically generates rewrite rules that describe how uniform resource locators (URLs) can be rewritten to reduce or eliminate different URLs that redundantly refer to the same or substantially the same content. The rewrite rules can be applied to URLs received when crawling a network to increase the efficiency of the crawl and the corresponding document index generated from the crawl.

Abstract: A rewrite component automatically generates rewrite rules that describe how uniform resource locators (URLs) can be rewritten to reduce or eliminate different URLs that redundantly refer to the same or substantially the same content. The rewrite rules can be applied to URLs received when crawling a network to increase the efficiency of the crawl and the corresponding document index generated from the crawl.

Abstract: A wrapper builder application provides a variety of features for facilitating the creation of wrappers that are used to extract information from Web sites. In one aspect, the wrapper builder application provides a tool with which the process of creating a wrapper, which typically resembles coding, can be accomplished by a graphical design process involving drag and drop operations, clicking on objects, and filling in forms. A web viewer component provides a web browser frame, a source code frame, and a tree view frame, enabling the user to identify semistructured information of interest on Web sites. A wrapper editor component provides a graphical design environment in which a wrapper can be graphically constructed from operations and links. A wrapper model component provides a functioning internal representation of the graphically designed wrapper using Java objects and methods.

Abstract: According to the invention, a system and method for extracting information from a semistructured information source. The system includes a listing stack for holding extracted information. A means for matching at least one extractor to the semistructured information to return a list of potential matches is also included. The system can also include a means for iterating through the list of potential matches and a means for retrieving information from a particular match in the list of potential matches. A means for adding a particular match into the listing stack can also be part of the system.

Abstract: According to the invention, a method is provided for forming an information closure of a plurality of rows in a listing stack built by a wrapper program for accessing semistructured information. This method includes removing a first row from the listing stack and computing a cross product of the fields in the first row. A step of adding this cross product to a list of accepted rows can also be part of the method. For each remaining row in the listing stack, the method includes a step of computing a selective cross product according to a plurality of steps. In one step, a result is initialized to empty. Then, for each row in the list of accepted rows, a step of determining for a first new row from the accepted row, extended with the non-empty fields of the remaining row is performed. The method can also include a step of determining a second new row from the remaining row, extended with the non-empty fields in the accepted row. Thereupon, a step of adding the two new rows to the result can be performed.

Abstract: According to the invention, a method is provided for forming an information closure of a plurality of rows in a linkage stack built by a wrapper program for accessing semistructured information. This method includes removing a first row from the linkage stack and computing a cross product of the fields in the first row. A step of adding this cross product to a list of accepted rows can also be part of the method. For each remaining row in the linkage stack, the method includes a step of computing a selective cross product according to a plurality of steps. In one step, a result is initialized to empty. Then, for each row in the list of accepted rows, a step of determining for a first new row from the accepted row, extended with the non-empty fields of the remaining row is performed. The method can also include a step of determining a second new row from the remaining row, extended with the non-empty fields in the accepted row. Thereupon, a step of adding the two new rows to the result can be performed.

Abstract: According to the invention, a system and method for extracting information from a semistructured information source. The system includes a listing stack for holding extracted information. A means for matching at least one extractor to the semistructured information to return a list of potential matches is also included. The system can also include a means for iterating through the list of potential matches and a means for retrieving information from a particular match in the list of potential matches. A means for adding a particular match into the listing stack can also be part of the system.

Abstract: According to the invention, methods for gathering data around forms having one or more fields, enabling a wrapper program to extract semistructured information by determining combinations of values for fields associated with particular forms; submitting the particular forms repeatedly for all combinations of interest and providing the results returned for further processing. In select embodiments, the combinations of values for fields is a Cartesian product of the possible values for the fields. Values to be submitted in the form fields may be specified by using a programming language such as Site Description Language (SDL) or Java.