Abstract: A web crawler system has an automatic website crawler and a virtual browser that provides script related information to the website crawler. The virtual browser transforms an HTML document included in a web page of the website into an XML document, and builds a document object model containing document objects in a tree structure based on the XML document. The virtual browser extracts from the DOM scripts that are potentially executable, and executes the extracted scripts using a browser object model provided for the virtual browser containing objects and methods and properties that are used for script execution so as to capture script related information generated by execution of the scripts.

Abstract: The invention relates generally to computers and communications, and more specifically, to a method and system for analyzing Web sites and similar data structures. Tools exist for analyzing Web sites and locating problems, or simply collecting data, but the existing tools are very limited in what they can do, generally restricting the User to a selection of predetermined “tick-box” options. The invention provides an environment in which flexible and sophisticated search parameters may be defined by the User via regular expressions and a standard language specification. This allows the User to tailor his searches to match his specific Web site policy. The “extensible scan rules” of the invention also allow logic tests and analysis to be implemented, so that the search results are far more useful and relevant to the User.