Abstract: A method and system for crawling the World Wide Web is described. One embodiment avoids becoming bogged down by dynamically generated Uniform Resource Locators (URLs) pointing to Web pages having the same or substantially similar content (e.g., URLs generated by a “spam poison” Web site) by browsing automatically and systematically Web pages within a first domain of the World Wide Web, each Web page having its own content; determining that the content of a currently visited Web page is the same as that of a predetermined number of other Web pages that have already been visited; and ceasing to browse the first domain and instead browsing a second domain of the World Wide Web different from the first domain in response to determining that the content of the currently visited Web page is the same as that of a predetermined number of other Web pages that have already been visited.