Abstract: A computer executed method is provided for crawling documents within an Internet domain, the method comprising: (a) having computer executable logic retrieve a document identified by a document address and a crawl depth; (b) having computer executable logic identify any links in the document; (c) having computer system identify which of the identified links in the document are (i) out-of domain links because the identified links do not specify the same Internet domain as the document address, (ii) lateral links to continuation documents of the document by identifying that there are continuation document terms associated with the links, and (iii) standard links to documents lower in the Internet domain's hierarchy by identifying that there are no continuation document terms associated with the links; (d) performing steps (a)-(c) for documents that are identified as being laterally linked to the document of step (a), where the same crawl depth is employed for the laterally linked documents as the crawl depth fo