Abstract: A system and method for searching networked based content limits searching unnecessary content. The first time a web site is encountered, an exhaustive search is conducted (522), and a site map (300) is generated (524) and the URL of the web site is added to a directory of encountered web sites (526). The next time the web site is encountered, the system utilizes the site map and directory to search only for relevant content (614). Web sites are revisited, in accordance with information derived from previous visits, to conduct subsequent exhaustive searches in order to update the site map and directory. A site map includes a structured data storage format, wherein content of the web site is organized in levels.