Abstract: A system and method for generating a search ranking score for a web page. The system comprises a training data processor effective to receive training data including at least a first page, a first label, a second page and a second label. The system further comprises a feature extraction processor connected to the training data processor, the feature extraction processor is effective to receive the first page, identify first features in the first page and calculate first values relating to the first features; the feature extraction processor is further effective to receive the second page and identify second features and calculate second values relating to the second features. A machine learning processor is connected to the feature extraction processor.

Abstract: Automated crawling of page links associated with a site domain that was previously crawled involves computing the dynamicity of a site based on totals of continuous dead links, live links and/or prerequisite pages encountered while crawling page links corresponding to the site. The degree to which links are crawled is optimized based on the dynamicity of the site. Some pages require that another particular page (i.e., a prerequisite page) is retrieved from the host prior to retrieving a given page, e.g., so that the prerequisite page can set a cookie. Prerequisite pages are determined based on stored information about pages that were retrieved, during a previous crawl, prior to retrieving a page. Prerequisite pages are identified to a search system so that when a user clicks on the URL for the page, the request is redirected to the prerequisite page to set the cookie appropriately.