Abstract: A computer implemented method for clustering documents that includes receiving at least a first first-level cluster of documents and a second first-level cluster of documents, wherein the first first-level cluster includes one or more first documents associated with a first domain, the first documents satisfying a first first-level classification criterion, and the second first-level cluster including one or more second documents associated with a second domain, the second documents satisfying a second first-level classification criterion. The method also includes creating, by a processor of the computer system, a second-level cluster by combining the first first-level cluster with the second first-level cluster when a) the first first-level cluster and second first-level cluster satisfy a second-level classification criterion; and b) the first domain does not match the second domain.

Abstract: A system and method is disclosed for detecting invalid webpages by analyzing server redirects. A storage comprising a set of previously stored target addresses is queried to determine whether one or more of the set of previously stored target addresses result from a redirect initiated from more than a predetermined number of originating addresses. On determining that a target address resulted from a redirect initiated from more than the predetermined number of originating addresses, the originating addresses are analyzed to determine, for each address, a difference between information previously stored for the originating address and information associated with the respective target address. If the difference satisfies a predetermined threshold, the originating address is marked as not valid or is removed.