Abstract: A system may identify a set of first documents associated with an organization, and identify clusters to which the first documents belong. Each of a number of the identified clusters may include a group of documents that includes one of the first documents and one or more second documents associated with one or more different organizations. The system may determine a quality score for each of the documents in each of the identified clusters, and determine, for each of the number of the identified clusters, whether the quality score of the one of the first documents in the identified cluster is higher than the quality score of the one or more second documents in the identified cluster. The system may generate a proxy pad score based on the determinations, and store the proxy pad score.

Abstract: A system may identify a set of first documents associated with an organization, and identify clusters to which the first documents belong. Each of a number of the identified clusters may include a group of documents that includes one of the first documents and one or more second documents associated with one or more different organizations. The system may determine a quality score for each of the documents in each of the identified clusters, and determine, for each of the number of the identified clusters, whether the quality score of the one of the first documents in the identified cluster is higher than the quality score of the one or more second documents in the identified cluster. The system may generate a proxy pad score based on the determinations, and store the proxy pad score.

Abstract: A system may identify a set of related documents, identify one or more documents in the set of related documents that are sources of redirects, and identify organizations that are targets of the redirects. The system may also determine a redirect score based on the number of the identified documents that are sources of the redirects, determine a spam score based on a number of the organizations that are targets of the redirects, determine whether to classify the set of related documents as a bounce pad based on the redirect score and the spam score, and storing information associated with the result of the determination of whether to classify the set of related documents as a bounce pad.

Abstract: A system receives a document that includes new content and aged content, and compares the document with a prior version of the document that includes the aged content but not the new content. The system also separates the new content and the aged content based on the comparison, determines ages associated with the new content and the aged content, and determines whether the ages of the new content and the aged content are greater than or equal to an age threshold. The system further calculates a checksum of the document based on the aged content when the age of the aged content is greater than or equal to the age threshold, and the age of the new content is less than the age threshold, and stores the calculated checksum.

Abstract: A system may identify a set of related documents, identify one or more documents in the set of related documents that are sources of redirects, and identify organizations that are targets of the redirects. The system may also determine a redirect score based on the number of the identified documents that are sources of the redirects, determine a spam score based on a number of the organizations that are targets of the redirects, determine whether to classify the set of related documents as a bounce pad based on the redirect score and the spam score, and storing information associated with the result of the determination of whether to classify the set of related documents as a bounce pad.

Abstract: A system receives a document that includes new content and aged content, and compares the document with a prior version of the document that includes the aged content but not the new content. The system also separates the new content and the aged content based on the comparison, determines ages associated with the new content and the aged content, and determines whether the ages of the new content and the aged content are greater than or equal to an age threshold. The system further calculates a checksum of the document based on the aged content when the age of the aged content is greater than or equal to the age threshold, and the age of the new content is less than the age threshold, and stores the calculated checksum.

Abstract: A method may include identifying documents in a current clustering operation, assigning the identified documents to one or more clusters, selecting a current representative document for each of the one or more clusters, determining whether the current representative document has been re-crawled, determining a previous representative document with which the current representative document was previously associated in a prior clustering operation, if it is determined that the current representative document has not been re-crawled, determining one of the one or more clusters to which the previous representative document has been assigned in the current clustering operation, combining one of the one or more clusters associated with the current representative document that has not been re-crawled with the one of the one or more clusters associated with the previous representative document into a combined cluster, and storing information regarding the combined cluster.