Abstract: Implementations include a batch-optimized render and fetch architecture. An example method performed by the architecture includes receiving a request from a batch process to render a web page and initializing a virtual clock and a task list for rendering the web page. The virtual clock stands still when a request for an embedded item is outstanding and when a task is ready to run. The method may also include generating a rendering result for the web page when the virtual clock matches a run time for a stop task in the task list, and providing the rendering result to the batch process. Another example method includes receiving a request from a batch process to render a web page, identifying an embedded item in the web page, and determining, based on a rewrite rule, that the embedded item has content that is duplicative of content for a previously fetched embedded item.

Abstract: Implementations include a batch-optimized render and fetch architecture. An example method performed by the architecture includes receiving a request from a batch process to render a web page and initializing a virtual clock and a task list for rendering the web page. The virtual clock stands still when a request for an embedded item is outstanding and when a task is ready to run. The method may also include generating a rendering result for the web page when the virtual clock matches a run time for a stop task in the task list, and providing the rendering result to the batch process. Another example method includes receiving a request from a batch process to render a web page, identifying an embedded item in the web page, and determining, based on a rewrite rule, that the embedded item has content that is duplicative of content for a previously fetched embedded item.

Abstract: Implementations include a batch-optimized render and fetch architecture. An example method performed by the architecture includes receiving a request from a batch process to render a web page and initializing a virtual clock and a task list for rendering the web page. The virtual clock stands still when a request for an embedded item is outstanding and when a task is ready to run. The method may also include generating a rendering result for the web page when the virtual clock matches a run time for a stop task in the task list, and providing the rendering result to the batch process. Another example method includes receiving a request from a batch process to render a web page, identifying an embedded item in the web page, and determining, based on a rewrite rule, that the embedded item has content that is duplicative of content for a previously fetched embedded item.

Abstract: A system may identify a set of first documents associated with an organization, and identify clusters to which the first documents belong. Each of a number of the identified clusters may include a group of documents that includes one of the first documents and one or more second documents associated with one or more different organizations. The system may determine a quality score for each of the documents in each of the identified clusters, and determine, for each of the number of the identified clusters, whether the quality score of the one of the first documents in the identified cluster is higher than the quality score of the one or more second documents in the identified cluster. The system may generate a proxy pad score based on the determinations, and store the proxy pad score.

Abstract: Implementations include a batch-optimized render and fetch architecture. An example method performed by the architecture includes receiving a request from a batch process to render a web page and initializing a virtual clock and a task list for rendering the web page. The virtual clock stands still when a request for an embedded item is outstanding and when a task is ready to run. The method may also include generating a rendering result for the web page when the virtual clock matches a run time for a stop task in the task list, and providing the rendering result to the batch process. Another example method includes receiving a request from a batch process to render a web page, identifying an embedded item in the web page, and determining, based on a rewrite rule, that the embedded item has content that is duplicative of content for a previously fetched embedded item.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating modified web documents. In one aspect, a method includes retrieving a web document including markup language code and having references to resources external to the web document and parsing the retrieved web document to interpret the markup language code and identify references to resources external to the retrieved web document. Data relating to at least a portion of the resources external to the retrieved web document are retrieved, and a modified web document including the retrieved data is generated and stored for use in responding to a request for retrieval of content of the web document.

Abstract: System and method for indexing rendered web page images. A web crawling engine stores the content and crawl time of a web page. A scheduling engine sends the content and crawl time to a rendering engine, and processes requests for embedded objects. If a requested object has been crawled, it sends the contents to the rendering engine. Otherwise it schedules the crawl of the object, and once the object is crawled, it resends the content and crawl time of the web page to the rendering engine. The rendering engine receives the content and crawl time of a web page, requests all embedded objects, and renders the web page to an image once all embedded objects are received.

Abstract: A system may identify a set of first documents associated with an organization, and identify clusters to which the first documents belong. Each of a number of the identified clusters may include a group of documents that includes one of the first documents and one or more second documents associated with one or more different organizations. The system may determine a quality score for each of the documents in each of the identified clusters, and determine, for each of the number of the identified clusters, whether the quality score of the one of the first documents in the identified cluster is higher than the quality score of the one or more second documents in the identified cluster. The system may generate a proxy pad score based on the determinations, and store the proxy pad score.

Abstract: A system may identify a set of related documents, identify one or more documents in the set of related documents that are sources of redirects, and identify organizations that are targets of the redirects. The system may also determine a redirect score based on the number of the identified documents that are sources of the redirects, determine a spam score based on a number of the organizations that are targets of the redirects, determine whether to classify the set of related documents as a bounce pad based on the redirect score and the spam score, and storing information associated with the result of the determination of whether to classify the set of related documents as a bounce pad.

Abstract: A URL inspector may determine a uniform resource locator (URL) which includes an indexable fragment. A URL separator may separate, from the URL which includes the indexable fragment, a base URL occurring prior to the indexable fragment. An indexer may process content of the base URL to obtain processed content thereof. A rendering system may render the processed content together with the URL which includes the indexable fragment to obtain rendered content. A content converter may convert the rendered content into indexable content.

Abstract: A URL inspector may determine a uniform resource locator (URL) which includes an indexable fragment. A URL separator may separate, from the URL which includes the indexable fragment, a base URL occurring prior to the indexable fragment. An indexer may process content of the base URL to obtain processed content thereof. A rendering system may render the processed content together with the URL which includes the indexable fragment to obtain rendered content. A content converter may convert the rendered content into indexable content.

Abstract: System and method for indexing rendered web page images. A web crawling engine stores the content and crawl time of a web page. A scheduling engine sends the content and crawl time to a rendering engine, and processes requests for embedded objects. If a requested object has been crawled, it sends the contents to the rendering engine. Otherwise it schedules the crawl of the object, and once the object is crawled, it resends the content and crawl time of the web page to the rendering engine. The rendering engine receives the content and crawl time of a web page, requests all embedded objects, and renders the web page to an image once all embedded objects are received.

Abstract: A system may identify a set of related documents, identify one or more documents in the set of related documents that are sources of redirects, and identify organizations that are targets of the redirects. The system may also determine a redirect score based on the number of the identified documents that are sources of the redirects, determine a spam score based on a number of the organizations that are targets of the redirects, determine whether to classify the set of related documents as a bounce pad based on the redirect score and the spam score, and storing information associated with the result of the determination of whether to classify the set of related documents as a bounce pad.

Abstract: A method for providing Internet protocol (IP) addresses with a domain name server (DNS) for multiple web servers of an Internet host. In one embodiment, each web server of an Internet host having multiple web servers is assigned a relative weight based on the individual processing power of the particular web server. As DNS resolution requests are received from client domains, the DNS returns IP addresses for the web servers such that the total number of times that each IP address of the web server is returned is proportional to the relative weight of each server relative to the total weight of all the servers. In another embodiment, the client domains that have most frequently accessed the web servers according to recent web server access logs are identified. In addition, the total number of accesses of each client domain is considered such that static arrays may be constructed to apportion the web servers among the client domains that most frequently access the web servers.

Abstract: A method for providing Internet protocol (IP) addresses with a domain name server (DNS) for multiple web servers of an Internet host. In one embodiment, each web server of an Internet host having multiple web servers is assigned a relative weight based on the individual processing power of the particular web server. As DNS resolution requests are received from client domains, the DNS returns IP addresses for the web servers such that the total number of times that each IP address of the web server is returned is proportional to the relative weight of each server relative to the total weight of all the servers. In another embodiment, the client domains that have most frequently accessed the web servers according to recent web server access logs are identified. In addition, the total number of accesses of each client domain is considered such that static arrays may be constructed to apportion the web servers among the client domains that most frequently access the web servers.

Abstract: A method for providing Internet protocol (IP) addresses with a domain name server (DNS) for multiple web servers of an Internet host. In one embodiment, each web server of an Internet host having multiple web servers is assigned a relative weight based on the individual processing power of the particular web server. As DNS resolution requests are received from client domains, the DNS returns IP addresses for the web servers such that the total number of times that each IP address of the web server is returned is proportional to the relative weight of each server relative to the total weight of all the servers. In another embodiment, the client domains that have most frequently accessed the web servers according to recent web server access logs are identified. In addition, the total number of accesses of each client domain is considered such that static arrays may be constructed to apportion the web servers among the client domains that most frequently access the web servers.

Abstract: A method for providing Internet protocol (IP) addresses with a domain name server (DNS) for multiple web servers of an Internet host. In one embodiment, each web server of an Internet host having multiple web servers is assigned a relative weight based on the individual processing power of the particular web server. As DNS resolution requests are received from client domains, the DNS returns IP addresses for the web servers such that the total number of times that each IP address of the web server is returned is proportional to the relative weight of each server relative to the total weight of all the servers. In another embodiment, the client domains that have most frequently accessed the web servers according to recent web server access logs are identified. In addition, the total number of accesses of each client domain is considered such that static arrays may be constructed to apportion the web servers among the client domains that most frequently access the web servers.