Abstract: A method of analyzing documents or relationships between documents includes receiving a notification of an available metadata document containing information about one or more network-accessible documents, obtaining a document format indicator associated with the metadata document, selecting a document crawler using the document format indicator, and crawling at least some of the network-accessible documents using the selected document crawler.

Abstract: Methods, systems and apparatus, including computer program products, for generating cookies. Multiple advertisements for multiple advertisers are stored on an ad server. Hostnames specific to each of the multiple advertisers are generated dynamically. The generated hostnames identify the ad server. A reference is generated identifying a particular advertisement of the multiple advertisements and the ad server. The ad server is identified in the reference with the generated hostname associated with the advertiser of the advertisement. The reference is provided to a client. A request is received from the client for the advertisement. The request includes the generated hostname. In response to the request, a client cookie, which includes a unique identifier, is sent to the client.

Abstract: Methods, systems and apparatus, including computer program products, for generating cookies. Multiple advertisements for multiple advertisers are stored on an ad server. Hostnames specific to each of the multiple advertisers are generated dynamically. The generated hostnames identify the ad server. A reference is generated identifying a particular advertisement of the multiple advertisements and the ad server. The ad server is identified in the reference with the generated hostname associated with the advertiser of the advertisement. The reference is provided to a client. A request is received from the client for the advertisement. The request includes the generated hostname. In response to the request, a client cookie, which includes a unique identifier, is sent to the client.

Abstract: Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.

Abstract: A client-side application (such as a browser, a browser plug-in, a browser toolbar plug-in, etc. on an end user's computer) is used to support the serving of content-relevant ads to the client device. The client-side application may provide such support by sending document information (such as a document identifier, document content, content relevance information, etc.) to a content ad server. The client-side application may also be used to combine content of the document and the content-relevant ads. For example, the client-side application may combine content of the document and the ads in a window (e.g., in a browser window), may provide the ads in a window above, below, adjacent to a document window, may provide the ads in “chrome” of the browser, etc.

Abstract: Methods, systems and apparatus, including computer program products, for generating cookies. Multiple advertisements for multiple advertisers are stored on an ad server. Hostnames specific to each of the multiple advertisers are generated dynamically. The generated hostnames identify the ad server. A reference is generated identifying a particular advertisement of the multiple advertisements and the ad server. The ad server is identified in the reference with the generated hostname associated with the advertiser of the advertisement. The reference is provided to a client. A request is received from the client for the advertisement. The request includes the generated hostname. In response to the request, a client cookie, which includes a unique identifier, is sent to the client.

Abstract: A method and system for allowing a resource associated with a target document usable for ad rendering by a first entity to be used for rendering of content by a second entity. Depending on one or more conditions, the first entity either uses the resource to render at least one ad or allows a second entity to use at least a portion of the resource to render content.

Abstract: Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.

Abstract: Methods and systems for a sitemap generating client for web crawlers are described. The client accesses one or more sources of document information about the documents available on a website, such as the file system, access logs, or pre-made URL lists. Document information is extracted from the sources and one or more sitemaps are generated based on the extracted document information. A notification is transmitted to a remote computer, informing that the sitemap(s) are available for access and likely have been updated. If the remote computer is associated with a web crawler, the remote computer may access the sitemap(s) and use the sitemaps to schedule a crawl of documents included or available on the website.

Abstract: Advertisers are permitted to put targeted ads on page on the web (or some other document of any media type). The present invention may do so by (i) obtaining content that includes available spots for ads, (ii) determining ads relevant to content, and/or (iii) combining content with ads determined to be relevant to the content.

Abstract: A client-side application (such as a browser, a browser plug-in, a browser toolbar plug-in, etc. on an end user's computer) is used to support the serving of content-relevant ads to the client device. The client-side application may provide such support by sending document information (such as a document identifier, document content, content relevance information, etc.) to a content ad server. The client-side application may also be used to combine content of the document and the content-relevant ads. For example, the client-side application may combine content of the document and the ads in a window (e.g., in a browser window), may provide the ads in a window above, below, adjacent to a document window, may provide the ads in “chrome” of the browser, etc.

Abstract: Advertisers are permitted to put targeted ads on page on the web (or some other document of any media type). The present invention may do so by (i) obtaining content that includes available spots for ads, (ii) determining ads relevant to content, and/or (iii) combining content with ads determined to be relevant to the content.

Abstract: Methods, systems, and apparatus, including computer program products, for generating sitemaps. The method includes scanning network traffic between a server and one or more clients requesting resources from the server, the network traffic including resource request messages from the one or more clients and resources served by the server in response to the resource request messages. The method also includes automatically extracting data from the traffic served by the server to the one or more clients, the extracted data including one or more Uniform Resource Locators that identify the resources served by the server to the one or more clients. The method automatically generates a sitemap from the extracted data, and stores the sitemap in a computer-readable memory.

Abstract: A method and system for allowing a resource associated with a target document usable for ad rendering by a first entity to be used for rendering of content by a second entity. Depending on one or more conditions, the first entity either uses the resource to render at least one ad or allows a second entity to use at least a portion of the resource to render content.

Abstract: Methods and systems for a sitemap generating client for web crawlers are described. The client accesses one or more sources of document information about the documents available on a website, such as the file system, access logs, or pre-made URL lists. Document information is extracted from the sources and one or more sitemaps are generated based on the extracted document information. A notification is transmitted to a remote computer, informing that the sitemap(s) are available for access and likely have been updated. If the remote computer is associated with a web crawler, the remote computer may access the sitemap(s) and use the sitemaps to schedule a crawl of documents included or available on the website.

Abstract: Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.

Abstract: Methods and systems for a sitemap generating client for web crawlers are described. The client accesses one or more sources of document information about the documents available on a website, such as the file system, access logs, or pre-made URL lists. Document information is extracted from the sources and one or more sitemaps are generated based on the extracted document information. A notification is transmitted to a remote computer, informing that the sitemap(s) are available for access and likely have been updated. If the remote computer is associated with a web crawler, the remote computer may access the sitemap(s) and use the sitemaps to schedule a crawl of documents included or available on the website.

Abstract: Methods and systems for a web crawler scheduler that utilizes sitemaps from websites are described. A web crawler scheduling system receives a notification from a website or web server. In response to the notification, the system accesses one or more sitemap(s) for documents associated with the website or web server. The system schedules crawls of the documents based on information identified from the sitemaps. The system crawls at least a subset of the documents scheduled for crawling.

Abstract: Advertisers are permitted to put targeted ads on page on the web (or some other document of any media type). The present invention may do so by (i) obtaining content that includes available spots for ads, (ii) determining ads relevant to content, and/or (iii) combining content with ads determined to be relevant to the content.

Abstract: Advertisers are permitted to put targeted ads on page on the web (or some other document of any media type). The present invention may do so by (i) obtaining content that includes available spots for ads, (ii) determining ads relevant to content, and/or (iii) combining content with ads determined to be relevant to the content.