Abstract: Documents in a local computing system are parsed for indexing and sent to a cloud-based computing system. The cloud-based search service indexes documents that are parsed on the local computing system, as well as documents that reside in the cloud-based computing system. When a user launches a search, the search is executed against a single search index in the cloud-based computing system, which includes an index of content sources from both the local computing system and the cloud-based computing system.

Abstract: Documents in a local computing system are parsed for indexing and sent to a cloud-based computing system. The cloud-based search service indexes documents that are parsed on the local computing system, as well as documents that reside in the cloud-based computing system. When a user launches a search, the search is executed against a single search index in the cloud-based computing system, which includes an index of content sources from both the local computing system and the cloud-based computing system.

Abstract: An alert search mechanism is used with search engines such as a crawler to search for desired documents and/or resources. Particular documents are found by using search queries. The search mechanism track values of a set of relevant properties in queries. Whenever a document is searched for by the system, the values of these set of properties are matched with the old value. If there is no match, this is an indication that the document has changed.

Abstract: A smart algorithm for processing transaction from a crawl queue. If the crawler has in memory a predetermined number of URLs for a given host, the crawler reads from the crawl queue URLs from other hosts. As a result the crawler processes multiple hosts concurrently, and thus, uses machine resources more effectively and efficiently to process the URLs. The smart algorithm can further consider other criteria in deciding which URLs to read from the queue. These criteria can include the response time for each repository (host) the crawler processes. Additionally, the crawler can allocate its resources according to content groups (e.g., two pools), one group for faster content delivery and the second group one for slower content delivery. Thus, crawler resources can be partitioned or divided across different pools depending on repository response time. Other criteria can be provided and considered as well.

Abstract: Balancing resource allocations based on priority may be provided. First, a plurality of repositories may be divided into at least two categories. Next, a first portion of computing resources may be dedicated to a first one of the at least two categories. Then a second portion of the computing resources may be dedicated to a second one of the at least two categories. A crawl may then be performed on the plurality of repositories with the computing resources.

Abstract: A process takes advantage of a structure of a server hosting a network site that includes a change log stored in a database to batch index documents for search queries. The content of the site is batched and shipped in bulk from the server to an indexer. The change log keeps track of the changes to the content of the site. The indexer incrementally requests updates to the index using the change log and batches the changes so that the bandwidth usage and processor overhead costs are reduced.

Abstract: An alert search mechanism is used with search engines such as a crawler to search for desired documents and/or resources. Particular documents are found by using search queries. The search mechanism track values of a set of relevant properties in queries. Whenever a document is searched for by the system, the values of these set of properties are matched with the old value. If there is no match, this is an indication that the document has changed.

Abstract: A process takes advantage of a structure of a server hosting a network site that includes a change log stored in a database to batch index documents for search queries. The content of the site is batched and shipped in bulk from the server to an indexer. The change log keeps track of the changes to the content of the site. The indexer incrementally requests updates to the index using the change log and batches the changes so that the bandwidth usage and processor overhead costs are reduced.