Abstract: A search engine is configured to return search results with increased diversity among the search results. When a query is received from a client device, items are identified as search results for the query and grouped into categories. The categories are also grouped into classes. A user interface is generated with items grouped together by category and categories grouped together by class. The user interface is returned to the client device as a response to the query.

Abstract: A search engine is provided that generates search results proportional to a natural distribution. Search results identified and ranked by the search engine for a search query are redistributed so that a fixed number of top ranked search results include search result attributes proportional to the natural distribution, as determined from interaction tracking of prior search results. A natural distribution index mapping prior search queries to search result attributes and a proportional representation of interactions with the search result attributes is generated. The natural distribution is determined from the proportional representation of interactions. The search engine redistributes the fixed number of top ranked search results by minimizing a gap between the ranked set of search results for the search query and the natural distribution.

Abstract: A search engine is configured to return increased diversity results based on past user interactions with search results. For a given query, historical data is analyzed to generate an item score describing a past quantity of users that navigated to a given page of an item. The historical data can further be used to generate a category score describing a past quantity of users that navigated to a given category of items. The category of items can be analyzed to generate a diversity score describing their diversity with respect to each other. Results for the given query can be arranged using items scores, category scores, and diversity scores.

Abstract: A search engine is configured to return increased diversity results based on past user interactions with search results. For a given query, historical data is analyzed to generate an item score describing a past quantity of users that navigated to a given page of an item. The historical data can further be used to generate a category score describing a past quantity of users that navigated to a given category of items. The category of items can be analyzed to generate a diversity score describing their diversity with respect to each other. Results for the given query can be arranged using items scores, category scores, and diversity scores.

Abstract: A method and system for building an elastic cloud web server farm. The method includes registering a web application on a serving cloud and copying the web application to a distributed store. A load of the web application is specified, and a plurality of nodes is added for the web application based on the load. A web server corresponding to a node of the plurality of nodes is then initialized. A web request is received from a user and a web server is selected from a list of available web servers to process the web request. The web request is further transmitted to the web server. A web response, based on the web request, is transmitted back to the user. The system includes a central registry, a distributed store, a process coordinator, one or more web servers, and a router.

Abstract: An approach is provided for creating a framework for executing code in nodes of a distributed storage platform. An in-node code framework receives at least one job for execution over a distributed computing platform. The in-node code framework determines at least one distributed storage platform containing at least a portion of the data for executing the at least one job. The in-node then processes and/or facilitates a processing of the at least one job to determine code for executing at one or more storage nodes of the at least one distributed storage platform, wherein the code supports the at least one job.

Abstract: A method and system for building an elastic cloud web server farm. The method includes registering a web application on a serving cloud and copying the web application to a distributed store. A load of the web application is specified, and a plurality of nodes is added for the web application based on the load. A web server corresponding to a node of the plurality of nodes is then initialized. A web request is received from a user and a web server is selected from a list of available web servers to process the web request. The web request is further transmitted to the web server. A web response, based on the web request, is transmitted back to the user. The system includes a central registry, a distributed store, a process coordinator, one or more web servers, and a router.

Abstract: A method and system for building an elastic cloud web server farm. The method includes registering a web application on a serving cloud and copying the web application to a distributed store. A load of the web application is specified, and a plurality of nodes is added for the web application based on the load. A web server corresponding to a node of the plurality of nodes is then initialized. A web request is received from a user and a web server is selected from a list of available web servers to process the web request. The web request is further transmitted to the web server. A web response, based on the web request, is transmitted back to the user. The system includes a central registry, a distributed store, a process coordinator, one or more web servers, and a router.

Abstract: An approach is provided for creating a framework for executing code in nodes of a distributed storage platform. An in-node code framework receives at least one job for execution over a distributed computing platform. The in-node code framework determines at least one distributed storage platform containing at least a portion of the data for executing the at least one job. The in-node then processes and/or facilitates a processing of the at least one job to determine code for executing at one or more storage nodes of the at least one distributed storage platform, wherein the code supports the at least one job.

Abstract: A method and system for building an elastic cloud web server farm. The method includes registering a web application on a serving cloud and copying the web application to a distributed store. A load of the web application is specified, and a plurality of nodes is added for the web application based on the load. A web server corresponding to a node of the plurality of nodes is then initialized. A web request is received from a user and a web server is selected from a list of available web servers to process the web request. The web request is further transmitted to the web server. A web response, based on the web request, is transmitted back to the user. The system includes a central registry, a distributed store, a process coordinator, one or more web servers, and a router.

Abstract: A method for performing activities on a web site is disclosed. A user's browsing activities on a web site are captured. The user's browsing activities includes affixing labels to web pages and filling out forms. The captured activities are analyzed for patterns. Rules for performing activities on a web site are generated based on the patterns. Further activities are performed on the web site according to the rules and content from the web site is fetched. The fetched content is used in various web service applications, including crawlers.

Abstract: Feed Processing. An example method of processing a feed stored in a storage device includes receiving an input feed. Each record of the feed is associated with one or more unique identifiers. A first unique identifier for each record of the input feed is then generated. Each record of the input feed and each record of the feed is grouped as changed or not changed based on the first unique identifier for each record of the input feed and a first unique identifier for each record of the feed. A second unique identifier for each record of the input feed grouped as changed is also generated. Each record of the input feed grouped as changed and each record of the feed grouped as changed are then regrouped based on the second unique identifier for each record of the input feed and a second unique identifier for each record of the feed. Further, the feed is updated based on the regrouping, whereby a user accessing a record from the storage device obtains updated version of the record.

Abstract: A method for providing feedback to a web crawler is provided. A content processor determines whether a crawled web page is useful to an application. This determination is passed to a learning module. The learning module analyzes crawled web pages and the determinations of usefulness made by the content processor and generates rules for crawling more useful web pages and less non-useful web pages. The learning module provides these rules to the crawler, which applies them in making crawling decisions. Rules expire after a period of time. Paths from a web site's main web page to useful web pages are preserved. A token tree is constructed to facilitate the generation of rules.

Abstract: A method for performing activities on a web site is disclosed. A user's browsing activities on a web site are captured. The user's browsing activities includes affixing labels to web pages and filling out forms. The captured activities are analyzed for patterns. Rules for performing activities on a web site are generated based on the patterns. Further activities are performed on the web site according to the rules and content from the web site is fetched. The fetched content is used in various web service applications, including crawlers.

Abstract: Automatically determining values for fields in an electronic document. In one embodiment, an intelligent form filler automatically fills in at least some of the fields based on as set of rules associated with a domain. A particular set of domain rules may have class definitions that define how to classify a field for that domain and group definitions that define how to group fields. The domain rules also describe how values can be determined for the fields, based on the classifications, groupings, and other factors. In one embodiment, the intelligent form filler submits more than one form such that different combinations of values are submitted. The values that were used to fill in the form(s) may be provided to an extraction tool, which use the values to facilitate extraction of information from a document returned in response to submitting the form.