Abstract: Dynamic identification and removal of potential stopwords from a search query is provided. A search query comprising plural terms is received. A possible stopword among the terms that appears in a predefined list of possible stopwords is identified. Query splits are determined based on one or more query patterns, each query split comprising a location portion using a first subset of the plural terms and a subject portion using a second subset. Either the location portion or the subject portion of the query splits is modified, by removing the possible stopword from that portion. The search engine is queried with the location portion and the subject portion for each of the query splits, and scored search results are obtained for each of the query splits. The search results returned for at least one of the query splits is selected to return as a response to the search query.

Abstract: Methods, systems and computer-readable storage medium for determining a crawling schedule. In an aspect, a method includes obtaining crawl history data for a Web site having Web pages, determining a status of the Web pages, determining a total quantity of Web pages that have a status of deleted, calculating a probability that another Web page of the Web site will be removed based on the total quantity, and storing data associating the calculated probability with the Web site. The method can further include determining, for a plurality of sets of the previous time periods, a respective crawl penalty as a combination of a penalty for crawling the Web site and a penalty for showing a deleted Web page based on the calculated probability, and determining a re-crawl schedule based on the crawl penalties.

Abstract: Methods, systems and computer-readable storage medium for determining a crawling schedule. In an aspect, a method includes obtaining crawl history data for a Web site having Web pages, determining a status of the Web pages, determining a total quantity of Web pages that have a status of deleted, calculating a probability that another Web page of the Web site will be removed based on the total quantity, and storing data associating the calculated probability with the Web site. The method can further include determining, for a plurality of sets of the previous time periods, a respective crawl penalty as a combination of a penalty for crawling the Web site and a penalty for showing a deleted Web page based on the calculated probability, and determining a re-crawl schedule based on the crawl penalties.