Abstract: Techniques for optimizing the performance of a webpage crawler are described. According to various embodiments, historical web crawler performance data is accessed, the data describing a performance of a web crawler during various time periods in one or more prior days. A capacity of the web crawler to fulfill uniform resource locator (URL) crawl requests for an upcoming given time period is then estimated, based on the historical web crawler performance data. Thereafter, a plurality of URL crawl requests are distributed to the web crawler during the upcoming given time period, based on the estimated capacity of the web crawler.

Abstract: Techniques for optimizing the performance of a webpage crawler are described. According to various embodiments, historical web crawler performance data is accessed, the data describing a performance of a web crawler during various time periods in one or more prior days. A capacity of the web crawler to fulfil uniform resource locator (URL) crawl requests for an upcoming given time period is then estimated, based on the historical web crawler performance data. Thereafter, a plurality of URL crawl requests are distributed to the web crawler during the upcoming given time period, based on the estimated capacity of the web crawler.

Abstract: Techniques for providing improved product suggestions for related items are described. According to various embodiments, a product listing webpage associated with a retailer website that describes a specific product may be crawled, the specific product being included in a product inventory of the retailer website. Thereafter, product relations information associated with the specific product may be identified in the product listing webpage, the product relations information describing a group of one or more additional products in the product inventory having a particular relationship with the specific product. The product relations information may then be transposed to a second product inventory associated with a second retailer.

Abstract: Techniques for competitive pricing analysis and inventory management are described. According to various exemplary embodiments, a competitive pricing system is configured to crawl competitor websites for comparative pricing information at various time intervals. Moreover, the competitive pricing system is configured to determine if a price for an item on a home retailer website represents a “deal”, based on information crawled from competitor websites. According to various exemplary embodiments, a managed inventory repository system may enable improved identification of deals and specials within an inventory of a retailer website. For example, the managed inventory repository system may perform data mining operations to identify deals or specials offered for inventory items on a home retailer website. In some embodiments, a “special” can be defined in a variety of ways to suit different business units, campaigns, metrics, etc.