Abstract: A classifier may be integrated into a pipeline of a general web crawler. The classifier may classify crawled webpages as either video pages or non-video pages. Video pages and information regarding domain importance may be aggregated. Ones of the domains of the video pages may be selected based on domain importance rankings. Webpages of the selected domains may be randomly sampled. The sampled webpages may be structurally analyzed and hint information may be generated with respect to each of the selected domains. The hint information may guide a deep crawling operation for discovering all video pages within the selected domains. Video links within the video pages may be found, one or more videos may be downloaded, and one or more representations of the one or more videos may be indexed.

Abstract: A classifier may be integrated into a pipeline of a general web crawler. The classifier may classify crawled webpages as either video pages or non-video pages. Video pages and information regarding domain importance may be aggregated. Ones of the domains of the video pages may be selected based on domain importance rankings. Webpages of the selected domains may be randomly sampled. The sampled webpages may be structurally analyzed and hint information may be generated with respect to each of the selected domains. The hint information may guide a deep crawling operation for discovering all video pages within the selected domains. Video links within the video pages may be found, one or more videos may be downloaded, and one or more representations of the one or more videos may be indexed.