Abstract: Job listings retrieved from external sources are pre-processed prior to being stored in the search engine production database and duplicate records identified prior to storage in a production database for the search engine. Inter-source and intra-source hash values are calculated for each job listing and the values compared. Job listings having the same intra-source hash are judged to be duplicates of each other. Descriptions whose intra-source hash values do not match, but whose inter-source hash values match are judged to be duplicate candidates and subject to further processing. Suffixes for each such record are stored to a data structure such as a suffix array and the records searched and compared based on the suffix arrays. Records having a pre-determined number of contiguous words in common are judged to be duplicates. Duplicate records are identified before the data set is stored to the production data base.

Abstract: Job listings retrieved from external sources are pre-processed prior to being stored in the search engine production database and duplicate records identified prior to storage in a production database for the search engine. Inter-source and intra-source hash values are calculated for each job listing and the values compared. Job listings having the same intra-source hash are judged to be duplicates of each other. Descriptions whose intra-source hash values do not match, but whose inter-source hash values match are judged to be duplicate candidates and subject to further processing. Suffixes for each such record are stored to a data structure such as a suffix array and the records searched and compared based on the suffix arrays. Records having a pre-determined number of contiguous words in common are judged to be duplicates. Duplicate records are identified before the data set is stored to the production data base.

Abstract: A method of accumulating, processing, and classifying online job listings for searches by users. Job listings are automatically, and more effectively, categorized, allowing users to more quickly and easily search job listings. Also provided are expanded tools for more powerful job searching. First, online job listings are automatically accumulated and stored in a job database, such as by web crawlers or robots. The accumulated job listings are then “normalized,” i.e., additional metadata are appended to the stored job listings, which assists in both classifying jobs and in subsequent user searches. These normalized, or augmented, job listings are then classified into job categories by an automated classifier. Assigning jobs to categories in this manner allows for greater search functionality. For instance, users can search by job category, and find additional search terms via other jobs in the same category.