Abstract: Embodiments of the disclosure are directed to providing a single source for adverse event data by taking a layered approach to standardizing, harmonizing and detecting duplicates across multiple data sources at different scales. In one embodiment, a method is provided. The method includes parsing datasets stored in a data store. These datasets are enriched using standardization and normalization. In the candidate duplicates and feature engineering step, the method may join send the data to hashing algorithm to generate candidate duplicates. Features are extracted from each duplicate candidate pair using the term-pair set adjustment technique. These candidates and associate features are sampled using a sampling technique and are labeled as duplicates or non-duplicates. Upon a conflict in labels, a conflict resolution strategy is applied to create a master list of duplicate pairs. A classifier is trained on the master list to classify the rest of the candidate pairs as duplicates/non-duplicates.