Abstract: A method for generating blocking filters for record linkage includes providing a training database and an initial filter comprising a set of blocking keys, generating a set of positive training examples from said training database using said initial blocking keys and a given scoring method, generating from said positive training examples one or more acceptable blocking filters with a high recall with respect to said training examples, estimating a reduction rate of each of said acceptable filters, and selecting those acceptable filters with the reduction rates that exceed a predetermined threshold.

Abstract: A method for data sensitive filtering in patient database searches includes providing a search criteria comprising one or more search locator fields, determining a retrieval formula from said search criteria that maximizes error tolerance in said search criteria while satisfying a predefined response time requirement, retrieving said candidate records from said database, wherein if no retrieval formula can be found to satisfies said response time requirements, requesting additional search criteria, scoring each said candidate record by comparing a search criteria locator field with a corresponding retrieved record field, determining whether said score of said candidate record exceeds a predefined threshold, and if said candidate score does exceed said threshold, adding said candidate record to a list of records to be returned in response to said search criteria.

Abstract: A method for probabilistic record linkage includes providing a record pair comprising a plurality of fields, providing a plurality of scenarios, each scenario relating to a distribution of patterns among a plurality of attribute statuses, and comparing the record pair to determine a record difference. The method includes determining a probability of a status for each of a plurality of attributes based on the distance metric of the plurality of fields, wherein each field corresponds to a respective attribute, wherein the field is observable and the attribute is hidden, determining a probability of each scenario based on the probability of the status for each attribute and the Bayesian net representing the probabilistic model on the relationship between scenarios and attributes, and outputting a probability of duplication or non-duplication of the record pair determined from the probabilities of the plurality of scenarios.

Abstract: A system and method for optimizing database access for record linkage by tiling the space of record pairs are provided, the system including a processor, a segmentation and pairing unit in signal communication with the processor for segmenting database data into data segments and pairing the data segments into data quadrants, and a duplicate detection unit in signal communication with the processor for detecting duplicates for each quadrant; and the method including receiving database data, segmenting the database data into data segments, pairing the data segments into data quadrants, and detecting duplicates for each quadrant.

Abstract: A method for determining a blocking key includes selecting, randomly, a plurality of record pairs from a pair space that can be formed from a plurality of records of a database, scoring the plurality of record pairs, and comparing a score of each of the plurality of record pairs to a threshold to determine a label for each record pair. The method further includes comparing, character-by-character, each field of each of the plurality of record pairs, wherein a result of the comparison is a binary vector entered in a binary vector matrix, and determining a blocking key based on the binary vector matrix.