Abstract: A method, system and computer program product for data deduplication in a travel and transportation data processing system includes loading into memory for comparison from a database of a multiplicity of multi-field records, a pair of two different multi-field records and submitting the pair to a similarity model that correlates a companion passenger with a specified individual so as to indicate a probability of duplication whenever the companion passenger appears in a pair of records submitted to the model for comparison. Thereafter, if the model produces a high probability, the pair is automatically merged into a single record without manual intervention, but if the model produces a medium probability, the pair is placed in a queue pending manual intervention and manual merging, and if the similarity model produces a low probability, the pair is omitted from consideration of merging. Finally, the process repeats for each other pair of different multi-field records in the database.

Abstract: A method, system and computer program product for data deduplication in a travel and transportation data processing system includes loading into memory for comparison from a database of a multiplicity of multi-field records, a pair of two different multi-field records and submitting the pair to a similarity model that correlates a companion passenger with a specified individual so as to indicate a probability of duplication whenever the companion passenger appears in a pair of records submitted to the model for comparison. Thereafter, if the model produces a high probability, the pair is automatically merged into a single record without manual intervention, but if the model produces a medium probability, the pair is placed in a queue pending manual intervention and manual merging, and if the similarity model produces a low probability, the pair is omitted from consideration of merging. Finally, the process repeats for each other pair of different multi-field records in the database.