Abstract: Record clustering is performed by learning from verified clusters which are used as the source of training data in a deduplication workflow utilizing supervised machine learning.

Abstract: Record clustering is performed for a collection of records using training rules, training-rule labels, training data created from a sample of pairs of records, a pair-wise classifier, and a clustering algorithm. Record clustering is also performed for a collection of records using prediction rules, prediction-rule labels, a pair-wise classifier, and a clustering algorithm.

Abstract: Given a number of records and a number of target classes to which these records belong to, a (weakly) supervised machine learning classification method leverages known possibly dirty classification rules, efficiently and accurately learns a classification model from training data, and applies the learned model to the data records to predict their classes.

Abstract: An end-to-end data curation system and the various methods used in linking, matching, and cleaning large-scale data sources. The goal of this system is to provide scalable and efficient record deduplication. The system uses a crowd of experts to train the system. The system operator can optionally provide a set of hints to reduce the number of questions send to the experts. The system solves the problem of schema mapping and record deduplication a holistic way by unifying these problems into a unified linkage problem.

Abstract: A collection of clusters are selected to be used in training in an active learning workflow when using clusters to train supervised entity resolution in data sets. A collection of records is provided wherein each record in the collection has a cluster membership. A collection of record pairs is also provided, each record pair containing two distinct records from the collection of records, and each record pair having a similarity score. A collection of clusters is generated with uncertainty from the collection of records and the collection of record pairs. A subset of the collection of clusters with uncertainty is then selected using weighted sampling, wherein a function of the cluster uncertainty is used as the weight in the weighted sampling. The subset of the collection of clusters with uncertainty is the collection of clusters for training in and active learning workflow when using clusters to train supervised entity resolution in data sets.

Abstract: Methods are provided to represent proposed changes to clusterings for ease of review, as well as tools to help subject matter experts identify clusters that warrant review versus those that do not. These tools make overall assessment of proposed clustering changes and targeted curation practical at large scale. Use of these tools and method enables efficient data management operations when dealing with extreme scale, such as where entity resolution involves clusterings created from data sources involving millions of entities.

Abstract: A method is provided for producing a record clustering with estimated accuracy metrics with confidence intervals. These metrics can be used to determine whether a clustering should be accepted as the output of the system, and whether model training is necessary to meet desired clustering accuracy. A collection of test records is used in the process, wherein each test record is a member of a collection of input records.

Abstract: Given a number of records and a number of target classes to which these records belong to, a (weakly) supervised machine learning classification method leverages known possibly dirty classification rules, efficiently and accurately learns a classification model from training data, and applies the learned model to the data records to predict their classes.

Abstract: Fast record deduplication is accomplished by providing as an input, data records having multiple attributes, and local similarity functions of individual attributes with local similarity thresholds. Bin IDs are then generated based on the local similarity functions and the local similarity thresholds. The Bin IDs are unique identifiers of a respective bin of records, and the bin of records is a set of records that are possibly pairwise similar. Local candidate pairs are identified based on data records that share Bin IDs. The local candidate pairs are aggregated to produce a set of global candidate pairs. The set of global candidate pairs are filtered by deciding whether a pair of data records represents a duplicate.

Abstract: Record clustering is performed by learning from verified clusters which are used as the source of training data in a deduplication workflow utilizing supervised machine learning.

Abstract: A data curation system is provided that includes various methods to enable efficient reuse of human and machine effort. To reuse effort, various facilities are presented that model, save, and allow for querying of provenance and state information of a curation workflow and allow for incremental, stateful transitions of the data and metadata thereof.

Abstract: A system and method of use resolves the frustration of repeated manual work during schema mapping. The system utilizes a transformation graph—a collection of nodes (unified attributes) and edges (transformations) in which source attributes are mapped and transformed. The system further leverages existing mappings and transformations for the purpose of suggesting to a user the optimal paths (i.e., the lowest cost paths) for mapping new sources, which is particularly useful when new sources share similarity with previously mapped sources and require the same transformations. As such, the system also promotes an evolving schema by allowing users to select which unified attributes they want to include in a target schema at any time. The system addresses the technical challenge of finding optimal transformation paths and how to present these to the user for evaluation.

Abstract: An end-to-end data curation system and the various methods used in linking, matching, and cleaning large-scale data sources. The goal of this system is to provide scalable and efficient record deduplication. The system uses a crowd of experts to train the system. The system operator can optionally provide a set of hints to reduce the number of questions sent to the experts. The system solves the problem of schema mapping and record deduplication in a holistic way by unifying these problems into a unified linkage problem.

Abstract: Given a local distance metric for geospatial features, a binning is produced that is guaranteed to label features within a given distance threshold with the same bin, while labeling a minimum number of features separated by a distance that is greater than the threshold with the same bin.

Abstract: A system and method of use resolves the frustration of repeated manual work during schema mapping. The system utilizes a transformation graph—a collection of nodes (unified attributes) and edges (transformations) in which source attributes are mapped and transformed. The system further leverages existing mappings and transformations for the purpose of suggesting to a user the optimal paths (i.e., the lowest cost paths) for mapping new sources, which is particularly useful when new sources share similarity with previously mapped sources and require the same transformations. As such, the system also promotes an evolving schema by allowing users to select which unified attributes they want to include in a target schema at any time. The system addresses the technical challenge of finding optimal transformation paths and how to present these to the user for evaluation.

Abstract: Structured metadata is automatically captured regarding issues reported by a user when the user interacts with application software for presentation, analysis, or management of structured data. The reported issues correspond to structured data that is displayed by the application software. During user interaction with the application software, a user interface display screen is presented that includes one or more fields for reporting an issue with respect to structured data that is presently being displayed by the application software. Structured metadata is then automatically captured related to the reported issue. The structured metadata includes at least a location within the structured data for the reported issue. Records are stored of each reported issue in a database. Each record includes the reported issue, and the automatically captured structured metadata related to the reported issue.

Abstract: Given a number of records and a number of target classes to which these records belong to, a (weakly) supervised machine learning classification method leverages known possibly dirty classification rules, efficiently and accurately learns a classification model from training data, and applies the learned model to the data records to predict their classes.

Abstract: A very efficient computer system is presented to generate all pairs of records that have a certain similarity. Similarity is defined in terms of the textual similarity of the record attributes and/or absolute difference for numeric record attributes. Software assigns each record to a number of bins, and then compares pairs of records that belong to the same bin. This is more efficient than comparing all pairs of records since the number of records compared to each other is much smaller.

Abstract: An end-to-end data curation system and the various methods used in linking, matching, and cleaning large-scale data sources. The goal of this system is to provide scalable and efficient record deduplication. The system uses a crowd of experts to train the system. The system operator can optionally provide a set of hints to reduce the number of questions send to the experts. The system solves the problem of schema mapping and record deduplication a holistic way by unifying these problems into a unified linkage problem.