Abstract: Methods, systems, and computer program products for vision-based cell structure recognition using hierarchical neural networks and cell boundaries to structure clustering are provided herein. A computer-implemented method includes detecting a style of the given table using at least one style classification model; selecting, based at least in part on the detected style, a cell detection model appropriate for the detected style; detecting cells within the given table using the selected cell detection model; and outputting, to at least one user, information pertaining to the detected cells comprising image coordinates of one or more bounding boxes associated with the detected cells.

Abstract: Methods, systems, and computer program products for cooperative neural networks with spatial containment constraints are provided herein. A computer-implemented method includes dividing a processing task into multiple sub-tasks; training multiple independent neural networks, such that at least some of the multiple sub-tasks correspond to different ones of the multiple independent neural networks; defining, via implementing constraint-based domain knowledge related to the processing task in connection with the multiple independent neural networks, a constraint loss for a given one of the multiple sub-tasks, the constraint loss being dependent on output from at least one of the other multiple sub-tasks; and effecting re-training of at least a portion of the multiple independent neural networks by incorporating the constraint loss into at least one of the multiple independent neural networks.

Abstract: Methods, systems, and computer program products for vision-based cell structure recognition using hierarchical neural networks and cell boundaries to structure clustering are provided herein. A computer-implemented method includes detecting a style of the given table using at least one style classification model; selecting, based at least in part on the detected style, a cell detection model appropriate for the detected style; detecting cells within the given table using the selected cell detection model; and outputting, to at least one user, information pertaining to the detected cells comprising image coordinates of one or more bounding boxes associated with the detected cells.

Abstract: Methods, systems, and computer program products for vision-based cell structure recognition using hierarchical neural networks and cell boundaries to structure clustering are provided herein. A computer-implemented method includes detecting a style of the given table using at least one style classification model; selecting, based at least in part on the detected style, a cell detection model appropriate for the detected style; detecting cells within the given table using the selected cell detection model; and outputting, to at least one user, information pertaining to the detected cells comprising image coordinates of one or more bounding boxes associated with the detected cells.

Abstract: A computer-implemented method is provided that includes identifying an input dataset formatted as an input matrix, the input matrix including a plurality of rows and a plurality of columns. The computer-implemented method also includes dividing the input matrix into a plurality of input matrix blocks. Further, the computer-implemented method includes distributing the input matrix blocks to a plurality of different machines across a distributed filesystem, and sampling, by at least two of the different machines in parallel, at least two of the input matrix blocks. Finally, the computer-implemented method includes generating at least one sample matrix based on the sampling of the at least two of the input matrix blocks.

Abstract: Methods, systems, and computer program products for vision-based cell structure recognition using hierarchical neural networks and cell boundaries to structure clustering are provided herein. A computer-implemented method includes detecting a style of the given table using at least one style classification model; selecting, based at least in part on the detected style, a cell detection model appropriate for the detected style; detecting cells within the given table using the selected cell detection model; and outputting, to at least one user, information pertaining to the detected cells comprising image coordinates of one or more bounding boxes associated with the detected cells.

Abstract: Methods, systems, and computer program products for cooperative neural networks with spatial containment constraints are provided herein. A computer-implemented method includes dividing a processing task into multiple sub-tasks; training multiple independent neural networks, such that at least some of the multiple sub-tasks correspond to different ones of the multiple independent neural networks; defining, based at least in part on constraint-based domain knowledge related to the processing task, at least one constraint loss for a given one of the multiple sub-tasks, the at least one constraint loss being dependent on output from at least one of the other multiple sub-tasks; and re-training at least a portion of the multiple independent neural networks, the re-training being dependent on using the at least one constraint loss.

Abstract: A computer-implemented method is provided that includes identifying an input dataset formatted as an input matrix, the input matrix including a plurality of rows and a plurality of columns. The computer-implemented method also includes dividing the input matrix into a plurality of input matrix blocks. Further, the computer-implemented method includes distributing the input matrix blocks to a plurality of different machines across a distributed filesystem, and sampling, by at least two of the different machines in parallel, at least two of the input matrix blocks. Finally, the computer-implemented method includes generating at least one sample matrix based on the sampling of the at least two of the input matrix blocks.

Abstract: A computer-implemented method is provided that includes identifying an input dataset formatted as an input matrix, the input matrix including a plurality of rows and a plurality of columns. The computer-implemented method also includes dividing the input matrix into a plurality of input matrix blocks. Further, the computer-implemented method includes distributing the input matrix blocks to a plurality of different machines across a distributed filesystem, and sampling, by at least two of the different machines in parallel, at least two of the input matrix blocks. Finally, the computer-implemented method includes generating at least one sample matrix based on the sampling of the at least two of the input matrix blocks.

Abstract: Described herein are methods, systems and computer program products for entity resolution. Entity resolution, also known as entity matching or record linkage, seeks to identify equivalent data objects between or among datasets. An example method includes creating a deterministic model by defining an entity to be resolved, selecting two datasets for comparison, defining matching predicates for attributes of the datasets to select a set of candidate matches, and defining a precedence rule for the candidate matches to select a subset of the candidate matches. The method includes running the deterministic model on the two datasets. Running the deterministic model includes applying the matching predicates and the precedence rule to data in the datasets that correspond to the attributes. The method also includes applying a cardinality rule to results of the running, and outputting the matching candidates for which the cardinality rule is satisfied.

Abstract: In a method for analyzing a large data set using a statistical computing environment language operation, a processor generates code from the statistical computing environment language operation that can be understood by a software system for processing machine learning algorithms in a MapReduce environment. A processor transfers the code to the software system for processing machine learning algorithms in a MapReduce environment. A processor invokes execution of the code with the software system for processing machine learning algorithms in a MapReduce environment.

Abstract: A computer-implemented method is provided that includes identifying an input dataset formatted as an input matrix, the input matrix including a plurality of rows and a plurality of columns. The computer-implemented method also includes dividing the input matrix into a plurality of input matrix blocks. Further, the computer-implemented method includes distributing the input matrix blocks to a plurality of different machines across a distributed filesystem, and sampling, by at least two of the different machines in parallel, at least two of the input matrix blocks. Finally, the computer-implemented method includes generating at least one sample matrix based on the sampling of the at least two of the input matrix blocks.

Abstract: Embodiments relate to entity resolution. One aspect includes creating a deterministic model by defining an entity to be resolved, selecting two datasets for comparison, defining matching predicates for attributes of the datasets to select a set of candidate matches, and defining a precedence rule for the candidate matches to select a subset of the candidate matches. An aspect further includes running the deterministic model on the two datasets. Running the deterministic model includes applying the matching predicates and the precedence rule to data in the datasets that correspond to the attributes. An aspect also includes applying a cardinality rule to results of the running, and outputting the matching candidates for which the cardinality rule is satisfied.

Abstract: In a method for analyzing a large data set using a statistical computing environment language operation, a processor generates code from the statistical computing environment language operation that can be understood by a software system for processing machine learning algorithms in a MapReduce environment. A processor transfers the code to the software system for processing machine learning algorithms in a MapReduce environment. A processor invokes execution of the code with the software system for processing machine learning algorithms in a MapReduce environment.

Abstract: A system evaluates a first data cleansing application and a second data cleansing application. The system includes a test data generator, an application execution module, and a results reporting module. The test data generator creates a dirty set of sample data from a clean set of data. The application execution module cleanses the dirty set of sample data. The application execution module utilizes the first data cleansing application to cleanse the dirty set of sample data and create a first cleansed output. The application execution module further utilizes the second data cleansing application to cleanse the dirty set of sample data and create a second cleansed output. The results reporting module evaluates the first and second cleansed output. The results reporting module produces an output of scores and statistics for each of the first and second data cleansing applications.

Abstract: A system views results of a data cleansing application. The system includes a results visualization module and a learning visualization interface module. The results visualization module organizes output of the data cleansing application into a predefined format. The results visualization module displays the output to a user. The learning visualization interface module facilitates interaction with the data cleansing application by the user.

Abstract: A system evaluates a data cleansing application. The system includes a collection of records cleansed by the data cleansing application, a plurality of dirtying functions for operating upon the collection to introduce errors to the collection, and a record of the errors introduced to the cleansed collection. The plurality of dirtying functions produces a collection of dirty records.

Abstract: A system builds an extended dictionary for a data cleansing application. The system includes a record collection. Each record in the collection includes a list of fields and data contained in each field. The system further includes an input dictionary defining predetermined valid values for variants of values in at least one of the fields and a set of rules derived from patterns of the field values. The system still further includes an extended dictionary including the input dictionary and the rules.

Abstract: A system learns a record similarity measurement. The system includes a set of record clusters. Each record in each cluster may have a list of fields and data contained in each field. The system may further include a predetermined threshold score for two of the records in one of the clusters to be considered similar and at least one decision tree constructed from a portion of the set of clusters. The decision tree encodes rules for determining a field similarity score of a related set of fields. The system may further include an output set of record pairs that are determined to be duplicate records. The output set of record pairs may have a record similarity score greater than or equal to the predetermined threshold score.

Abstract: A system represents data during a data cleansing application. The system includes a record collection. Each record in the collection includes a list of fields and data contained in each field. The system further includes a predetermined sequence of operations to be performed on the record collection and a plurality of bit-maps representing the record collection. The system still further includes a partitioned sequence of operations for parallel processing of the bit-maps by a plurality of separate devices.