Abstract: The present disclosure generally relates to a cost-based optimizer for efficiently processing data through the use of multiple different data processing platforms. The cost-based optimizer may receive an input plan for processing data that includes a number of base operators. The cost-based optimizer may then determine execution operators for each base operator, where each execution operator corresponds to a different data processing platform. From the execution operators, the cost-based optimizer may determine possible subplans for executing the input plan on one or more data processing platforms. The cost-based optimizer may determine the cost of executing each possible subplan and choose the subplan with the lowest cost.

Abstract: A computer implemented method for integrating data into a target database may include: providing a plurality of source databases which each may include a relational schema and data for integration into the target database; generating at least one complexity model based on the relational schema and data of each source database, each complexity model indicating at least one inconsistency between two or more of the data sources which may be require to be resolved to integrate the data from the data sources into the target database; and generating an effort model that may include an effort value for each inconsistency indicated by each complexity model, each effort value indicating at least one of a time period and a financial cost to resolve the inconsistency to integrate data from the data sources into the target database.

Abstract: A dataset profiling tool configured to identify unique and non-unique column combinations in a dataset which includes a plurality of tuples, the tool including: an inserts handler module configured to: receive one or more new tuples for insertion into the dataset, receive one or more minimal uniques and one or more maximal non-uniques for the dataset, identify and group, for each minimal unique, any tuples of the dataset and any of the one or more new tuples which contain duplicate values in the column combinations of the minimal unique, to form grouped tuples which are grouped according to the minimal unique to which the tuples relate, validate the grouped tuples to identify supersets of the minimal uniques for which duplicate values were identified, to generate a new set of one or more minimal uniques and one or more maximal non-uniques, and output the new set of one or more updated minimal uniques and one or more maximal non-uniques.

Abstract: A computer implemented method for integrating data into a target database may include: providing a plurality of source databases which each may include a relational schema and data for integration into the target database; generating at least one complexity model based on the relational schema and data of each source database, each complexity model indicating at least one inconsistency between two or more of the data sources which may be require to be resolved to integrate the data from the data sources into the target database; and generating an effort model that may include an effort value for each inconsistency indicated by each complexity model, each effort value indicating at least one of a time period and a financial cost to resolve the inconsistency to integrate data from the data sources into the target database.

Abstract: A dataset profiling tool configured to identify unique and non-unique column combinations in a dataset which comprises a plurality of tuples, the tool including: an inserts handler module configured to: receive one or more new tuples for insertion into the dataset, receive one or more minimal uniques and one or more maximal non-uniques for the dataset, identify and group, for each minimal unique, any tuples of the dataset and any of the one or more new tuples which contain duplicate values in the column combinations of the minimal unique, to form grouped tuples which are grouped according to the minimal unique to which the tuples relate, validate the grouped tuples to identify supersets of the minimal uniques for which duplicate values were identified, to generate a new set of one or more minimal uniques and one or more maximal non-uniques, and output the new set of one or more updated minimal uniques and one or more maximal non-uniques.

Abstract: A method of processing data stored in a database which comprises a plurality of rows and columns, the method comprising identifying a plurality of sets of column combinations, each set of column combinations comprising an identifier of at least one column allocating each set of column combinations to one of a plurality of nodes mapping the nodes to a lattice structure in which the nodes are connected in a superset or subset relationship according to the set of column combinations of each node selecting a current node processing the data in the set of columns of the current node to detect if the column combination is unique or non-unique traversing the lattice to a next node which is connected to the current node processing the data in the set of columns of the next node to detect if the column combination of the next node is unique or non-unique; and storing a record of whether each processed set of column combinations is unique or non-unique.