Abstract: Apparatus and methods for reconciling data in a big data environment is provided. Methods may receive a first data set and a second data set for reconciliation. Methods may identify a first set of metadata associated with the first data set. Methods may identify a second set of metadata associated with the second data set. Methods may include a data reconciliation algorithm. The algorithm may compare the first set of metadata to the second set of metadata to obtain a subset of data found within the first data set and a subset of data found within the second data set that are joinable. Methods may dynamically construct one or more SQL queries to identify any discrepancies between the first data set and the second data set. Methods may execute the one or more SQL queries that identify any discrepancies between the first data set and the second data set.

Abstract: Apparatus and methods for reconciling data in a big data environment is provided. Methods may receive a first data set and a second data set for reconciliation. Methods may identify a first set of metadata associated with the first data set. Methods may identify a second set of metadata associated with the second data set. Methods may include a data reconciliation algorithm. The algorithm may compare the first set of metadata to the second set of metadata to obtain a subset of data found within the first data set and a subset of data found within the second data set that are joinable. Methods may dynamically construct one or more SQL queries to identify any discrepancies between the first data set and the second data set. Methods may execute the one or more SQL queries that identify any discrepancies between the first data set and the second data set.

Abstract: Apparatus and methods for reconciling data in a big data environment is provided. Methods may receive a first data set and a second data set for reconciliation. Methods may identify a first set of metadata associated with the first data set. Methods may identify a second set of metadata associated with the second data set. Methods may include a data reconciliation algorithm. The algorithm may compare the first set of metadata to the second set of metadata to obtain a subset of data found within the first data set and a subset of data found within the second data set that are joinable. Methods may dynamically construct one or more SQL queries to identify any discrepancies between the first data set and the second data set. Methods may execute the one or more SQL queries that identify any discrepancies between the first data set and the second data set.

Abstract: Aggregation of data and subsequent diving and/or grouping (i.e., slicing and dicing) of the aggregated data occurs by multi-stage parallel processing. Initially, multiple aggregation jobs are executed in parallel to aggregate data from a granular level to a higher level. Subsequently, query sets are executed in parallel for each of the aggregation jobs to divide and/or group the higher level data according to the queries and the segregated results are stored in a single table, which is partitioned for each aggregation job. Parallel processing of the query sets and storage in a single table is made possible by implementing a hybrid dynamic database schema, in which each table in the database includes one or more additional fields, each additional field comprising a set of dimension-dimension value pairs.

Abstract: A hybrid dynamic database schema is described. The schema is a hybrid of both horizontal and vertical database schema, in that, while a vertical schema has only one dimension—dimension value pair per table/record, the hybrid schema of the present invention provides for multiple dimension—dimension value pairs per table/record. The schema provides for additional fields, so called “DimSet” fields, with each additional field comprising one and typically multiple dimension—dimension value pairs. Thus, while a vertical database schema can only perform aggregation/segregation using one isolated dimension—dimension pair, the hybrid schema of the present invention is capable of performing multiple aggregations/segregation processes simultaneously (i.e., in parallel) using the multiple dimension fields. The dynamic nature of the database schema is realized by the ability of the schema to support various different additional fields (i.e.

Abstract: A hybrid dynamic database schema is described. The schema is a hybrid of both horizontal and vertical database schema, in that, while a vertical schema has only one dimension—dimension value pair per table/record, the hybrid schema of the present invention provides for multiple dimension—dimension value pairs per table/record. The schema provides for additional fields, so called “DimSet” fields, with each additional field comprising one and typically multiple dimension—dimension value pairs. Thus, while a vertical database schema can only perform aggregation/segregation using one isolated dimension—dimension pair, the hybrid schema of the present invention is capable of performing multiple aggregations/segregation processes simultaneously (i.e., in parallel) using the multiple dimension fields. The dynamic nature of the database schema is realized by the ability of the schema to support various different additional fields (i.e.

Abstract: Aggregation of data and subsequent diving and/or grouping (i.e., slicing and dicing) of the aggregated data occurs by multi-stage parallel processing. Initially, multiple aggregation jobs are executed in parallel to aggregate data from a granular level to a higher level. Subsequently, query sets are executed in parallel for each of the aggregation jobs to divide and/or group the higher level data according to the queries and the segregated results are stored in a single table, which is partitioned for each aggregation job. Parallel processing of the query sets and storage in a single table is made possible by implementing a hybrid dynamic database schema, in which each table in the database includes one or more additional fields, each additional field comprising a set of dimension-dimension value pairs.