Abstract: Embodiments maintain a data pool that includes heterogeneous data sets, and receiving a first data batch of a data set from a data source into the data pool. Embodiments determine a current state of the data set based on a data set state diagram including a plurality of data set states, and identify a condition of the first data batch. Embodiments further set a data batch state for the first data batch, based on a data batch state diagram, and update the data batch state of a prior data batch received before the first data batch, based on the condition of the first data batch. Embodiments additionally transition the data set state diagram, based on the condition of the first data batch, to an updated data set state. Embodiments maintain a data state repository storing the data set state for each of the plurality of heterogeneous data sets.

Abstract: Embodiments of the system in include a memory that stores a metamodel including a plurality of predefined characteristics for data sets. A data repository stores a plurality of heterogeneous data sets, each of the plurality of data sets comprising a plurality of data batches received over time. An interface receives a new data set for storage into the data repository, and data health reasoner to retrieve the stored metamodel from the memory, the stored metamodel including a plurality of predefined characteristics. The data health reasoner determines measured values of a subset of the plurality of predefined characteristics identified based on the stored metamodel, and determines a set of data health metrics for the data set based on the measured values of the subset of the set of the predefined characteristics. The data health reasoner formulates a plurality of data validation assertions for the data set and apply the plurality of data validation assertions to each instance of the data set.

Abstract: Embodiments maintain a data pool that includes heterogeneous data sets, and receiving a first data batch of a data set from a data source into the data pool. Embodiments determine a current state of the data set based on a data set state diagram including a plurality of data set states, and identify a condition of the first data batch. Embodiments further set a data batch state for the first data batch, based on a data batch state diagram, and update the data batch state of a prior data batch received before the first data batch, based on the condition of the first data batch. Embodiments additionally transition the data set state diagram, based on the condition of the first data batch, to an updated data set state. Embodiments maintain a data state repository storing the data set state for each of the plurality of heterogeneous data sets.

Abstract: Embodiments of the system in include a memory that stores a metamodel including a plurality of predefined characteristics for data sets. A data repository stores a plurality of heterogeneous data sets, each of the plurality of data sets comprising a plurality of data batches received over time. An interface receives a new data set for storage into the data repository, and data health reasoner to retrieve the stored metamodel from the memory, the stored metamodel including a plurality of predefined characteristics. The data health reasoner determines measured values of a subset of the plurality of predefined characteristics identified based on the stored metamodel, and determines a set of data health metrics for the data set based on the measured values of the subset of the set of the predefined characteristics. The data health reasoner formulates a plurality of data validation assertions for the data set and apply the plurality of data validation assertions to each instance of the data set.

Abstract: Transaction consistency query support is available for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a transaction completion indicator of one of commit and abort. In response to receiving a query about a transaction of the transactions, a set of records are retrieved from the external data store for the transaction. From the set of records, records whose sequence identifier values are larger than a maximum transaction commit sequence identifier are removed. From the set of records, remaining records having transaction consistency are returned.

Abstract: Transaction consistency query support is available for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a transaction completion indicator of one of commit and abort. In response to receiving a query about a transaction of the transactions, a set of records are retrieved from the external data store for the transaction. From the set of records, records whose sequence identifier values are larger than a maximum transaction commit sequence identifier are removed. From the set of records, remaining records having transaction consistency are returned.

Abstract: Provided are techniques for replicating relational transactional log data to a big data platform. Change records contained in change data tables are fetched. A relational change history with transaction snapshot consistency is rebuilt to generate consistent change records by joining the change data tables and a unit of work table based on a commit sequence identifier. The consistent change records are stored on the big data platform, and queries are answered on the big data platform using the consistent change records.

Abstract: Provided are techniques for replicating relational transactional log data to a big data platform. Change records contained in change data tables are fetched. A relational change history with transaction snapshot consistency is rebuilt to generate consistent change records by joining the change data tables and a unit of work table based on a commit sequence identifier. The consistent change records are stored on the big data platform, and queries are answered on the big data platform using the consistent change records.

Abstract: Provided are techniques for transaction consistency query support for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a transaction completion indicator of one of commit and abort. In response to receiving a query about a transaction of the transactions, a set of records are retrieved from the external data store for the transaction. From the set of records, records whose sequence identifier values are larger than a maximum transaction commit sequence identifier are removed. From the set of records, remaining records having transaction consistency are returned.

Abstract: Provided are techniques for replicating relational transactional log data to a big data platform. Change records contained in change data tables are fetched. A relational change history with transaction snapshot consistency is rebuilt to generate consistent change records by joining the change data tables and a unit of work table based on a commit sequence identifier. The consistent change records are stored on the big data platform, and queries are answered on the big data platform using the consistent change records.

Abstract: A system, method and computer program product for enabling light weight table comparison with high-accuracy (high confidence) of tables where one is a copy of the other, which copy may be maintained synchronized by replication. The method performs database comparison using a sample-based, statistics-based, or materialized query tables-based approaches. The method first identifies a block comprising a sub-set of rows of data of a source database table and a corresponding block from a target database table, and obtains a statistical value associated with each block. Then the statistical values for the corresponding source and target block are compared and a consistency evaluation of source and target database is determined based on comparing results. Further methods enable a determination of the data as being persistent or not in manner that accounts for real-time data modifications to underlying source and target database tables while identified blocks are being compared.

Abstract: Provided are techniques for replicating relational transactional log data to a big data platform. Change records contained in change data tables are fetched. A relational change history with transaction snapshot consistency is rebuilt to generate consistent change records by joining the change data tables and a unit of work table based on a commit sequence identifier. The consistent change records are stored on the big data platform, and queries are answered on the big data platform using the consistent change records.

Abstract: Provided are techniques for transaction consistency query support for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a transaction completion indicator of one of commit and abort. In response to receiving a query about a transaction of the transactions, a set of records are retrieved from the external data store for the transaction. From the set of records, records whose sequence identifier values are larger than a maximum transaction commit sequence identifier are removed. From the set of records, remaining records having transaction consistency are returned.

Abstract: According to one embodiment of the present invention, a system uses parallel transaction messages for database replication. The system receives transaction messages from a source system via a plurality of parallel send queues in a receive queue. Each transaction message includes a message identifier indicating a commit order for that transaction. The system reads transaction messages in the receive queue in order in which they were committed against a source database based on the message identifier, and applies changes described by the transaction messages to a target database. Two or more transaction messages in the receive queue are applied to the target database in parallel. The system deletes transaction messages from the receive queue asynchronously to applying changes described by those transaction messages to the target database. Embodiments of the present invention further include a method and computer program product for parallel transaction messages in substantially the same manners described above.

Abstract: A system, method and computer program product for enabling light weight table comparison with high-accuracy (high confidence) of tables where one is a copy of the other, which copy may be maintained synchronized by replication. The method performs database comparison using a sample-based, statistics-based, or materialized query tables-based approaches. The method first identifies a block comprising a sub-set of rows of data of a source database table and a corresponding block from a target database table, and obtains a statistical value associated with each block. Then the statistical values for the corresponding source and target block are compared and a consistency evaluation of source and target database is determined based on comparing results. Further methods enable a determination of the data as being persistent or not in manner that accounts for real-time data modifications to underlying source and target database tables while identified blocks are being compared.

Abstract: According to one embodiment of the present invention, a system uses parallel transaction messages for database replication. The system receives transaction messages from a source system via a plurality of parallel send queues in a receive queue. Each transaction message includes a message identifier indicating a commit order for that transaction. The system reads transaction messages in the receive queue in order in which they were committed against a source database based on the message identifier, and applies changes described by the transaction messages to a target database. Two or more transaction messages in the receive queue are applied to the target database in parallel. The system deletes transaction messages from the receive queue asynchronously to applying changes described by those transaction messages to the target database. Embodiments of the present invention further include a method and computer program product for parallel transaction messages in substantially the same manners described above.

Abstract: A computer implemented method, data processing system, and computer usable code are provided for automatically generating object setup scripts. A form is presented to a user in a graphical user interface. The user provides input in a set of fields within the form, which is received and validated. A set of object setup scripts are then generated based on the user input to the form. To generate the set of object setup scripts based on the user input an analysis is performed on the set of fields as a set of objects, and at least one variable is set to hold the value of the set of objects. String manipulations are used to obtain the values in the set of fields, and the values are entered into the set of object setup scripts.

Abstract: A computer implemented method, data processing system, and computer usable code are provided for automatically generating object setup scripts. A form is presented to a user in a graphical user interface. The user provides input in a set of fields within the form, which is received and validated. A set of object setup scripts are then generated based on the user input to the form. To generate the set of object setup scripts based on the user input an analysis is performed on the set of fields as a set of objects, and at least one variable is set to hold the value of the set of objects. String manipulations are used to obtain the values in the set of fields, and the values are entered into the set of object setup scripts.

Abstract: A method and system for member initialization to and deactivation from an asynchronous data replication group in a database system is disclosed. The method and system allows new members to be added to the replication group or existing members to be removed from the replication group, without requiring the halting of the asynchronous replication of data. The performance advantages of asynchronous replication are still realized during member initialization or deactivation.

Abstract: A method and system for member initialization to and deactivation from an asynchronous data replication group in a database system is disclosed. The method and system allows new members to be added to the replication group or existing members to be removed from the replication group, without requiring the halting of the asynchronous replication of data. The performance advantages of asynchronous replication are still realized during member initialization or deactivation.