Abstract: A data processing system defines a unit test for a dataflow graph comprising a plurality of executable components. The system receives an indication of a portion of a dataflow graph for testing. The system receives a parameter set indicating expected output data to be generated by execution of a component of the graph. The system receives input data for the component. The input data are indicated by the parameter set and configured to test a functionality of the component. The system defines a unit test of the component based on the parameter set. The unit test provides the input data to inputs of the dataflow graph, causes processing of the input data by the component of the dataflow graph to generate output data, and generates results data indicating a correspondence between the output data and the expected output data indicated by the parameter.

Abstract: In an aspect, a method for migrating data records to a federated database system includes obtaining data records from a data source in a first federated database system; generating a data snapshot file based on the obtained data records and data indicative of a characteristic associated with the obtained data records; generating a hash of the data snapshot file to prevent modification of the data snapshot file; storing the data snapshot file and the generated hash in a data storage; migrating the obtained data records from the data snapshot file to a data target in a second federated database system, the migrating including: retrieving the data records from the data snapshot file stored in the data storage; providing the retrieved data records to the data target according to a mapping between a characteristic of the data source and a characteristic of the data target.

Abstract: In an aspect, a method for migrating data records to a federated database system includes obtaining data records from a data source in a first federated database system; generating a data snapshot file based on the obtained data records and data indicative of a characteristic associated with the obtained data records; generating a hash of the data snapshot file to prevent modification of the data snapshot file; storing the data snapshot file and the generated hash in a data storage; migrating the obtained data records from the data snapshot file to a data target in a second federated database system, the migrating including: retrieving the data records from the data snapshot file stored in the data storage; providing the retrieved data records to the data target according to a mapping between a characteristic of the data source and a characteristic of the data target.

Abstract: In an aspect, a method for migrating data records to a federated database system includes obtaining data records from a data source in a first federated database system; generating a data snapshot file based on the obtained data records and data indicative of a characteristic associated with the obtained data records; generating a hash of the data snapshot file to prevent modification of the data snapshot file; storing the data snapshot file and the generated hash in a data storage; migrating the obtained data records from the data snapshot file to a data target in a second federated database system, the migrating including: retrieving the data records from the data snapshot file stored in the data storage; providing the retrieved data records to the data target according to a mapping between a characteristic of the data source and a characteristic of the data target.

Abstract: A data processing system defines a unit test for a dataflow graph comprising a plurality of executable components. The system receives an indication of a portion of a dataflow graph for testing. The system receives a parameter set indicating expected output data to be generated by execution of a component of the graph. The system receives input data for the component. The input data are indicated by the parameter set and configured to test a functionality of the component. The system defines a unit test of the component based on the parameter set. The unit test provides the input data to inputs of the dataflow graph, causes processing of the input data by the component of the dataflow graph to generate output data, and generates results data indicating a correspondence between the output data and the expected output data indicated by the parameter.