Abstract: A synchronization service synchronizes a source dataset on a source database and target dataset on a target database using the Rateless Invertible Bloom Lookup Table (IBLT) algorithm. The synchronization service iteratively requests coded symbols from the source database and target database generated at these databases according to the Rateless IBLT algorithm implemented using a relational database management system. The synchronization service attempts to decode the coded symbols at each iteration. When decoding succeeds, the synchronization service communicates differences between the source dataset and the target dataset indicated in the decoded symbols to the source database and/or the target database for dataset reconciliation.

Abstract: A domain specific language, the “data pipeline language” (DPL), has been developed for programmatic creation of data pipelines for ETL/ELT with a human-readable format. The DPL syntax is abstracted from the specific technologies used for building a data pipeline, so the DPL that defines various components of a data pipeline (e.g., data sources, data warehouses, and/or cloud-based processing components) is independent of the specific vendors used to provide these components. A user that defines a data pipeline in the DPL specifies any external vendors that will be leveraged for execution of the data pipeline in a configuration file. During execution of a data pipeline defined in with the DPL, a pipeline execution platform maps data pipeline components identified from the DPL file to the respective external vendor(s) indicated in the configuration and “translates” the DPL components into their corresponding commands that are sent to the external vendor(s).

Abstract: A domain specific language, the “data pipeline language” (DPL), has been developed for programmatic creation of data pipelines for ETL/ELT with a human-readable format. The DPL syntax is abstracted from the specific technologies used for building a data pipeline, so the DPL that defines various components of a data pipeline (e.g., data sources, data warehouses, and/or cloud-based processing components) is independent of the specific vendors used to provide these components. A user that defines a data pipeline in the DPL specifies any external vendors that will be leveraged for execution of the data pipeline in a configuration file. During execution of a data pipeline defined in with the DPL, a pipeline execution platform maps data pipeline components identified from the DPL file to the respective external vendor(s) indicated in the configuration and “translates” the DPL components into their corresponding commands that are sent to the external vendor(s).