Abstract: The present embodiments relate to metadata value-based mapping during a data load in a data integration job. A computing device can receive a first data set from a source system and computer-readable instructions to load data into a target system. The device can receive a first metadata set from the target system that describe destinations. The computing device can identify a first data value of the first data set that matches a metadata value of the first metadata set. The device can receive a data integration mapping of the second data value of the first data set to a data field associated with the matching metadata value of the first metadata set. The device can load the second data value of the first data set from the source system into the target system pursuant to the mapping and the computer-readable instructions.

Abstract: The present embodiments relate to metadata value-based mapping during a data load in a data integration job. A computing device can receive a first data set from a source system and computer-readable instructions to load data into a target system. The device can receive a first metadata set from the target system that describe destinations. The computing device can identify a first data value of the first data set that matches a metadata value of the first metadata set. The device can receive a data integration mapping of the second data value of the first data set to a data field associated with the matching metadata value of the first metadata set. The device can load the second data value of the first data set from the source system into the target system pursuant to the mapping and the computer-readable instructions.

Abstract: Techniques describes herein updating pipeline flows in within data systems to maintain data integrity and consistency without manual curation. In certain embodiments, data integration system may detect and/or receive indications of a schema change within a source system of the data integration system. One or more data objects affected by the schema change may be identified, and a set of pipeline rules may be invoked for each of the affected schema changes. The pipeline rules may define a single transformation or a multi-step transformation process by which the data in the source system is provided to one or more target systems. After applying the pipeline rules to the updated source schema, the data received from the source system may be processed using the updated pipeline rules, transformed, and transmitted to the target system(s) to maintain the data integrity of the system.

Abstract: A data flow design system is presented that includes support for both static elements and dynamic elements. Thus, the data flow design system provides a design environment graphical tool to users to design data flows that leverage both the repeatability of static elements and the adaptability of dynamic elements. Static elements process data that typically do not change over time, while dynamic elements process data that do change. For instance, new data fields are added to an input data source of a data flow. The dynamic elements automatically link the new data fields to mapping fragments of the data flow. Mapping fragments process data based on configuration parameters including expression and filter rules. Users use a user interface of the design environment to view and add mapping fragments, static links, and dynamic links to the data flow.

Abstract: A data flow design system is presented that includes support for both static elements and dynamic elements. Thus, the data flow design system provides a design environment graphical tool to users to design data flows that leverage both the repeatability of static elements and the adaptability of dynamic elements. Static elements process data that typically do not change over time, while dynamic elements process data that do change. For instance, new data fields are added to an input data source of a data flow. The dynamic elements automatically link the new data fields to mapping fragments of the data flow. Mapping fragments process data based on configuration parameters including expression and filter rules. Users use a user interface of the design environment to view and add mapping fragments, static links, and dynamic links to the data flow.