Abstract: Data sources provide access to data. The data stored by the data source may be transformed before use by an application. Different data sources support different transformations. A data agent sidecar for the application accepts work orders from the application and submits work orders to data sources. A work order identifies a data source from which data is requested. The work order optionally includes one or more transformations to be applied to the data from the data source. The data agent sidecar determines, for the data source from which data is requested, which transformations can be performed by the data source and which transformations are not supported by the data source. The data transformations that can be performed by the data source are included in the work order to the data source. The remaining data transformations are performed by the data agent sidecar.

Abstract: Disclosed herein are system, method, and computer program product embodiments for automatically generating events. In some embodiments, a server receives a request to subscribe to an event type for an application. The server identifies an entity in a data model corresponding to the event type. The entity is associated with application data of the application. Furthermore, the server subscribes to receive alerts associated with any change to the entity in the data model. The server receives an alert indicating a change to the entity in the data model. The alert includes the change to the entity. Moreover, the server generates an event corresponding to the event type based on the change to the entity and transmits the event to an event infrastructure.

Abstract: Data sources provide access to data. The data stored by the data source may be transformed before use by an application. Different data sources support different transformations. A data agent sidecar for the application accepts work orders from the application and submits work orders to data sources. A work order identifies a data source from which data is requested. The work order optionally includes one or more transformations to be applied to the data from the data source. The data agent sidecar determines, for the data source from which data is requested, which transformations can be performed by the data source and which transformations are not supported by the data source. The data transformations that can be performed by the data source are included in the work order to the data source. The remaining data transformations are performed by the data agent sidecar.

Abstract: Data sources provide access to data. The data stored by the data source may be transformed before use by an application. Different data sources support different transformations. A data agent sidecar for the application accepts work orders from the application and submits work orders to data sources. A work order identifies a data source from which data is requested. The work order optionally includes one or more transformations to be applied to the data from the data source. The data agent sidecar determines, for the data source from which data is requested, which transformations can be performed by the data source and which transformations are not supported by the data source. The data transformations that can be performed by the data source are included in the work order to the data source. The remaining data transformations are performed by the data agent sidecar.

Abstract: Disclosed herein are various embodiments for a data replication system. An embodiment operates by determining that a plurality of records of a source table are copied from a source system to a target table of a target system. It is determined that a change table, of the source system, associated with the source table is populated with a plurality of changes occurring to at least a subset of records of the plurality of records while the plurality of records are being copied from the source table of the source system to the target table of the target system. It is determined that the plurality of changes are copied to a shadow table of the target system. It is determined that the target table includes the plurality of records from the source table. The plurality of changes of the shadow table are merged with the target table.

Abstract: Disclosed herein are system, method, and computer program product embodiments for automatically generating events. In some embodiments, a server receives a request to subscribe to an event type for an application. The server identifies an entity in a data model corresponding to the event type. The entity is associated with application data of the application. Furthermore, the server subscribes to receive alerts associated with any change to the entity in the data model. The server receives an alert indicating a change to the entity in the data model. The alert includes the change to the entity. Moreover, the server generates an event corresponding to the event type based on the change to the entity and transmits the event to an event infrastructure.

Abstract: Disclosed herein are various embodiments for a data replication system. An embodiment operates by determining that a plurality of records of a source table are copied from a source system to a target table of a target system. It is determined that a change table, of the source system, associated with the source table is populated with a plurality of changes occurring to at least a subset of records of the plurality of records while the plurality of records are being copied from the source table of the source system to the target table of the target system. It is determined that the plurality of changes are copied to a shadow table of the target system. It is determined that the target table includes the plurality of records from the source table. The plurality of changes of the shadow table are merged with the target table.

Abstract: Disclosed herein are system, method, and computer program product embodiments for implementing static runtime with dynamic work orders. An embodiment operates by generating, by a controller, a runtime instance based on a runtime template and assigning, by the controller, a work order to the runtime instance. The work order is generated based on an Extract-Transform-Load (E-T-L) process. The embodiment further operates by executing, by the controller, the work order on the runtime instance and updating, by the controller, the work order in a storage.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to accelerate data extraction from a source system to a target system. An embodiment operates receiving a data change for a data extraction from a producer job at the source system. The embodiment stores the data change in a staging area of a persistent storage together with a respective sequence identifier. The embodiment receives a request for a next package of data changes in the staging area from a consumer job at the target system. The embodiment generates the next package from the staging area. The embodiment transmits the next package to the consumer job. The embodiment receives a commit notification for the next package from the consumer job. The embodiment then removes the data changes in the next package from the staging area in response to receiving the commit notification.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to guarantee the ability to rollback and reconstruct a package while minimizing locking duration and enabling multiple packages to be processed in a data pipeline simultaneously. An embodiment operates by receiving a package from a source system for processing through a data pipeline. The embodiment stores the package in a persistent storage together with a respective package status. The embodiment transmits the package to the data pipeline in response to the storing. The embodiment receives a commit notification for the package from a target system in response to the transmitting. The embodiment then removes the package from the persistent storage in response to receiving the commit notification for the package.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to accelerate data extraction from a source system to a target system. An embodiment operates receiving a data change for a data extraction from a producer job at the source system. The embodiment stores the data change in a staging area of a persistent storage together with a respective sequence identifier. The embodiment receives a request for a next package of data changes in the staging area from a consumer job at the target system. The embodiment generates the next package from the staging area. The embodiment transmits the next package to the consumer job. The embodiment receives a commit notification for the next package from the consumer job. The embodiment then removes the data changes in the next package from the staging area in response to receiving the commit notification.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to guarantee the ability to rollback and reconstruct a package while minimizing locking duration and enabling multiple packages to be processed in a data pipeline simultaneously. An embodiment operates by receiving a package from a source system for processing through a data pipeline. The embodiment stores the package in a persistent storage together with a respective package status. The embodiment transmits the package to the data pipeline in response to the storing. The embodiment receives a commit notification for the package from a target system in response to the transmitting. The embodiment then removes the package from the persistent storage in response to receiving the commit notification for the package.

Abstract: Provided is a framework including a generic communication layer that facilitates a separation between communication connections for different communication technologies and an application interfacing with the communication connections. The framework generally includes a remote interface layer including communication interfaces that generically communicate with a client, an application layer that includes applications represented by a set of services that use the communication interfaces to communicate with a client, and a service layer that includes one or more services. The framework includes mechanisms to map parameters of a specific communication technology to a standardized parameter structure that can be used by different service-based applications.

Abstract: Provided is a framework including a generic communication layer that facilitates a separation between communication connections for different communication technologies and an application interfacing with the communication connections. The framework generally includes a remote interface layer including communication interfaces that generically communicate with a client, an application layer that includes applications represented by a set of services that use the communication interfaces to communicate with a client, and a service layer that includes one or more services. The framework includes mechanisms to map parameters of a specific communication technology to a standardized parameter structure that can be used by different service-based applications.