Abstract: Disclosed herein are system, method, and computer program product embodiments for accessing a semantic change data capture (CDC). An embodiment operates by receiving a request for changes made to a source system from a consumer at a target system. The embodiment determines a high watermark stored in a consumer registration table based on the consumer. The embodiment further determines a set of open transactions on an enhanced data table. The embodiment then determines a set of changes made to the source system based on the high watermark and the set of open transactions. The embodiment transmits the set of changes to the consumer at the target system. The embodiment then receives a confirmation from the consumer at the target system that the set of changes were committed to the target system. The embodiment then updates the high watermark in the consumer registration table based on the confirmation.

Abstract: Example methods and systems are directed to automatically toggling between replication and federation for access for data received from remote sources. The system may initially use a fully federated strategy, so that results for any query can be generated. Based on one or more of predefined criteria, analysis of the queries being performed, network performance data, and other factors, data sets that would benefit from replication are identified. The identified data sets are accessed from the remote source and replicated in local storage. Future queries for replicated data are satisfied by accessing the replicated data. Other queries continue to be served using federation. After replication has begun, analysis of the benefits of replication may continue. When replication is no longer a net benefit, the local data may be deleted and further access of the data provided through federation.

Abstract: Provided are systems and methods for transforming data from a format of a source system into a format of a target system based on a dedicated virtual machine. In one example, a method may include storing programming code which comprises logic for transforming data from a source system to a target format of a target system, compiling the programming code into bytecode, generating a work order which includes a payload of data from the source system and the bytecode, reading the work order via a virtual machine and executing the bytecode on the payload of data via the virtual machine, wherein the virtual machine transforms the payload of data from the source format to the target format based on the bytecode, and loading the transformed payload of data into the target system.

Abstract: Data sources provide access to data. The data stored by the data source may be transformed before use by an application. Different data sources support different transformations. A data agent sidecar for the application accepts work orders from the application and submits work orders to data sources. A work order identifies a data source from which data is requested. The work order optionally includes one or more transformations to be applied to the data from the data source. The data agent sidecar determines, for the data source from which data is requested, which transformations can be performed by the data source and which transformations are not supported by the data source. The data transformations that can be performed by the data source are included in the work order to the data source. The remaining data transformations are performed by the data agent sidecar.

Abstract: Disclosed herein are system, method, and computer program product embodiments for automatically generating events. In some embodiments, a server receives a request to subscribe to an event type for an application. The server identifies an entity in a data model corresponding to the event type. The entity is associated with application data of the application. Furthermore, the server subscribes to receive alerts associated with any change to the entity in the data model. The server receives an alert indicating a change to the entity in the data model. The alert includes the change to the entity. Moreover, the server generates an event corresponding to the event type based on the change to the entity and transmits the event to an event infrastructure.

Abstract: Data sources provide access to data. The data stored by the data source may be transformed before use by an application. Different data sources support different transformations. A data agent sidecar for the application accepts work orders from the application and submits work orders to data sources. A work order identifies a data source from which data is requested. The work order optionally includes one or more transformations to be applied to the data from the data source. The data agent sidecar determines, for the data source from which data is requested, which transformations can be performed by the data source and which transformations are not supported by the data source. The data transformations that can be performed by the data source are included in the work order to the data source. The remaining data transformations are performed by the data agent sidecar.

Abstract: Data sources provide access to data. The data stored by the data source may be transformed before use by an application. Different data sources support different transformations. A data agent sidecar for the application accepts work orders from the application and submits work orders to data sources. A work order identifies a data source from which data is requested. The work order optionally includes one or more transformations to be applied to the data from the data source. The data agent sidecar determines, for the data source from which data is requested, which transformations can be performed by the data source and which transformations are not supported by the data source. The data transformations that can be performed by the data source are included in the work order to the data source. The remaining data transformations are performed by the data agent sidecar.

Abstract: Disclosed herein are various embodiments for a data replication system. An embodiment operates by determining that a plurality of records of a source table are copied from a source system to a target table of a target system. It is determined that a change table, of the source system, associated with the source table is populated with a plurality of changes occurring to at least a subset of records of the plurality of records while the plurality of records are being copied from the source table of the source system to the target table of the target system. It is determined that the plurality of changes are copied to a shadow table of the target system. It is determined that the target table includes the plurality of records from the source table. The plurality of changes of the shadow table are merged with the target table.

Abstract: Disclosed herein are system, method, and computer program product embodiments for automatically generating events. In some embodiments, a server receives a request to subscribe to an event type for an application. The server identifies an entity in a data model corresponding to the event type. The entity is associated with application data of the application. Furthermore, the server subscribes to receive alerts associated with any change to the entity in the data model. The server receives an alert indicating a change to the entity in the data model. The alert includes the change to the entity. Moreover, the server generates an event corresponding to the event type based on the change to the entity and transmits the event to an event infrastructure.

Abstract: Disclosed herein are various embodiments for a data replication system. An embodiment operates by determining that a plurality of records of a source table are copied from a source system to a target table of a target system. It is determined that a change table, of the source system, associated with the source table is populated with a plurality of changes occurring to at least a subset of records of the plurality of records while the plurality of records are being copied from the source table of the source system to the target table of the target system. It is determined that the plurality of changes are copied to a shadow table of the target system. It is determined that the target table includes the plurality of records from the source table. The plurality of changes of the shadow table are merged with the target table.

Abstract: Disclosed herein are system, method, and computer program product embodiments for implementing static runtime with dynamic work orders. An embodiment operates by generating, by a controller, a runtime instance based on a runtime template and assigning, by the controller, a work order to the runtime instance. The work order is generated based on an Extract-Transform-Load (E-T-L) process. The embodiment further operates by executing, by the controller, the work order on the runtime instance and updating, by the controller, the work order in a storage.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to accelerate data extraction from a source system to a target system. An embodiment operates receiving a data change for a data extraction from a producer job at the source system. The embodiment stores the data change in a staging area of a persistent storage together with a respective sequence identifier. The embodiment receives a request for a next package of data changes in the staging area from a consumer job at the target system. The embodiment generates the next package from the staging area. The embodiment transmits the next package to the consumer job. The embodiment receives a commit notification for the next package from the consumer job. The embodiment then removes the data changes in the next package from the staging area in response to receiving the commit notification.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to guarantee the ability to rollback and reconstruct a package while minimizing locking duration and enabling multiple packages to be processed in a data pipeline simultaneously. An embodiment operates by receiving a package from a source system for processing through a data pipeline. The embodiment stores the package in a persistent storage together with a respective package status. The embodiment transmits the package to the data pipeline in response to the storing. The embodiment receives a commit notification for the package from a target system in response to the transmitting. The embodiment then removes the package from the persistent storage in response to receiving the commit notification for the package.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to accelerate data extraction from a source system to a target system. An embodiment operates receiving a data change for a data extraction from a producer job at the source system. The embodiment stores the data change in a staging area of a persistent storage together with a respective sequence identifier. The embodiment receives a request for a next package of data changes in the staging area from a consumer job at the target system. The embodiment generates the next package from the staging area. The embodiment transmits the next package to the consumer job. The embodiment receives a commit notification for the next package from the consumer job. The embodiment then removes the data changes in the next package from the staging area in response to receiving the commit notification.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing look-ahead-staging (LAS) to guarantee the ability to rollback and reconstruct a package while minimizing locking duration and enabling multiple packages to be processed in a data pipeline simultaneously. An embodiment operates by receiving a package from a source system for processing through a data pipeline. The embodiment stores the package in a persistent storage together with a respective package status. The embodiment transmits the package to the data pipeline in response to the storing. The embodiment receives a commit notification for the package from a target system in response to the transmitting. The embodiment then removes the package from the persistent storage in response to receiving the commit notification for the package.

Abstract: Provided is a framework including a generic communication layer that facilitates a separation between communication connections for different communication technologies and an application interfacing with the communication connections. The framework generally includes a remote interface layer including communication interfaces that generically communicate with a client, an application layer that includes applications represented by a set of services that use the communication interfaces to communicate with a client, and a service layer that includes one or more services. The framework includes mechanisms to map parameters of a specific communication technology to a standardized parameter structure that can be used by different service-based applications.

Abstract: Provided is a framework including a generic communication layer that facilitates a separation between communication connections for different communication technologies and an application interfacing with the communication connections. The framework generally includes a remote interface layer including communication interfaces that generically communicate with a client, an application layer that includes applications represented by a set of services that use the communication interfaces to communicate with a client, and a service layer that includes one or more services. The framework includes mechanisms to map parameters of a specific communication technology to a standardized parameter structure that can be used by different service-based applications.