Abstract: Systems and methods include execution of a first instance of a connector registry service to receive a first registration request from a first instance of a first connector to a first data system, the first registration request comprising a first connector identifier, a first connection type and a first endpoint type, store the first connector identifier, the first connection type and the first endpoint type in a data store, read a plurality of associated connector identifiers, connection types and endpoint types from the data store, the plurality of associated connector identifiers, connection types and endpoint types including the first connector identifier, the first connection type and the first endpoint type, and transmit the plurality of associated connector identifiers, connection types and endpoint types to a first instance of a gateway service.

Abstract: A scheduling framework may include a change rate data store that contains information about replication change rates for a source system over time. A computing resources scheduling server may access change rate information from the change rate data store representing data replication from the source system to a target system. The scheduling server may automatically calculate a computing resource value (e.g., a number of replication-worker instances) based on a Gaussian ceiling function and the change rate information. The scheduling server can then dynamically adjust at least one replication computing resource allocation in accordance with the calculated computing resource value. The system may arrange for the allocated computing resource to facilitate data replication from the source system to the target system. The dynamic adjustment of the replication computing resource allocation might also be based on a start-up time, a boundary, prior change rates, a PID controller, etc.

Abstract: A cloud-based data platform is disclosed, enabling storage-agnostic data management and analytics in a unified environment. The system may integrate a data lake implemented as a hyperscaler object store, a cloud-based database management system (DBMS), elastic compute resources, and analytics engines. Data may be stored in open table formats, such as Apache Parquet, Delta Lake, and Apache Iceberg, supporting ACID transactions, schema evolution, and efficient query processing. Virtual tables may map data stored in the data lake to the DBMS, enabling in-situ query processing via SQL interfaces. The platform may support advanced features, including change data capture (CDC), time travel, and lifecycle management using a SAGA pattern for atomic operations. Security may be ensured through X.509 certificates, web tokens, and role-based access controls. Elastic compute resources, such as Apache Spark, facilitate large-scale data transformations and analytics.

Abstract: In an example embodiment, a data warehousing environment, or a similar architecture, is extended to permit data storage in either an in-memory database or a Lakehouse architecture, which leverages one or more hyperscalers for the underlying storage. More specifically, a single artifact is defined in Datasphere that stores data in either the in-memory database or the Lakehouse architecture, and does so in a storage-agnostic manner.

Abstract: A solution for transforming and managing data in a data lake environment is disclosed. A definition of a transformation flow is received via a graphical user interface. The transformation flow specifies operations to transform source data into target data stored in a first format object storage. A virtual procedure corresponding to the transformation flow is generated and stored in an in-memory cloud database. Then the virtual procedure is executed, causing retrieval of metadata from the first format object storage and triggering a transformation job on a computing cluster. The transformation job reads the source data from the first format object storage, applies one or more transformations specified in the definition of the transformation flow, and writes results to target data in the first format object storage.

Abstract: A landing zone is established in an open table format object storage. More specifically, the landing zone is implemented as a local ingestion table file. This represents a special subtype of the local data file that is used in the inbound layer when ingesting data from an external system. This local ingestion table file is further subdivided into an active data portion and an inbound buffer. When data is written from the external system, it is first placed in the inbound buffer. This data is then merged into the active data, after processing and postprocessing. The result of these postprocessing operations is then stored in the active data portion of the local ingestion table file.

Abstract: According to some embodiments, a replication tool may arrange for a copy of original records from a data source to be written into a read-only replica in an application stack. Write buffer logic in the application stack may determine that information in an original record of the read-only replica has been locally updated at the application stack by application logic. The write buffer logic may then automatically create an updated version of the original record in a write buffer of the application stack based on the local update by the application logic. Moreover, prioritization logic may automatically determine whether information in the original record or the updated record will be accessed.

Abstract: System, method, and various embodiments for a federated execution system are described herein. An embodiment operates by determining that an managed flow for a transfer of data from a source to a target is managed by an orchestrator system that communicates with each of a plurality of data engines during the transfer. Flow metadata is generated for each of the plurality of data engines. Each of the plurality of data engines is configured with a flow component configured to process the flow metadata and provide output from the respective data engine to the component data engine in accordance with the flow metadata. A component flow for the transfer of data from the source to the target is initiated.

Abstract: Systems and methods include reception of a first request from a first instance of a first connector service, the first request specifying a first connection identifier, determination that first connection details associated with the first connection identifier are stored in a local memory buffer, transmission of the first connection details from the local memory buffer to the first instance in response to the determination that the first connection details are stored in the volatile memory buffer, and reception of a notification indicating that the first connection details have changed. In response to the notification, the notification is stored in a persistent storage, the first connection details stored in the local memory buffer are invalidated, and an instruction is transmitted to the first instance of the first connector service to invalidate the first connection details stored at the first instance.

Abstract: Systems and methods include reception of a first call to a first interface at a first connector associated with a first data system, the first call requesting to write first data to the first data system and conforming to a data model, conversion, at the first connector, of the first call to a first data model of the first data system, transmission of the converted first call from the first connector to the first data system, reception of a second call to the first interface at a second connector associated with a second data system, the second call requesting to write second data to the second data system and conforming to the data model, conversion, at the second connector, of the second call to a second data model of the second data system, and transmission of the converted second call from the second connector to the second data system.

Abstract: Systems and methods include execution of a first instance of a connector registry service to receive a first registration request from a first instance of a first connector to a first data system, the first registration request comprising a first connector identifier, a first connection type and a first endpoint type, store the first connector identifier, the first connection type and the first endpoint type in a data store, read a plurality of associated connector identifiers, connection types and endpoint types from the data store, the plurality of associated connector identifiers, connection types and endpoint types including the first connector identifier, the first connection type and the first endpoint type, and transmit the plurality of associated connector identifiers, connection types and endpoint types to a first instance of a gateway service.

Abstract: According to some embodiments, a replication tool may arrange for a copy of original records from a data source to be written into a read-only replica in an application stack. Write buffer logic in the application stack may determine that information in an original record of the read-only replica has been locally updated at the application stack by application logic. The write buffer logic may then automatically create an updated version of the original record in a write buffer of the application stack based on the local update by the application logic. Moreover, prioritization logic may automatically determine whether information in the original record or the updated record will be accessed.

Abstract: Disclosed herein are system, method, and computer program product embodiments for accessing a semantic change data capture (CDC). An embodiment operates by receiving a request for changes made to a source system from a consumer at a target system. The embodiment determines a high watermark stored in a consumer registration table based on the consumer. The embodiment further determines a set of open transactions on an enhanced data table. The embodiment then determines a set of changes made to the source system based on the high watermark and the set of open transactions. The embodiment transmits the set of changes to the consumer at the target system. The embodiment then receives a confirmation from the consumer at the target system that the set of changes were committed to the target system. The embodiment then updates the high watermark in the consumer registration table based on the confirmation.

Abstract: Example methods and systems are directed to automatically toggling between replication and federation for access for data received from remote sources. The system may initially use a fully federated strategy, so that results for any query can be generated. Based on one or more of predefined criteria, analysis of the queries being performed, network performance data, and other factors, data sets that would benefit from replication are identified. The identified data sets are accessed from the remote source and replicated in local storage. Future queries for replicated data are satisfied by accessing the replicated data. Other queries continue to be served using federation. After replication has begun, analysis of the benefits of replication may continue. When replication is no longer a net benefit, the local data may be deleted and further access of the data provided through federation.