Abstract: A cloud-based data platform is disclosed, enabling storage-agnostic data management and analytics in a unified environment. The system may integrate a data lake implemented as a hyperscaler object store, a cloud-based database management system (DBMS), elastic compute resources, and analytics engines. Data may be stored in open table formats, such as Apache Parquet, Delta Lake, and Apache Iceberg, supporting ACID transactions, schema evolution, and efficient query processing. Virtual tables may map data stored in the data lake to the DBMS, enabling in-situ query processing via SQL interfaces. The platform may support advanced features, including change data capture (CDC), time travel, and lifecycle management using a SAGA pattern for atomic operations. Security may be ensured through X.509 certificates, web tokens, and role-based access controls. Elastic compute resources, such as Apache Spark, facilitate large-scale data transformations and analytics.

Abstract: A solution for transforming and managing data in a data lake environment is disclosed. A definition of a transformation flow is received via a graphical user interface. The transformation flow specifies operations to transform source data into target data stored in a first format object storage. A virtual procedure corresponding to the transformation flow is generated and stored in an in-memory cloud database. Then the virtual procedure is executed, causing retrieval of metadata from the first format object storage and triggering a transformation job on a computing cluster. The transformation job reads the source data from the first format object storage, applies one or more transformations specified in the definition of the transformation flow, and writes results to target data in the first format object storage.