Abstract: A cloud-based data platform is disclosed, enabling storage-agnostic data management and analytics in a unified environment. The system may integrate a data lake implemented as a hyperscaler object store, a cloud-based database management system (DBMS), elastic compute resources, and analytics engines. Data may be stored in open table formats, such as Apache Parquet, Delta Lake, and Apache Iceberg, supporting ACID transactions, schema evolution, and efficient query processing. Virtual tables may map data stored in the data lake to the DBMS, enabling in-situ query processing via SQL interfaces. The platform may support advanced features, including change data capture (CDC), time travel, and lifecycle management using a SAGA pattern for atomic operations. Security may be ensured through X.509 certificates, web tokens, and role-based access controls. Elastic compute resources, such as Apache Spark, facilitate large-scale data transformations and analytics.

Abstract: A system and method to receive a replicated representation of a data entity and a replicated representation of application specific permissions associated with the data entity, the application specific permissions being defined by a source application sourcing the data entity; store the replicated representations of the data entity and the application specific permissions in a dedicated storage space for the source application within a consolidated cloud storage; automatically generate a secured data entity based on an integration of the replicated representation of the application specific permissions with the replicated representation of the data entity; and store the generated secured data entity in the dedicated storage space for the source application.

Abstract: A system and method to receive a replicated representation of a data entity and a replicated representation of application specific permissions associated with the data entity, the application specific permissions being defined by a source application sourcing the data entity; store the replicated representations of the data entity and the application specific permissions in a dedicated storage space for the source application within a consolidated cloud storage; automatically generate a secured data entity based on an integration of the replicated representation of the application specific permissions with the replicated representation of the data entity; and store the generated secured data entity in the dedicated storage space for the source application.

Abstract: In one respect, there is provided a distributed database system that includes a global scheduler node and a first local scheduler node. The global scheduler node can be configured to generate an execution plan for a query requiring data stored at and/or managed by a first and a second worker node assigned to the first local scheduler node. The execution plan can include a first fragment having operations to be performed by the first and second worker node. The first local scheduler node can be configured to coordinate the execution of a portion of the first fragment of the execution plan by distributing, to the first and second worker node, operations included in the first fragment of the execution plan, and routing, between the first and second worker node, messages associated with executing the first fragment of the execution plan. Related methods and computer program products are also provided.

Abstract: The present disclosure involves systems, software, and computer implemented methods for preventing reader starvation in a distributed system. One example method includes receiving, at a node that manages a data stream, a write-intent request from a writer for a first partition, the data stream associated with a graph that includes levels of partitions including a current level; identifying the first partition for the writer; providing partition information for the first partition to enable the writer to write to the first partition; determining that the writer has previously written to a second partition associated with the current level; in response to determining that the writer has previously written to the second partition: creating a new current level; associating the first partition with the new current level; identifying partitions associated with the current level; and providing a close-level instruction to each local node that includes a partition associated with the current level.

Abstract: In one respect, there is provided a distributed database system that includes a first local scheduler node and a second local scheduler node. The first local scheduler node can be configured to coordinate a performance of a first task and a second task by a first worker node associated with the first local scheduler node. The second local scheduler node can be configured to respond to a defect at the first local scheduler node by at least: collecting, from the first worker node, a log indicating one or more tasks completed by the first worker node prior to the defect at the first local scheduler node; determining, based at least on the log, that the first worker node has completed the first task but not the second task; and coordinating a performance of the second task by the first worker node. Related methods and articles of manufacture are also provided.

Abstract: In one respect, there is provided a distributed database system. The distributed database system can include a plurality of nodes and a plurality of optimizers. A first optimizer of the plurality of optimizers can be configured to: receive, from a client, a command; generate, based at least in part on global metadata cached by the first optimizer, an execution plan for fulfilling the command that includes one or more tasks to be performed by at least a first node of the plurality of nodes; determine that at least one task of the one or more tasks are not successfully performed by the first node; and in response to determining that at least one task of the one or more tasks are not successfully performed by the first node, updating the cached global metadata at the first optimizer.

Abstract: A system includes reception of a request to modify the data of a database, the request including first data, execution of processing to fulfill the request, determination, during execution of the processing, that a validation exit is associated with a current state of the processing, storage of the first data in a local temporary table in response to the determination, passage of the local temporary table to the validation exit, and execution of the validation exit to validate the first data based on the local temporary table and on the data of the database.

Abstract: In one aspect, a computer-implemented method includes calculating cardinality differences and combined sizes for one or more data block pairs in a relational database. The method also includes selecting a data block pair from the one or more data block pairs based on a calculated cardinality difference and a calculated combined size associated with the selected data block pair. The method further includes merging the selected data block pair to form a merged data block, and replacing the selected data block pair in the relational database by the merged data block.

Abstract: A distributed database system can include a plurality of nodes. In order to execute a query at the distributed database system, one or more of the plurality of nodes can perform operations included in an execution plan for the query. A first node of the plurality of nodes can group the plurality of nodes into at least a first cluster. The first node can further select a second node to act as a gateway node for the first cluster. The first node can determine that at least one recipient node of a message is included in the first cluster and route the message to the recipient node by at least sending the message to the second node. Related methods and articles of manufacture are also disclosed.

Abstract: A system includes acquisition of executable code of a database job, generation of a definition of the database job in a database repository, the definition of the database job comprising metadata specifying the executable code of the database job and a schedule for executing the executable code, association of a user with the database job, and activation of the database job.

Abstract: The present disclosure involves systems, software, and computer implemented methods for preventing reader starvation in a distributed system. One example method includes receiving, at a node that manages a data stream, a write-intent request from a writer for a first partition, the data stream associated with a graph that includes levels of partitions including a current level; identifying the first partition for the writer; providing partition information for the first partition to enable the writer to write to the first partition; determining that the writer has previously written to a second partition associated with the current level; in response to determining that the writer has previously written to the second partition: creating a new current level; associating the first partition with the new current level; identifying partitions associated with the current level; and providing a close-level instruction to each local node that includes a partition associated with the current level.

Abstract: A system, a method, and a computer program product for scalable processes for write-order preserving data stream consumption. A data partition in a plurality of data partitions of data stream is selected based on a request received from a client processing node. The plurality of data partitions are distributed among a plurality of broker nodes in a distributed messaging system. A broker node in the plurality of broker nodes hosting the selected data partition is identified. The identified broker node provides the selected data partition to the client processing node for performing at least one function.

Abstract: In one aspect, a computer-implemented method includes calculating cardinality differences and combined sizes for one or more data block pairs in a relational database. The method also includes selecting a data block pair from the one or more data block pairs based on a calculated cardinality difference and a calculated combined size associated with the selected data block pair. The method further includes merging the selected data block pair to form a merged data block, and replacing the selected data block pair in the relational database by the merged data block.

Abstract: A system includes activation of an OData service exposing a parameterized database view, the OData service exposing a Parameter Entity Type and a Result Entity Type, wherein the Parameter Entity Type specifies one or more input parameters of the database view as properties of the Parameter Entity Type, and defines a Parameter Entity Type key formed by each of the one or more input parameters, and wherein the Result Entity Type specifies the one or more input parameters and one or more output of the database view as properties of the Result Entity Type, and defines a Result Entity Type key formed by each of the one or more input parameters and at least one of the one or more output parameters.

Abstract: In one respect, there is provided a distributed database system that includes a first local scheduler node and a second local scheduler node. The first local scheduler node can be configured to coordinate a performance of a first task and a second task by a first worker node associated with the first local scheduler node. The second local scheduler node can be configured to respond to a defect at the first local scheduler node by at least: collecting, from the first worker node, a log indicating one or more tasks completed by the first worker node prior to the defect at the first local scheduler node; determining, based at least on the log, that the first worker node has completed the first task but not the second task; and coordinating a performance of the second task by the first worker node. Related methods and articles of manufacture are also provided.

Abstract: In one respect, there is provided a distributed database system that includes a global scheduler node and a first local scheduler node. The global scheduler node can be configured to generate an execution plan for a query requiring data stored at and/or managed by a first and a second worker node assigned to the first local scheduler node. The execution plan can include a first fragment having operations to be performed by the first and second worker node. The first local scheduler node can be configured to coordinate the execution of a portion of the first fragment of the execution plan by distributing, to the first and second worker node, operations included in the first fragment of the execution plan, and routing, between the first and second worker node, messages associated with executing the first fragment of the execution plan. Related methods and computer program products are also provided.

Abstract: In one respect, there is provided a distributed database system. The distributed database system can include a plurality of nodes and a plurality of optimizers. A first optimizer of the plurality of optimizers can be configured to: receive, from a client, a command; generate, based at least in part on global metadata cached by the first optimizer, an execution plan for fulfilling the command that includes one or more tasks to be performed by at least a first node of the plurality of nodes; determine that at least one task of the one or more tasks are not successfully performed by the first node; and in response to determining that at least one task of the one or more tasks are not successfully performed by the first node, updating the cached global metadata at the first optimizer.

Abstract: In one respect, there is provided a distributed database system. The distributed database system can include a plurality of nodes. A first node of the plurality of nodes can be configured to: group the plurality of nodes into at least a first cluster; select a second node to act as a gateway node for the first cluster; determine that at least one recipient node of a message is included in the first cluster; and route the message to the recipient node by at least sending the message to the second node. Related methods and articles of manufacture are also disclosed.

Abstract: A system, a method, and a computer program product for scalable processes for write-order preserving data stream consumption. A data partition in a plurality of data partitions of data stream is selected based on a request received from a client processing node. The plurality of data partitions are distributed among a plurality of broker nodes in a distributed messaging system. A broker node in the plurality of broker nodes hosting the selected data partition is identified. The identified broker node provides the selected data partition to the client processing node for performing at least one function.