Abstract: Distributed appending of transactions in data lakes is described. A first message is received, at a first ingestion node of a plurality of ingestion nodes, as part of a transaction. The first message identifies a transaction identifier (ID) and a portion of data for the transaction. The data of the first message is persisted in temporary storage. A count of messages for the transaction for the first ingestion node is determined. Based on at least the count of messages, it is determined that the first ingestion node has received a complete set of messages for the transaction for the first ingestion node. A metadata write request is transmitted, by the first ingestion node, to a coordinator. The metadata write request includes a self-describing reference to persisted data. The self-describing reference identifies the first ingestion node, location information of the persisted data, and a range of the first data.

Abstract: Storage file size in distributed data lakes is optimized. At a first ingestion node of a plurality of ingestion nodes, a merge advisory is received from a coordinator. The merge advisory indicates a transaction identifier (ID). Received data associated with the transaction ID is persisted, which includes: determining whether the received data, persisted together in a single file will exceed a maximum desired file size; based on determining that the maximum desired file size will not be exceeded, persisting the received data in a single file; and based on determining that the maximum desired file size will be exceeded, persisting the received data in a plurality of files that each does not exceed the maximum desired file size. A location of the persisted received data in the permanent storage is identified, by the first ingestion node, to the coordinator.

Abstract: References to changing data sets in distributed data lakes are optimized. As part of a transaction, a first message is received. The first message identifies a table and first data to be written to the table. Based on at least the table, the first message is routed to a first ingestion node of a plurality of ingestion nodes. The first data is persisted in temporary storage. Location information of the persisted first data is determined. A data available message comprising a self-describing reference to the first data is published, by the first ingestion node, to a first reader node of a plurality of reader nodes. The self-describing reference identifies the first ingestion node, the location information of the first data, and a range of the first data.