Abstract: This disclosure generally relates to data delivery in distributed applications. One example method includes identifying a data source associated with a shuffle operation, the data source configured provide data from a data set associated with the shuffle operation; identifying a data sink associated with the shuffle operation, the data sink configured to receive data provided by the data source; associating a shuffler component with the shuffle operation, the shuffler component configured to receive data from the data source and provide the data to the data sink; receiving, by the shuffler component, a first data portion from the data source; providing, by the shuffler component, the first data portion to the data sink; receiving, by the shuffler component, a second data portion from the data source, the second data portion being received from the data source prior to or concurrent with providing the first data portion to the data sink.

Abstract: This disclosure generally relates to data delivery in distributed applications. One example method includes identifying a data source associated with a shuffle operation, the data source configured provide data from a data set associated with the shuffle operation; identifying a data sink associated with the shuffle operation, the data sink configured to receive data provided by the data source; associating a shuffler component with the shuffle operation, the shuffler component configured to receive data from the data source and provide the data to the data sink; receiving, by the shuffler component, a first data portion from the data source; providing, by the shuffler component, the first data portion to the data sink; receiving, by the shuffler component, a second data portion from the data source, the second data portion being received from the data source prior to or concurrent with providing the first data portion to the data sink.

Abstract: Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.

Abstract: Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.

Abstract: Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.

Abstract: Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.

Abstract: A software transactional memory system is provided with overflow handling. The system includes a global version counter with an epoch number and a version number. The system accesses the global version counter prior to and subsequent to memory accesses of transactions to validate read accesses of the transaction. The system includes mechanisms to detect global version number overflow and may allow some or all transactions to execute to completion subsequent to the global version number overflowing. The system also provides publication, privatization, and granular safety properties.