Abstract: An approach to parallel access of data from a distributed filesystem provides parallel access to one or more named units (e.g., files) in the filesystem by creating multiple parallel data streams such that all the data of the desired units is partitioned over the multiple streams. In some examples, the multiple streams form multiple inputs to a parallel implementation of a computation system, such as a graph-based computation system, dataflow-based system, and/or a (e.g., relational) database system.

Abstract: A method for managing a distributed data processing system, the method implementing counters to track durability states of data units in the distributed data processing system, wherein the counters are used to manage processing of the data units in the distributed data processing system.

Abstract: An approach to parallel access of data from a distributed filesystem provides parallel access to one or more named units (e.g., files) in the filesystem by creating multiple parallel data streams such that all the data of the desired units is partitioned over the multiple streams. In some examples, the multiple streams form multiple inputs to a parallel implementation of a computation system, such as a graph-based computation system, dataflow-based system, and/or a (e.g., relational) database system.

Abstract: A method for processing data using a distributed data processing system includes providing input data to the distributed data processing system from a distributed data processing system interface component in communication with the distributed data processing system. The distributed data processing system processes the input data to generate a response, which it provides to the distributed data processing system interface component. The distributed data processing system interface component holds the response in escrow until it receives an indication from the distributed data processing system that the response has reached a sufficient level of durability.

Abstract: An approach to parallel access of data from a distributed filesystem provides parallel access to one or more named units (e.g., files) in the filesystem by creating multiple parallel data streams such that all the data of the desired units is partitioned over the multiple streams. In some examples, the multiple streams form multiple inputs to a parallel implementation of a computation system, such as a graph-based computation system, dataflow-based system, and/or a (e.g., relational) database system.

Abstract: A computing system includes nodes executing data processing programs that each process at least one stream of data units. A data storage system stores shared data accessible by at least two of the programs. Processing at least one stream using a first data processing program includes: processing a first stream of data units that includes multiple subsets of contiguous data units; initiating termination of processing within the first data processing program, between processing a first subset of contiguous data units and processing a second subset of contiguous data units adjacent to the first subset of contiguous data units within the first stream of data units; durably storing at least some changes to the shared data caused by processing the first subset of contiguous data units after determining that the termination of processing within the first data processing program has completed; and resuming processing within the first data processing program.

Abstract: Managing potentially concurrent transactions in a distributed database system includes: maintaining, at a first node, records of transactions executing on one or more nodes, each having a transaction state, execution of a first transaction at a second node including an operation for accessing a first data element stored on the second node and execution of a second transaction at the second node including an operation for accessing the first data element; receiving, at the second node from the first node, a list of transactions including any transactions executing on the second node and having a first transaction state at a time of initiation of the second transaction, and including the first transaction; determining that an outcome of the second transaction depends on an outcome of the first transaction based on the list; and suspending execution of the second transaction until after the first transaction is complete based on the determining.

Abstract: A method for managing a distributed data processing system, the method implementing counters to track durability states of data units in the distributed data processing system, wherein the counters are used to manage processing of the data units in the distributed data processing system.

Abstract: A method for processing data using a distributed data processing system includes providing input data to the distributed data processing system from a distributed data processing system interface component in communication with the distributed data processing system. The distributed data processing system processes the input data to generate a response, which it provides to the distributed data processing system interface component. The distributed data processing system interface component holds the response in escrow until it receives an indication from the distributed data processing system that the response has reached a sufficient level of durability.

Abstract: A computing system includes nodes executing data processing programs that each process at least one stream of data units. A data storage system stores shared data accessible by at least two of the programs. Processing at least one stream using a first data processing program includes: processing a first stream of data units that includes multiple subsets of contiguous data units; initiating termination of processing within the first data processing program, between processing a first subset of contiguous data units and processing a second subset of contiguous data units adjacent to the first subset of contiguous data units within the first stream of data units; durably storing at least some changes to the shared data caused by processing the first subset of contiguous data units after determining that the termination of processing within the first data processing program has completed; and resuming processing within the first data processing program.

Abstract: Managing database transactions in a distributed database system includes: maintaining, at a first node, a first plurality of records of transactions, each associated with a transaction and including a start time of the transaction and a start time of an oldest transaction that was active at the start time of the transaction; maintaining, at a second node, a second plurality of records of transactions, including records of completed transactions associated with the second node, each including a transaction start time and a transaction end time; receiving at the second node, a message from the first node including a start time of an oldest transaction that was active at the transaction start time of the oldest currently active transaction in the system; and removing, from the second plurality of records, any records of completed transactions with a transaction end time occurring before the start time of the oldest transaction.

Abstract: Managing potentially concurrent transactions in a distributed database system includes: maintaining, at a first node, records of transactions executing on one or more nodes, each having a transaction state, execution of a first transaction at a second node including an operation for accessing a first data element stored on the second node and execution of a second transaction at the second node including an operation for accessing the first data element; receiving, at the second node from the first node, a list of transactions including any transactions executing on the second node and having a first transaction state at a time of initiation of the second transaction, and including the first transaction; determining that an outcome of the second transaction depends on an outcome of the first transaction based on the list; and suspending execution of the second transaction until after the first transaction is complete based on the determining.

Abstract: Among other aspects disclosed are a method and system for processing a batch of input data in a fault tolerant manner. The method includes reading a batch of input data including a plurality of records from one or more data sources and passing the batch through a dataflow graph. The dataflow graph includes two or more nodes representing components connected by links representing flows of data between the components. At least one but fewer than all of the components includes a checkpoint process for an action performed for each of multiple units of work associated with one or more of the records. The checkpoint process includes opening a checkpoint buffer stored in non-volatile memory at the start of processing for the batch.

Abstract: Techniques are disclosed that include a computer-implemented method, including storing information related to an initial state of a process upon being initialized, wherein execution of the process includes executing at least one execution phase and upon completion of the executing of the execution phase storing information representative of an end state of the execution phase; aborting execution of the process in response to a predetermined event; and resuming execution of the process from one of the saved initial and end states without needing to shut down the process.

Abstract: Techniques are disclosed that include a computer-implemented method including transmitting a message in response to a predetermined event through a process stage including at least first and second processes being executed as one or more tasks, the message instructing the abortion of the executing of the one or more tasks, and initiating abortion of execution of the one or more tasks by the one or more of the processes on receiving the messages.

Abstract: Techniques are disclosed that include a computer-implemented method, including storing information related to an initial state of a process upon being initialized, wherein execution of the process includes executing at least one execution phase and upon completion of the executing of the execution phase storing information representative of an end state of the execution phase; aborting execution of the process in response to a predetermined event; and resuming execution of the process from one of the saved initial and end states without needing to shut down the process.

Abstract: An approach to parallel access of data from a distributed filesystem provides parallel access to one or more named units (e.g., files) in the filesystem by creating multiple parallel data streams such that all the data of the desired units is partitioned over the multiple streams. In some examples, the multiple streams form multiple inputs to a parallel implementation of a computation system, such as a graph-based computation system, dataflow-based system, and/or a (e.g., relational) database system.

Abstract: Techniques are disclosed that include a computer-implemented method, including storing information related to an initial state of a process upon being initialized, wherein execution of the process includes executing at least one execution phase and upon completion of the executing of the execution phase storing information representative of an end state of the execution phase; aborting execution of the process in response to a predetermined event; and resuming execution of the process from one of the saved initial and end states without needing to shut down the process.

Abstract: A graph-based computation has a plurality of nodes representing graph components connected by one or more links representing data flows between the components. Preparing the computation includes: identifying at least a first set of components to process a plurality of transactions in the computation; and associating with a first transaction a first group of one or more data records and data operations corresponding to the data records, and associating with a second transaction a second group of one or more data records and data operations corresponding to the data records. Executing the computation to process at least the first and second transactions includes delaying execution by the first set of components of a data operation corresponding to an initial data record of the second group until after execution by the first set of components of a data operation corresponding to a final data record of the first group.

Abstract: Among other aspects disclosed are a method and system for processing a batch of input data in a fault tolerant manner. The method includes reading a batch of input data including a plurality of records from one or more data sources and passing the batch through a dataflow graph. The dataflow graph includes two or more nodes representing components connected by links representing flows of data between the components. At least one but fewer than all of the components includes a checkpoint process for an action performed for each of multiple units of work associated with one or more of the records. The checkpoint process includes opening a checkpoint buffer stored in non-volatile memory at the start of processing for the batch.