Abstract: A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports, including scalar data ports for sending or receiving a single data element and collection data ports for sending or receiving a collection of multiple data elements; and one or more links, each connecting an output port of an upstream component to an input port of a downstream component. Prepared code is generated representing subsets of the plurality of components, including: identifying one or more subset boundaries, including identifying one or more links connecting a collection data port of a component to a scalar data port of a component; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.

Abstract: A method includes determining a length of a file and storing the length of the file in a first memory location. An endpoint of a last complete record within the file is determined and the endpoint is stored in a second memory location. The length of the file stored in the first memory location is compared to a current length of the file, and a data structure associated with the file is updated beginning at the endpoint if the current length of the file exceeds the length of the file stored in the first memory location.

Abstract: Information representative of a graph-based program specification has a plurality of components, each of which corresponds to a task, and directed links between ports of said components. A program corresponding to said graph-based program specification is executed. A first component includes a first data port, a first control port, and a second control port. Said first data port is configured to receive data to be processed by a first task corresponding to said first component, or configured to provide data that was processed by said first task corresponding to said first component. Executing a program corresponding to said graph-based program specification includes: receiving said first control information at said first control port, in response to receiving said first control information, determining whether or not to invoke said first task, and after receiving said first control information, providing said second control information from said second control port.

Abstract: A graph-based program specification includes components, at least one having at least one input port for receiving a collection of data elements, or at least one collection type output port for providing a collection of data elements. Executing a program specified by the graph-based program specification at a computing node, includes: receiving data elements of a first collection into a first storage in a first order via a link connected to a collection type output port of a first component and an input port of a second component, and invoking a plurality of instances of a task corresponding to the second component to process data elements of the first collection, including retrieving the data elements from the first storage in a second order, without blocking invocation of any of the instances until after any particular instance completes processing one or more data elements.

Abstract: A method implemented by a data processing system for processing data items of a stream of data items, including: accessing a specification that represents the executable logic, wherein a state of the specification for a particular value of the key specifies one or more portions of the executable logic that are executable in that state; receiving, over an input device or port, data items of a stream of data; for a first one of the data items of the stream, identifying a first state of the specification for a value of the key associated with that first one of the data items; processing, by the data processing system, the first one of the data items according to one or more portions of executable logic that are represented in the specification as being associated with the first state.

Abstract: A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports, including scalar data ports for sending or receiving a single data element and collection data ports for sending or receiving a collection of multiple data elements; and one or more links, each connecting an output port of an upstream component to an input port of a downstream component. Prepared code is generated representing subsets of the plurality of components, including: identifying one or more subset boundaries, including identifying one or more links connecting a collection data port of a component to a scalar data port of a component; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.

Abstract: Distributed processing of a data collection includes receiving information for configuring a distributed processing system. A first configuration of components is formed including sources of data elements and workers configured to process data elements, distributed among computing resources. Each data element includes a partition value that identifies a subset of the workers according to a partition rule. Data elements are accepted from the sources for a first part of the data collection in a first processing epoch and the data elements are routed through the first configuration. After accepting a first part of the data collection, change of configuration is initiated to a second configuration. A succession of two or more transitions between configurations of components is performed to a succession of modified configurations, a last of which corresponds to the second configuration. Further data elements are accepted from sources of the second configuration in a second processing epoch.

Abstract: In general, a value of a numerical attribute of a record stored in a data structure is received. A numerical range is generated that includes the value of the numerical attribute. An entry is stored, in an index associated with the data structure, that specifies a location of the record within the data structure and that includes a first index key and a second index key. The first index key corresponds to a value of an attribute of the record different from the numerical attribute, and the second index key corresponds to the generated numerical range.

Abstract: A graph-based program specification includes components corresponding to tasks and directed links between ports of the components, including: a first type of link configuration defined by respective output and input ports of linked components, and a second type of link configuration defined by respective output and input ports of linked components. A compiler recognizes different types of link configurations and provides in a target program specification occurrences of a target primitive for executing a function for each occurrence of a data element flowing over a link of the second type. A computing node initiates execution of the target program specification, and determines at runtime, for components associated with the occurrences of the target primitive, an order in which instances of tasks corresponding to the components are to be invoked, and/or a computing node on which instances of tasks corresponding to the components are to be executed.

Abstract: Information representative of a graph-based program specification has a plurality of components, each of which corresponds to a task, and directed links between ports of said components. A program corresponding to said graph-based program specification is executed. A first component includes a first data port, a first control port, and a second control port. Said first data port is configured to receive data to be processed by a first task corresponding to said first component, or configured to provide data that was processed by said first task corresponding to said first component. Executing a program corresponding to said graph-based program specification includes: receiving said first control information at said first control port, in response to receiving said first control information, determining whether or not to invoke said first task, and after receiving said first control information, providing said second control information from said second control port.

Abstract: Processing data includes: receiving units of work that each include one or more work elements, and processing a first unit of work using a first compiled dataflow graph (160) loaded into a data processing system (100) in response to receiving the first unit of work. The processing includes: analysis to determine a characteristic of the first unit of work; identifying one or more compiled dataflow graphs from graphs stored in a data storage system (107) that include at least some that were compiled for processing a unit of work having the determined characteristic; loading one of the identified compiled dataflow graphs into the data processing system (100) as the first compiled dataflow graph (160); and generating one or more output work elements from at least one work element in the first unit of work.

Abstract: A graph-based program specification includes components, at least one having at least one input port for receiving a collection of data elements, or at least one collection type output port for providing a collection of data elements. Executing a program specified by the graph-based program specification at a computing node, includes: receiving data elements of a first collection into a first storage in a first order via a link connected to a collection type output port of a first component and an input port of a second component, and invoking a plurality of instances of a task corresponding to the second component to process data elements of the first collection, including retrieving the data elements from the first storage in a second order, without blocking invocation of any of the instances until after any particular instance completes processing one or more data elements.

Abstract: A computing system includes nodes executing data processing programs that each process at least one stream of data units. A data storage system stores shared data accessible by at least two of the programs. Processing at least one stream using a first data processing program includes: processing a first stream of data units that includes multiple subsets of contiguous data units; initiating termination of processing within the first data processing program, between processing a first subset of contiguous data units and processing a second subset of contiguous data units adjacent to the first subset of contiguous data units within the first stream of data units; durably storing at least some changes to the shared data caused by processing the first subset of contiguous data units after determining that the termination of processing within the first data processing program has completed; and resuming processing within the first data processing program.

Abstract: Work units are received into, and processed from, holding queues. A partition, associated with a unique holding queue, is assigned to each work unit stored in a given holding queue, according to a partitioning rule. Work units are separated from each holding queue into one of two or more other queues stored in the same storage as that holding queue. A ready queue (relocation queue) is for each work unit whose assigned partition is (is not) associated with the holding queue in which it is stored. Each work unit in each relocation queue is relocated to the holding queue associated with the partition assigned to that work unit. The work units are repartitioned in response to a change in the partitioning rule to a new partitioning rule before the holding queues are empty.

Abstract: A method for promoting fault tolerance and recovery in a computing system including at least one processing node includes promoting availability and recovery of a first processing node, by, at the first processing node, generating first spawn using a spawner that has been assigned a first generation-indicator so that its spawn inherits the first generation indicator, beginning a checkpoint interval to generate nodal recovery information, suspending the spawner from generating spawn, assigning, to the spawner, a second generation-indicator that differs from the first one, resuming the spawner, so that it generates second spawn that inherits the second generation-indicator, controlling an extent to which the second spawn writes to memory, and after committing nodal recovery information acquired during the checkpoint to durable storage, releasing control over the extent to which the second spawn can write to memory.

Abstract: Controlling tasks includes: receiving ordering information that specifies at least a partial ordering among a plurality of tasks; and generating instructions for performing at least some of the tasks based at least in part on the ordering information. Instructions are stored for executing a first subroutine corresponding to a first task, including a first control section that controls execution of at least a second subroutine corresponding to a second task, the first control section including a function configured to change state information associated with the second task, and to determine whether or not to initiate execution of the second subroutine based on the changed state information. Instructions are stored for executing the second subroutine, including a task section for performing the second task and a second control section that controls execution of a third subroutine corresponding to a third task.

Abstract: A received graph-based program specification includes: a plurality of components, each corresponding to at least one operation; and a plurality of directed links each connecting an upstream component to a downstream component. Processed code is generated representing one or more groups of operations by: identifying a possible level of concurrency in a first group of operations based at least in part on a topology of the graph, such that multiple operations in the first group are not prevented by the topology of the graph from executing concurrently; analyzing at least some of the operations in the first group to determine runtime characteristics associated with the analyzed operations; and generating processed code for executing the operations, where the processed code enforces a reduced level of concurrency in the first group, lower than the identified possible level of concurrency, based at least in part on the determined runtime characteristics.

Abstract: An approach to automatically specifying, or assisting with the specification of, a parallel computation graph involves determining data processing characteristics of the linking elements that couple data processing elements of the graph. The characteristics of the linking elements are determined according to the characteristics of the upstream and/or downstream data processing elements associated with the linking element, for example, to enable computation by the parallel computation graph that is equivalent to computation of an associated serial graph.

Abstract: Controlling tasks includes: receiving ordering information that specifies at least a partial ordering among a plurality of tasks; and generating instructions for performing at least some of the tasks based at least in part on the ordering information. Instructions are stored for executing a first subroutine corresponding to a first task, including a first control section that controls execution of at least a second subroutine corresponding to a second task, the first control section including a function configured to change state information associated with the second task, and to determine whether or not to initiate execution of the second subroutine based on the changed state information. Instructions are stored for executing the second subroutine, including a task section for performing the second task and a second control section that controls execution of a third subroutine corresponding to a third task.

Abstract: Processing transactions in a distributed computing system that includes multiple processing modules includes: storing data items in a data storage system accessible to multiple processes running in the distributed computing system, where the data items are totally ordered according to an ordering rule, and at least some of the processes are running on different processing modules; and processing transactions using a plurality of the multiple processes. Processing a transaction using one of the plurality of the multiple processes includes: receiving a set of requests for accessing data items stored in the data storage system (where the requests are in a first order), obtaining locks on the data items sequentially in the first order if each of the locks is obtained within a first time interval, and, if any of the locks is not obtained within the first time interval, restarting the transaction being processed.