Abstract: Distributed processing of a data collection includes receiving information for configuring a distributed processing system. A first configuration of components is formed including sources of data elements and workers configured to process data elements, distributed among computing resources. Each data element includes a partition value that identifies a subset of the workers according to a partition rule. Data elements are accepted from the sources for a first part of the data collection in a first processing epoch and the data elements are routed through the first configuration. After accepting a first part of the data collection, change of configuration is initiated to a second configuration. A succession of two or more transitions between configurations of components is performed to a succession of modified configurations, a last of which corresponds to the second configuration. Further data elements are accepted from sources of the second configuration in a second processing epoch.

Abstract: Information representative of a graph-based program specification has a plurality of components, each of which corresponds to a task, and directed links between ports of said components. A program corresponding to said graph-based program specification is executed. A first component includes a first data port, a first control port, and a second control port. Said first data port is configured to receive data to be processed by a first task corresponding to said first component, or configured to provide data that was processed by said first task corresponding to said first component. Executing a program corresponding to said graph-based program specification includes: receiving said first control information at said first control port, in response to receiving said first control information, determining whether or not to invoke said first task, and after receiving said first control information, providing said second control information from said second control port.

Abstract: A computing system includes nodes executing data processing programs that each process at least one stream of data units. A data storage system stores shared data accessible by at least two of the programs. Processing at least one stream using a first data processing program includes: processing a first stream of data units that includes multiple subsets of contiguous data units; initiating termination of processing within the first data processing program, between processing a first subset of contiguous data units and processing a second subset of contiguous data units adjacent to the first subset of contiguous data units within the first stream of data units; durably storing at least some changes to the shared data caused by processing the first subset of contiguous data units after determining that the termination of processing within the first data processing program has completed; and resuming processing within the first data processing program.

Abstract: Entity data represent entities related according to a hierarchy, with at least one root entity at a highest level and one or more entities at one or more levels below the highest level. One or more entities each have multiple instances, and at least one attribute having respective elements for each of the instances. Input data includes expressions that express functions of arguments, with one or more of the arguments referencing an attribute of an entity represented in the entity data. Processing expressions includes: (a) analyzing an expression to determine, for each of one or more attributes, at least one of: (1) a position in the hierarchy of an entity of the attribute, or (2) a number of elements of a result of a function applied to an argument that references the attribute, and (b) evaluating the expression, or indicating an error based on a result of analyzing the expression.

Abstract: A method implemented by a data processing system for processing data items of a stream of data items, including: accessing a specification that represents the executable logic, wherein a state of the specification for a particular value of the key specifies one or more portions of the executable logic that are executable in that state; receiving, over an input device or port, data items of a stream of data; for a first one of the data items of the stream, identifying a first state of the specification for a value of the key associated with that first one of the data items; processing, by the data processing system, the first one of the data items according to one or more portions of executable logic that are represented in the specification as being associated with the first state.

Abstract: A graph-based program specification specifies at least a partial ordering among a plurality of tasks represented by its nodes. Executing a specified program includes: executing a first subroutine corresponding to a first task, including a first task section for performing the first task; storing state information indicating a state of the first task selected from a set of possible states that includes: a pending state in which the first task section is waiting to perform the first task, and a suppressed state in which the first task section has been prevented from performing the first task; and executing a second subroutine corresponding to a second task, including a second task section for performing the second task, and a control section that controls execution of the second task section based at least in part on the state of the first task indicated by the stored state information.

Abstract: A method for processing state update requests in a distributed data processing system with a number of processing nodes includes maintaining a number of counters including a working counter indicating a current time interval, a replication counter indicating a time interval for which all requests associated with that time interval are replicated at multiple processing nodes of the number of processing nodes, and a persistence counter indicating a time interval of the number of time intervals for which all requests associated with that time interval are stored in persistent storage. The counters are used to manage processing of the state update requests.

Abstract: One or more mappings each define a correspondence between one or more input attributes of an input entity and one or more output attributes of an output entity, where the input entity includes one or more key attributes identified as part of a unique key, and the output entity includes one or more key attributes identified as part of a unique key. Generating instances of the output entity includes: determining one or more mapped input attributes of the input entity that correspond to each of the key attributes of the output entity, based on the mappings; and comparing the mapped input attributes with the key attributes of the input entity to determine whether the mapped input attributes include: (1) all of the key attributes of the input entity, or (2) fewer than all of the key attributes of the input entity.

Abstract: Work units are received into, and processed from, holding queues. A partition, associated with a unique holding queue, is assigned to each work unit stored in a given holding queue, according to a partitioning rule. Work units are separated from each holding queue into one of two or more other queues stored in the same storage as that holding queue. A ready queue (relocation queue) is for each work unit whose assigned partition is (is not) associated with the holding queue in which it is stored. Each work unit in each relocation queue is relocated to the holding queue associated with the partition assigned to that work unit. The work units are repartitioned in response to a change in the partitioning rule to a new partitioning rule before the holding queues are empty.

Abstract: A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and one or more links, each connecting an output port of an upstream component of the plurality of components to an input port of a downstream component of the plurality of components. Prepared code is generated representing subsets of the plurality of components, including: identifying a plurality of subset boundaries between components in different subsets based at least in part on characteristics of linked components; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in that formed subset to be performed according to information embedded in the prepared code for that formed subset.

Abstract: Controlling tasks includes: receiving ordering information that specifies at least a partial ordering among a plurality of tasks; and generating instructions for performing at least some of the tasks based at least in part on the ordering information. Instructions are stored for executing a first subroutine corresponding to a first task, including a first control section that controls execution of at least a second subroutine corresponding to a second task, the first control section including a function configured to change state information associated with the second task, and to determine whether or not to initiate execution of the second subroutine based on the changed state information. Instructions are stored for executing the second subroutine, including a task section for performing the second task and a second control section that controls execution of a third subroutine corresponding to a third task.

Abstract: User input is received specifying components of a graph-based program specification. User input is received specifying links, at least some connecting an output port of an upstream component to an input port of a downstream component. The graph-based program specification is processed to identify one or more subsets of the components, including: identifying one or more subset entry points and one or more subset exit points that occur between components in different subsets based at least in part on data processing characteristics of linked components, and forming the subsets based on the identified subset entry points and exit points. A visual representation of the formed subsets is rendered within a user interface. Prepared code is generated for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.

Abstract: Managing queries performed on one or more data sources includes: storing at least a first query in a storage medium; selecting the first query for processing; instructing a query engine to process the first query on a first portion of data in the one or more data sources for a first query interval; receiving result data from the query engine based on processing the first query on the first portion of data; saving a state of the first query in the storage medium after the first query interval; instructing the query engine to process a second query during a second query interval after the first query interval; and instructing the query engine to process the first query on a second portion of data in the one or more data sources during a third query interval after the second query interval.

Abstract: A received graph-based program specification includes: a plurality of components, each corresponding to at least one operation; and a plurality of directed links each connecting an upstream component to a downstream component. Processed code is generated representing one or more groups of operations by: identifying a possible level of concurrency in a first group of operations based at least in part on a topology of the graph, such that multiple operations in the first group are not prevented by the topology of the graph from executing concurrently; analyzing at least some of the operations in the first group to determine runtime characteristics associated with the analyzed operations; and generating processed code for executing the operations, where the processed code enforces a reduced level of concurrency in the first group, lower than the identified possible level of concurrency, based at least in part on the determined runtime characteristics.

Abstract: Causing a computing system to process events from a sequence of events that defines a correct order for said events independent from an order in which those events are received includes: defining a first variable, defining, for the first variable, a chronology of operations on the first variable associated with received events, receiving a first event that pertains to the first variable, executing a first operation on said first variable that results in a first update of the chronology, receiving a delayed event that pertains to the first variable, executing a second operation on said first variable that results in a second update of the chronology, and determining whether the first update is valid or invalid, wherein the delayed event precedes the first event in the sequence, the first update is based on the first event, and the second update is based on the delayed event.

Abstract: Entity data represent entities related according to a hierarchy, with at least one root entity at a highest level and one or more entities at one or more levels below the highest level. One or more entities each have multiple instances, and at least one attribute having respective elements for each of the instances. Input data includes expressions that express functions of arguments, with one or more of the arguments referencing an attribute of an entity represented in the entity data. Processing expressions includes: (a) analyzing an expression to determine, for each of one or more attributes, at least one of: (1) a position in the hierarchy of an entity of the attribute, or (2) a number of elements of a result of a function applied to an argument that references the attribute, and (b) evaluating the expression, or indicating an error based on a result of analyzing the expression.

Abstract: A first component of a graph-based program specification includes an output control port. A second component includes an input control port and an input data port. A third component includes an output data port. The output control port is connected to the input control port, and the output data port is connected to the input data port. The first component includes control code that when executed causes the output control port to provide, to the input control port, at least one of suppression information or invocation information. The second component includes control code that when executed causes a computing system configured by the graph-based program specification to begin processing data received at the input data port in response to the invocation information if no suppression information is received at the input control port before the invocation information is received at the input control port.

Abstract: A graph-based program specification includes components, at least one having at least one input port for receiving a collection of data elements, or at least one collection type output port for providing a collection of data elements. Executing a program specified by the graph-based program specification at a computing node, includes: receiving data elements of a first collection into a first storage in a first order via a link connected to a collection type output port of a first component and an input port of a second component, and invoking a plurality of instances of a task corresponding to the second component to process data elements of the first collection, including retrieving the data elements from the first storage in a second order, without blocking invocation of any of the instances until after any particular instance completes processing one or more data elements.

Abstract: A method for managing a distributed data processing system, the method implementing counters to track durability states of data units in the distributed data processing system, wherein the counters are used to manage processing of the data units in the distributed data processing system.

Abstract: A method for managing data in a distributed data processing system including a number of processing nodes includes storing data units in data stores that are associated with a number of different levels of durability. The method includes maintaining indicators including a first indicator associated with a first durability level and a second indicator associated with a second durability level. The first indicator is maintained to reflect a time interval at which all sets of data units associated with the time interval are stored at the first durability level. The second indicator is maintained to reflect a timer interval at which all sets of data units associated with the time interval are stored at the second durability level. The first and second indicators are used to manage processing of the data units in the distributed data processing system.