Abstract: Information representative of a graph-based program specification has a plurality of components, each of which corresponds to a task, and directed links between ports of said components. A program corresponding to said graph-based program specification is executed. A first component includes a first data port, a first control port, and a second control port. Said first data port is configured to receive data to be processed by a first task corresponding to said first component, or configured to provide data that was processed by said first task corresponding to said first component. Executing a program corresponding to said graph-based program specification includes: receiving said first control information at said first control port, in response to receiving said first control information, determining whether or not to invoke said first task, and after receiving said first control information, providing said second control information from said second control port.

Abstract: A first component of a graph-based program specification includes an output control port. A second component includes an input control port and an input data port. A third component includes an output data port. The output control port is connected to the input control port, and the output data port is connected to the input data port. The first component includes control code that when executed causes the output control port to provide, to the input control port, at least one of suppression information or invocation information. The second component includes control code that when executed causes a computing system configured by the graph-based program specification to begin processing data received at the input data port in response to the invocation information if no suppression information is received at the input control port before the invocation information is received at the input control port.

Abstract: A received graph-based program specification includes: a plurality of components, each corresponding to at least one operation; and a plurality of directed links each connecting an upstream component to a downstream component. Processed code is generated representing one or more groups of operations by: identifying a possible level of concurrency in a first group of operations based at least in part on a topology of the graph, such that multiple operations in the first group are not prevented by the topology of the graph from executing concurrently; analyzing at least some of the operations in the first group to determine runtime characteristics associated with the analyzed operations; and generating processed code for executing the operations, where the processed code enforces a reduced level of concurrency in the first group, lower than the identified possible level of concurrency, based at least in part on the determined runtime characteristics.

Abstract: Managing data set objects for graph-based data processing includes: storing a group of one or more data set objects in a data storage system, the data set objects each representing a respective data set; and generating an association between at least a first data set object in the group and at least a first node of a dataflow graph for processing data in a data processing system, the first node representing a source or sink of data in a flow of data represented by a link in the dataflow graph, and the first data set object including a plurality of modes in which different transformational logic is applied to data processed by the first node.

Abstract: A first component of a graph-based program specification includes an output control port. A second component includes an input control port and an input data port. A third component includes an output data port. The output control port is connected to the input control port, and the output data port is connected to the input data port. The first component includes control code that when executed causes the output control port to provide, to the input control port, at least one of suppression information or invocation information. The second component includes control code that when executed causes a computing system configured by the graph-based program specification to begin processing data received at the input data port in response to the invocation information if no suppression information is received at the input control port before the invocation information is received at the input control port.

Abstract: Information representative of a graph-based program specification has components, and directed links between ports of said components, defining a dependency between said components. A directed link exists between a port of a first component and a port of a second component. The first component specifies first-component execution code that when compiled enables execution of a first task. The second component specifies second-component execution code that when compiled enables execution of a second task. Compiling the graph-based program specification includes grafting first control code to said first-component execution code, which changes a state of said second component to a pending state, an active state, or a suppressed state. Based on said state, said first control code causes at least one of: invoking said second component if said state changes from pending to active, or suppressing said second component if said state changes from pending to suppressed.

Abstract: A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports, including scalar data ports for sending or receiving a single data element and collection data ports for sending or receiving a collection of multiple data elements; and one or more links, each connecting an output port of an upstream component to an input port of a downstream component. Prepared code is generated representing subsets of the plurality of components, including: identifying one or more subset boundaries, including identifying one or more links connecting a collection data port of a component to a scalar data port of a component; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.

Abstract: Information representative of a graph-based program specification has a plurality of components, each of which corresponds to a task, and directed links between ports of said components. A program corresponding to said graph-based program specification is executed. A first component includes a first data port, a first control port, and a second control port. Said first data port is configured to receive data to be processed by a first task corresponding to said first component, or configured to provide data that was processed by said first task corresponding to said first component. Executing a program corresponding to said graph-based program specification includes: receiving said first control information at said first control port, in response to receiving said first control information, determining whether or not to invoke said first task, and after receiving said first control information, providing said second control information from said second control port.

Abstract: A graph-based program specification includes components, at least one having at least one input port for receiving a collection of data elements, or at least one collection type output port for providing a collection of data elements. Executing a program specified by the graph-based program specification at a computing node, includes: receiving data elements of a first collection into a first storage in a first order via a link connected to a collection type output port of a first component and an input port of a second component, and invoking a plurality of instances of a task corresponding to the second component to process data elements of the first collection, including retrieving the data elements from the first storage in a second order, without blocking invocation of any of the instances until after any particular instance completes processing one or more data elements.

Abstract: A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports, including scalar data ports for sending or receiving a single data element and collection data ports for sending or receiving a collection of multiple data elements; and one or more links, each connecting an output port of an upstream component to an input port of a downstream component. Prepared code is generated representing subsets of the plurality of components, including: identifying one or more subset boundaries, including identifying one or more links connecting a collection data port of a component to a scalar data port of a component; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.

Abstract: A graph-based program specification includes components corresponding to tasks and directed links between ports of the components, including: a first type of link configuration defined by respective output and input ports of linked components, and a second type of link configuration defined by respective output and input ports of linked components. A compiler recognizes different types of link configurations and provides in a target program specification occurrences of a target primitive for executing a function for each occurrence of a data element flowing over a link of the second type. A computing node initiates execution of the target program specification, and determines at runtime, for components associated with the occurrences of the target primitive, an order in which instances of tasks corresponding to the components are to be invoked, and/or a computing node on which instances of tasks corresponding to the components are to be executed.

Abstract: Information representative of a graph-based program specification has a plurality of components, each of which corresponds to a task, and directed links between ports of said components. A program corresponding to said graph-based program specification is executed. A first component includes a first data port, a first control port, and a second control port. Said first data port is configured to receive data to be processed by a first task corresponding to said first component, or configured to provide data that was processed by said first task corresponding to said first component. Executing a program corresponding to said graph-based program specification includes: receiving said first control information at said first control port, in response to receiving said first control information, determining whether or not to invoke said first task, and after receiving said first control information, providing said second control information from said second control port.

Abstract: A graph-based program specification includes components, at least one having at least one input port for receiving a collection of data elements, or at least one collection type output port for providing a collection of data elements. Executing a program specified by the graph-based program specification at a computing node, includes: receiving data elements of a first collection into a first storage in a first order via a link connected to a collection type output port of a first component and an input port of a second component, and invoking a plurality of instances of a task corresponding to the second component to process data elements of the first collection, including retrieving the data elements from the first storage in a second order, without blocking invocation of any of the instances until after any particular instance completes processing one or more data elements.

Abstract: A received graph-based program specification includes: a plurality of components, each corresponding to at least one operation; and a plurality of directed links each connecting an upstream component to a downstream component. Processed code is generated representing one or more groups of operations by: identifying a possible level of concurrency in a first group of operations based at least in part on a topology of the graph, such that multiple operations in the first group are not prevented by the topology of the graph from executing concurrently; analyzing at least some of the operations in the first group to determine runtime characteristics associated with the analyzed operations; and generating processed code for executing the operations, where the processed code enforces a reduced level of concurrency in the first group, lower than the identified possible level of concurrency, based at least in part on the determined runtime characteristics.

Abstract: A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and one or more links, each connecting an output port of an upstream component of the plurality of components to an input port of a downstream component of the plurality of components. Prepared code is generated representing subsets of the plurality of components, including: identifying a plurality of subset boundaries between components in different subsets based at least in part on characteristics of linked components; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in that formed subset to be performed according to information embedded in the prepared code for that formed subset.

Abstract: Entity data represent entities related according to a hierarchy, with at least one root entity at a highest level and one or more entities at one or more levels below the highest level. One or more entities each have multiple instances, and at least one attribute having respective elements for each of the instances. Input data includes expressions that express functions of arguments, with one or more of the arguments referencing an attribute of an entity represented in the entity data. Processing expressions includes: (a) analyzing an expression to determine, for each of one or more attributes, at least one of: (1) a position in the hierarchy of an entity of the attribute, or (2) a number of elements of a result of a function applied to an argument that references the attribute, and (b) evaluating the expression, or indicating an error based on a result of analyzing the expression.

Abstract: A graph-based program specification includes components, at least one having at least one input port for receiving a collection of data elements, or at least one collection type output port for providing a collection of data elements. Executing a program specified by the graph-based program specification at a computing node, includes: receiving data elements of a first collection into a first storage in a first order via a link connected to a collection type output port of a first component and an input port of a second component, and invoking a plurality of instances of a task corresponding to the second component to process data elements of the first collection, including retrieving the data elements from the first storage in a second order, without blocking invocation of any of the instances until after any particular instance completes processing one or more data elements.

Abstract: Information representative of a graph-based program specification has a plurality of components, each of which corresponds to a task, and directed links between ports of said components. A program corresponding to said graph-based program specification is executed. A first component includes a first data port, a first control port, and a second control port. Said first data port is configured to receive data to be processed by a first task corresponding to said first component, or configured to provide data that was processed by said first task corresponding to said first component. Executing a program corresponding to said graph-based program specification includes: receiving said first control information at said first control port, in response to receiving said first control information, determining whether or not to invoke said first task, and after receiving said first control information, providing said second control information from said second control port.

Abstract: Information representative of a graph-based program specification has components, and directed links between ports of said components, defining a dependency between said components. A directed link exists between a port of a first component and a port of a second component. The first component specifies first-component execution code that when compiled enables execution of a first task. The second component specifies second-component execution code that when compiled enables execution of a second task. Compiling the graph-based program specification includes grafting first control code to said first-component execution code, which changes a state of said second component to a pending state, an active state, or a suppressed state. Based on said state, said first control code causes at least one of: invoking said second component if said state changes from pending to active, or suppressing said second component if said state changes from pending to suppressed.

Abstract: A graph-based program specification includes components corresponding to tasks and directed links between ports of the components, including: a first type of link configuration defined by respective output and input ports of linked components, and a second type of link configuration defined by respective output and input ports of linked components. A compiler recognizes different types of link configurations and provides in a target program specification occurrences of a target primitive for executing a function for each occurrence of a data element flowing over a link of the second type. A computing node initiates execution of the target program specification, and determines at runtime, for components associated with the occurrences of the target primitive, an order in which instances of tasks corresponding to the components are to be invoked, and/or a computing node on which instances of tasks corresponding to the components are to be executed.