Abstract: Managing sets of parameter values includes: receiving a plurality of sets of parameter values for a generic computer program, and processing log entries associated with executions of instances of the generic computer program, each instance associated with one or more parameter values. The processing includes: analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class or a second class; processing a log entry associated with an execution of a first instance of the generic computer program to form a particular set of parameter values; and determining whether to add the particular set of parameter values to the plurality of sets of parameter values based on a comparison of a first identifier for the particular set of parameter values to identifiers for at least some of the sets of parameter values.

Abstract: Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Abstract: A specification including a description of a first directed graph including a first plurality of components interconnected by a first set of one or more directed links is received. A graph interface is formed for the first plurality of components including: forming a first interface element of the graph interface, the first interface element being associated with a first port of a first component of the first number of components, and configuring one or more properties of the first interface element such that the first port of the first component is consistent with the one or more properties of the first interface element. A first implementation of the graph interface is formed including the first number of components, the forming including forming a first correspondence between the first interface element and the first port of the first component of the first number of components.

Abstract: Managing sets of parameter values includes: receiving a plurality of sets of parameter values for a generic computer program, and processing log entries associated with executions of instances of the generic computer program, each instance associated with one or more parameter values. The processing includes: analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class or a second class; processing a log entry associated with an execution of a first instance of the generic computer program to form a particular set of parameter values; and determining whether to add the particular set of parameter values to the plurality of sets of parameter values based on a comparison of a first identifier for the particular set of parameter values to identifiers for at least some of the sets of parameter values.

Abstract: Managing sets of parameter values includes: receiving a plurality of sets of parameter values for a generic computer program, and processing log entries associated with executions of instances of the generic computer program, each instance associated with one or more parameter values. The processing includes: analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class or a second class; processing a log entry associated with an execution of a first instance of the generic computer program to form a particular set of parameter values; and determining whether to add the particular set of parameter values to the plurality of sets of parameter values based on a comparison of a first identifier for the particular set of parameter values to identifiers for at least some of the sets of parameter values.

Abstract: Specifying a dataflow graph includes: rendering, in a first user interface, a representation of a first dataflow graph. At least one component represents a computation associated with at least one of data flowing into an input port or data flowing out of an output port. A link between components is rendered based on user input indicating a flow of data. In a second user interface, a representation of a sub-graph of the first dataflow graph is rendered. A first flow junction represents a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. A link is rendered between the first flow junction and a second flow junction based on user input indicating a relationship between a first descriptor and a second descriptor.

Abstract: Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Abstract: Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Abstract: Managing lineage information includes: receiving lineage information representing one or more lineage relationships among two or more data processing programs and two or more logical datasets; receiving one or more runtime artifacts, each runtime artifact including information related to a previous execution of a data processing program of the two or more data processing programs; and analyzing the one or more runtime artifacts and the lineage information to determine one or more candidate modifications to the lineage information.

Abstract: In certain aspects, a method includes: receiving a first graph that includes components and flows, the components representing operations performed on data records, the flows representing flows of data records between components; receiving a specification that is separate from the first graph, the specification defining one or more insertions, each of the insertions associated with a flow of the first graph; generating one or more components that each corresponds to one of the insertions; and generating a second graph that includes components and flows that correspond to at least some of the components and flows of the first graph and the one or more generated components.

Abstract: Managing data set objects for graph-based data processing includes: storing a group of one or more data set objects in a data storage system, the data set objects each representing a respective data set; and generating an association between at least a first data set object in the group and at least a first node of a dataflow graph for processing data in a data processing system, the first node representing a source or sink of data in a flow of data represented by a link in the dataflow graph, and the first data set object including a plurality of modes in which different transformational logic is applied to data processed by the first node.

Abstract: Specifying a dataflow graph includes: rendering, in a first user interface, a representation of a first dataflow graph. At least one component represents a computation associated with at least one of data flowing into an input port or data flowing out of an output port. A link between components is rendered based on user input indicating a flow of data. In a second user interface, a representation of a sub-graph of the first dataflow graph is rendered. A first flow junction represents a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. A link is rendered between the first flow junction and a second flow junction based on user input indicating a relationship between a first descriptor and a second descriptor.

Abstract: A specification including a description of a first directed graph including a first plurality of components interconnected by a first set of one or more directed links is received. A graph interface is formed for the first plurality of components including: forming a first interface element of the graph interface, the first interface element being associated with a first port of a first component of the first number of components, and configuring one or more properties of the first interface element such that the first port of the first component is consistent with the one or more properties of the first interface element. A first implementation of the graph interface is formed including the first number of components, the forming including forming a first correspondence between the first interface element and the first port of the first component of the first number of components.

Abstract: In certain aspects, a method includes: receiving a first graph that includes components and flows, the components representing operations performed on data records, the flows representing flows of data records between components; receiving a specification that is separate from the first graph, the specification defining one or more insertions, each of the insertions associated with a flow of the first graph; generating one or more components that each corresponds to one of the insertions; and generating a second graph that includes components and flows that correspond to at least some of the components and flows of the first graph and the one or more generated components.

Abstract: Configurable testing of a computer program includes: storing a set of one or more testing specifications, and attribute information defining one or more attributes of a recognizable portion of the computer program; and processing, using at least one processor, the computer program according to at least a first testing specification associated with the computer program. The processing includes: traversing a representation of the computer program that includes elements that represent recognizable portions of the computer program, and while traversing the representation, recognizing recognizable portions of the computer program, and storing values of attributes, defined by the attribute information, of the recognized portions of the computer program.

Abstract: Managing lineage information includes: receiving lineage information representing one or more lineage relationships among two or more data processing programs and two or more logical datasets; receiving one or more runtime artifacts, each runtime artifact including information related to a previous execution of a data processing program of the two or more data processing programs; and analyzing the one or more runtime artifacts and the lineage information to determine one or more candidate modifications to the lineage information.

Abstract: Managing sets of parameter values includes: receiving a plurality of sets of parameter values for a generic computer program, and processing log entries associated with executions of instances of the generic computer program, each instance associated with one or more parameter values. The processing includes: analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class or a second class; processing a log entry associated with an execution of a first instance of the generic computer program to form a particular set of parameter values; and determining whether to add the particular set of parameter values to the plurality of sets of parameter values based on a comparison of a first identifier for the particular set of parameter values to identifiers for at least some of the sets of parameter values.

Abstract: Determining metadata associated with a graph-based computation includes functionally transforming metadata associated with a first portion of a graph to generate transformed metadata associated with a second portion of the graph; determining a third portion of the graph related to the second portion of the graph; and propagating the transformed metadata from the second portion of the graph to the third portion of the graph.

Abstract: Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Abstract: Controlling access to computational features includes: preparing a computational resource for execution by an execution system that has been provided a primary descriptor containing an identity value and that has associated a feature indicator with the primary descriptor; accessing a secondary descriptor containing the identity value and cryptographically assigned to the computational resource; and granting the computational resource access to a computational feature of the execution system based on the feature indicator.