Abstract: A graph-based program specification specifies at least a partial ordering among a plurality of tasks represented by its nodes. Executing a specified program includes: executing a first subroutine corresponding to a first task, including a first task section for performing the first task; storing state information indicating a state of the first task selected from a set of possible states that includes: a pending state in which the first task section is waiting to perform the first task, and a suppressed state in which the first task section has been prevented from performing the first task; and executing a second subroutine corresponding to a second task, including a second task section for performing the second task, and a control section that controls execution of the second task section based at least in part on the state of the first task indicated by the stored state information.

Abstract: A data processing system configured to perform: obtaining a first data lineage representing relationships among physical data elements, the first data lineage being generated at least in part by performing at least one of: (a) analyzing source code of at least one computer program configured to access the physical data elements; and (b) analyzing information obtained during runtime of the at least one computer program; obtaining, based on user input, a second data lineage representing relationships among business data elements; obtaining an association between at least some of the physical data elements of the first data lineage and at least some of the business data elements of the second data lineage; and generating, based on the association between the physical data elements and the business data elements, an indication of agreement or discrepancy between the first data lineage and the second data lineage.

Abstract: A method for processing state update requests in a distributed data processing system with a number of processing nodes includes maintaining a number of counters including a working counter indicating a current time interval, a replication counter indicating a time interval for which all requests associated with that time interval are replicated at multiple processing nodes of the number of processing nodes, and a persistence counter indicating a time interval of the number of time intervals for which all requests associated with that time interval are stored in persistent storage. The counters are used to manage processing of the state update requests.

Abstract: Specifying rules for processing data included in fields of elements of a dataset includes rendering user interface elements associated with a respective condition. The user interface elements include: first subsets of user interface elements, at least some of which are associated with an input value derived from at least one field, and second subsets of user interface elements, each configured to receive user input associated with a respective condition. Conditions are applied to at least a first element of the dataset based on user input received from at least some of the user interface elements, in response to receiving user input for a first user interface element associated with a first field. Instructions are generated for applying one or more selected conditions associated with fewer than all of the user interface elements, the selected conditions including at least a condition associated with the first user interface element.

Abstract: Managing task execution includes: receiving a specification of a plurality of tasks to be performed by respective functional modules; processing a flow of input data using a dataflow graph that includes nodes representing data processing components connected by links representing flows of data between data processing components; in response to at least one flow of data provided by at least one data processing component, generating a flow of messages; and in response to each of the messages in the flow of messages, performing an iteration of a set of one or more tasks using one or more corresponding functional modules.

Abstract: Managing lineage information includes processing a request for a representation of data lineage for a first node of a number of nodes. The processing includes determining an association between the first node and at least a first tag identifier of a number of tag identifiers, and determining a first subset of at least one and fewer than all of a number of possible tag values for the first tag identifier, and traversing nodes along a first lineage path of directed links from the first node to determine a data lineage for the first node. Determining the data lineage includes, for each traversed node determining whether to add the traversed node to the data lineage or to exclude the traversed node from the data lineage based at least in part on any tag identifiers or tag values associated with the traversed node.

Abstract: Managing lineage information includes processing a specification of a directed graph to associate nodes with information for processing requests for a representation of data lineage. The processing includes: identifying a first set of one or more nodes of the directed graph corresponding to normalizing data elements being stored in a data store and de-normalizing data elements being retrieved from the data store; and associating a first plurality of nodes connected to the first set of one or more nodes and a second plurality of nodes connected to the first set of one or more nodes with at least one tag identifier having a plurality of possible tag values, where the number of possible tag values is at least as large as the number of data elements being normalized, and where nodes representing different data elements in a de-normalized record are associated with different values of the tag identifier.

Abstract: In one aspect, in general, a method of generating a dataflow graph representing a database query includes receiving a query plan from a plan generator, the query plan representing operations for executing a database query on at least one input representing a source of data, producing a dataflow graph from the query plan, wherein the dataflow graph includes at least one node that represents at least one operation represented by the query plan, and includes at least one link that represents at least one dataflow associated with the query plan, and altering one or more components of the dataflow graph based on at least one characteristic of the at least one input representing the source of data.

Abstract: A first set of original records is processed by a first processing entity to generate a second set of records that includes the original records and one or more copies of each original record, each original record including one or more fields. The processing of each of at least some of the original records includes: generating at least one copy of the original record, and associating a first segment value with the original record and associating a second segment value with the copy. The method also includes partitioning the second set of records among a plurality of recipient processing entities based on the segment values associated with the records in the second set, and, at each recipient processing entity, performing an operation based on one or more data values of the records received at the recipient processing entity to generate results.

Abstract: Work units are received into, and processed from, holding queues. A partition, associated with a unique holding queue, is assigned to each work unit stored in a given holding queue, according to a partitioning rule. Work units are separated from each holding queue into one of two or more other queues stored in the same storage as that holding queue. A ready queue (relocation queue) is for each work unit whose assigned partition is (is not) associated with the holding queue in which it is stored. Each work unit in each relocation queue is relocated to the holding queue associated with the partition assigned to that work unit. The work units are repartitioned in response to a change in the partitioning rule to a new partitioning rule before the holding queues are empty.

Abstract: At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform: obtaining an automatically generated initial dataflow graph, the initial dataflow graph comprising a first plurality of nodes representing a first plurality of data processing operations and a first plurality of links representing flows of data among nodes in the first plurality of nodes; and generating an updated dataflow graph by iteratively applying dataflow graph optimization rules to update the initial dataflow graph, the updated dataflow graph comprising a second plurality of nodes representing a second plurality of data processing operations and a second plurality of links representing flows of data among nodes in the second plurality of nodes.

Abstract: A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and one or more links, each connecting an output port of an upstream component of the plurality of components to an input port of a downstream component of the plurality of components. Prepared code is generated representing subsets of the plurality of components, including: identifying a plurality of subset boundaries between components in different subsets based at least in part on characteristics of linked components; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in that formed subset to be performed according to information embedded in the prepared code for that formed subset.

Abstract: A computer-implemented method including: receiving data representing updates to transaction entries in a ledger of an accounting system, each transaction entry identifying an account that corresponds to the transaction, at least some of the updates a) each changing the identified account from a first account to a second account and b) having been submitted by multiple users of the accounting system; analyzing the updates to infer a correlation between the first account and the second account; generating a rule defining a modification of transaction entries, wherein the rule, if applied to transaction entries received by the accounting system, would modify transaction entries identifying the first account to identify the second account; and storing the rule in a storage system such that the stored rule can be accessed and executed by a computer program executing on one or more computer systems, the execution of the rule causing the modification of the transaction entries defined by the rule.

Abstract: Validation rules are specified for validating data included in fields of elements of a dataset. Cells are rendered in a two-dimensional grid that includes: one or more subsets of the cells extending in a direction along a first axis, each associated with a respective field, and multiple subsets of the cells extending in a direction along a second axis, one or more of the subsets associated with a respective validation rule. Validation rules are applied to at least one element based on user input received from at least some of the cells. Some cells, associated with a field and a validation rule, can each include: an input element for receiving input determining whether or not the associated validation rule is applied to the associated field, and/or an indicator for indicating feedback associated with a validation result based on applying the associated validation rule to data included in the associated field.

Abstract: A data processing system configured to perform: obtaining a first data lineage representing relationships among physical data elements, the first data lineage being generated at least in part by performing at least one of: (a) analyzing source code of at least one computer program configured to access the physical data elements; and (b) analyzing information obtained during runtime of the at least one computer program; obtaining, based on user input, a second data lineage representing relationships among business data elements; obtaining an association between at least some of the physical data elements of the first data lineage and at least some of the business data elements of the second data lineage; and generating, based on the association between the physical data elements and the business data elements, an indication of agreement or discrepancy between the first data lineage and the second data lineage.

Abstract: Controlling tasks includes: receiving ordering information that specifies at least a partial ordering among a plurality of tasks; and generating instructions for performing at least some of the tasks based at least in part on the ordering information. Instructions are stored for executing a first subroutine corresponding to a first task, including a first control section that controls execution of at least a second subroutine corresponding to a second task, the first control section including a function configured to change state information associated with the second task, and to determine whether or not to initiate execution of the second subroutine based on the changed state information. Instructions are stored for executing the second subroutine, including a task section for performing the second task and a second control section that controls execution of a third subroutine corresponding to a third task.

Abstract: Managing potentially concurrent transactions in a distributed database system includes: maintaining, at a first node, records of transactions executing on one or more nodes, each having a transaction state, execution of a first transaction at a second node including an operation for accessing a first data element stored on the second node and execution of a second transaction at the second node including an operation for accessing the first data element; receiving, at the second node from the first node, a list of transactions including any transactions executing on the second node and having a first transaction state at a time of initiation of the second transaction, and including the first transaction; determining that an outcome of the second transaction depends on an outcome of the first transaction based on the list; and suspending execution of the second transaction until after the first transaction is complete based on the determining.

Abstract: A dependency analyzer for a data processing system comprising at least one computer hardware processor configured to generate dependency information among variables, which may appear in any of multiple programs written in different source languages. The data processing system may parse each program, regardless of the language in which the module was written. Parsed information about each program may be recorded in a first-type data structure and then may be converted to a format representing dependencies among variables. Dependency information for each of the plurality of programs may be expressed as a combination of language independent constructs, which may be processed together, to generate dependency information for the data processing system. The dependency information may be recorded in a dependency data structure and further used for operations, such as data quality checking and change control for the data processing program.

Abstract: Managing queries performed on one or more data sources includes: storing at least a first query in a storage medium; selecting the first query for processing; instructing a query engine to process the first query on a first portion of data in the one or more data sources for a first query interval; receiving result data from the query engine based on processing the first query on the first portion of data; saving a state of the first query in the storage medium after the first query interval; instructing the query engine to process a second query during a second query interval after the first query interval; and instructing the query engine to process the first query on a second portion of data in the one or more data sources during a third query interval after the second query interval.