Abstract: A method for managing data in a distributed data processing system including a number of processing nodes includes storing data units in data stores that are associated with a number of different levels of durability. The method includes maintaining indicators including a first indicator associated with a first durability level and a second indicator associated with a second durability level. The first indicator is maintained to reflect a time interval at which all sets of data units associated with the time interval are stored at the first durability level. The second indicator is maintained to reflect a timer interval at which all sets of data units associated with the time interval are stored at the second durability level. The first and second indicators are used to manage processing of the data units in the distributed data processing system.

Abstract: A method for processing tasks in a distributed data processing system includes processing sets of tasks. The method includes maintaining, at a first processing node a number of counters including a working counter indicating a current time interval of the number of time intervals in the distributed data processing system, and a replication counter indicating a time interval of the number of time intervals for which at least one of (1) all tasks associated with that time interval, or (2) all corresponding results associated with that time interval, are replicated at multiple processing nodes of the number of processing nodes. The method includes providing messages from the first processing node to the other processing nodes of the number of processing nodes, the messages including the working counter and the replication counter.

Abstract: One or more mappings each define a correspondence between one or more input attributes of an input entity and one or more output attributes of an output entity, where the input entity includes one or more key attributes identified as part of a unique key, and the output entity includes one or more key attributes identified as part of a unique key. Generating instances of the output entity includes: determining one or more mapped input attributes of the input entity that correspond to each of the key attributes of the output entity, based on the mappings; and comparing the mapped input attributes with the key attributes of the input entity to determine whether the mapped input attributes include: (1) all of the key attributes of the input entity, or (2) fewer than all of the key attributes of the input entity.

Abstract: A method for managing a distributed data processing system, the method implementing counters to track durability states of data units in the distributed data processing system, wherein the counters are used to manage processing of the data units in the distributed data processing system.

Abstract: A method includes receiving information indicative of an output dataset generated by a data processing system; identifying, based on data lineage information relating to the output dataset, one or more upstream datasets on which the output dataset depends; analyzing one or more of the identified one or more upstream datasets on which the output dataset depends. The analyzing includes, for each particular upstream dataset of the one or more upstream datasets, applying one or more of: (i) a first rule indicative of an allowable deviation between a profile of the particular upstream dataset and a reference profile for the particular upstream dataset, and (ii) a second rule indicative of one or more allowable values or prohibited values for each of one or more data elements in the particular upstream dataset, and based on the results of applying the one or more rules, selecting one or more of the upstream datasets.

Abstract: A data processing system configured to perform: obtaining a first data lineage representing relationships among physical data elements, the first data lineage being generated at least in part by performing at least one of: (a) analyzing source code of at least one computer program configured to access the physical data elements; and (b) analyzing information obtained during runtime of the at least one computer program; obtaining, based on user input, a second data lineage representing relationships among business data elements; obtaining an association between at least some of the physical data elements of the first data lineage and at least some of the business data elements of the second data lineage; and generating, based on the association between the physical data elements and the business data elements, an indication of agreement or discrepancy between the first data lineage and the second data lineage.

Abstract: Managing sets of parameter values includes: receiving a plurality of sets of parameter values for a generic computer program, and processing log entries associated with executions of instances of the generic computer program, each instance associated with one or more parameter values. The processing includes: analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class or a second class; processing a log entry associated with an execution of a first instance of the generic computer program to form a particular set of parameter values; and determining whether to add the particular set of parameter values to the plurality of sets of parameter values based on a comparison of a first identifier for the particular set of parameter values to identifiers for at least some of the sets of parameter values.

Abstract: A method includes accessing a schema that specifies relationships among datasets, computations on the datasets, or transformations of the datasets, selecting a dataset from among the datasets, and identifying, from the schema, other datasets that are related to the selected dataset. Attributes of the datasets are identified, and logical data representing the identified attributes and relationships among the attributes is generated. The logical data is provided to a development environment, which provides access to portions of the logical data representing the identified attributes. A specification that specifies at least one of the identified attributes in performing an operation is received from the development environment.

Abstract: A method for configuring a first computer executable program includes through a user interface, receiving information indicative of a source of data and a data target; and receiving a characterization of a process, including a type of the process and values for characteristics associated with the process. The method includes based on the received information, automatically assigning values to respective parameters of the first computer executable program to cause the first computer executable program to, when executed, receive data from the source of data and output data to the data target. The method includes automatically configuring the first computer executable program to reference a second computer executable program, including identifying the second computer executable program based on the type of the process; and assigning values to respective parameters of the second computer executable program based on the values for the respective characteristics.

Abstract: A method of managing components in a processing environment is provided. The method includes monitoring (i) a status of each of one or more computing devices, (ii) a status of each of one or more applications, each application hosted by at least one of the computing devices, and (iii) a status of each of one or more jobs, each job associated with at least one of the applications; determining that one of the status of one of the computing devices, the status of one of the applications, and the status of one of the jobs is indicative of a performance issue associated with the corresponding computing device, application, or job, the determination being made based on a comparison of a performance of the computing device, application, or job and at least one predetermined criterion; and enabling an action to be performed associated with the performance issue.

Abstract: Techniques for monitoring execution of structured query language (SQL) queries. The techniques include: obtaining a SQL query; generating a query plan for the SQL query, the query plan comprising a plurality of operations to be performed by the data processing system including at least a first operation and a second operation; displaying a graphical user interface (GUI) containing a plurality of GUI elements including a first GUI element representing the first operation and a second GUI element representing the second operation; during execution of the SQL query, gathering tracking information for the SQL query including gathering first tracking information for the first operation and second tracking information for the second operation; and displaying, in the GUI, at least some of the first tracking information in association with the first GUI element and at least some of the second tracking information in association with the second GUI element.

Abstract: A computer-implemented method includes accessing a plurality of data records, each data record having a plurality of data fields. The method further includes analyzing values for one or more of the data fields for at least some of the plurality of data records and generating a profile of the plurality of data records based on the analyzing. The method further includes formulating at least one subsetting rule based on the profile; and selecting a subset of data records from the plurality of data records based on the at least one subsetting rule.

Abstract: Graph-based computation includes accepting specification information for the graph-based computation, the specification information including a plurality of graph elements, and providing a visual representation of the specification information to a user. A visual representation of one or more groups of the graph elements is determined based on the accepted specification information, including determining a spatial extend of a spatial region for at least a first group of the one or more groups, wherein the spatial extent of the spatial region is determined based at least in part on a spatial extent of each graph element of a subset of graph elements including one or more graph elements in the first group and at least one graph element out of the first group. A visual representation of spatial regions for the one or more groups is presented in conjunction with the visual information of the specification information.

Abstract: Records received from one or more sources in a network are processed. For each of multiple intervals of time, a matching procedure is attempted on sets of one or more records, including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets. The processing also includes, for at least some of the intervals of time, processing at least one complete set, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time, and for at least some of the intervals of time, processing at least one incomplete set, consisting of one or more records stored in the data store before the interval of time.

Abstract: Techniques for allowing a user to invoke lifecycle operations on a data store configured in accordance with a data management scheme, without knowledge of the data management scheme. Techniques include receiving user input from a user interface; identifying, in the received user input, an indication of a first lifecycle operation corresponding to a first entry in a template; based on information in the first entry, identifying in the received user input a first configuration value associated with the first lifecycle operation; formatting the first configuration value as at least a portion of at least one argument to a first command; and invoking, via the first command with the at least one argument, a first computer program configured to perform the first lifecycle operation on the data store.

Abstract: Software specification translation includes: receiving a first software specification specified in a first programming language, a second software specification specified in a second programming language, a third software specification specified in a third programming language different from the first and second programming languages, the third software specification defining one or more data relationships between the first software specification and the second software specification. A combined representation of the first software specification and the second software specification is formed in a fourth programming language different from the first, second, and third programming languages. Connections are formed in the fourth programming language between a representation of the first software specification and the representation of the second software specification according to identified data relationships.

Abstract: Entity data represent entities related according to a hierarchy, with at least one root entity at a highest level and one or more entities at one or more levels below the highest level. One or more entities each have multiple instances, and at least one attribute having respective elements for each of the instances. Input data includes expressions that express functions of arguments, with one or more of the arguments referencing an attribute of an entity represented in the entity data. Processing expressions includes: (a) analyzing an expression to determine, for each of one or more attributes, at least one of: (1) a position in the hierarchy of an entity of the attribute, or (2) a number of elements of a result of a function applied to an argument that references the attribute, and (b) evaluating the expression, or indicating an error based on a result of analyzing the expression.

Abstract: Techniques are described for increasing the functionality of a data processing system via a computer-executed tool that converts programs, written in a procedural language, into components that may be executed as a dataflow graph. The dataflow graph generated from a program written in a procedural programming language may support various forms of parallelism, such as pipeline parallelism and/or component parallelism. In some embodiments, parallelism may be achieved by parsing the program based on a grammar and identifying control flow relationships between data operations performed by the program. In particular, types of dataflow graph components may be identified, according to the grammar, as corresponding to particular data operations (or groups of data operations) of the program. A dataflow graph may be generated to comprise the identified components, which may be connected together with flows in an order according to the identified control flow relationships between data operations.

Abstract: A method is described for processing keyed data items that are each associated with a value of a key, the keyed data items being from a plurality of distinct data streams, the processing including collecting the keyed data items, determining, based on contents of at least one of the keyed data items, satisfaction of one or more specified conditions for execution of one or more actions and causing execution of at least one of the one or more actions responsive to the determining.