Abstract: Software specification translation includes: receiving a first software specification specified in a first programming language, a second software specification specified in a second programming language, a third software specification specified in a third programming language different from the first and second programming languages, the third software specification defining one or more data relationships between the first software specification and the second software specification. A combined representation of the first software specification and the second software specification is formed in a fourth programming language different from the first, second, and third programming languages. Connections are formed in the fourth programming language between a representation of the first software specification and the representation of the second software specification according to identified data relationships.

Abstract: Software specification translation includes: receiving a first software specification specified in a first programming language, a second software specification specified in a second programming language, a third software specification specified in a third programming language different from the first and second programming languages, the third software specification defining one or more data relationships between the first software specification and the second software specification. A combined representation of the first software specification and the second software specification is formed in a fourth programming language different from the first, second, and third programming languages. Connections are formed in the fourth programming language between a representation of the first software specification and the representation of the second software specification according to identified data relationships.

Abstract: A method includes receiving a request to process a set of data using a data processing application. The method includes, based on a feature associated with the set of data, selecting between (i) a first mode in which one or more running processes of the data processing application are used to process the set of data and (2) a second mode in which one or more new processes of the data processing application are started up. The method includes causing the data processing application to be executed according to the selected mode to process the set of data.

Abstract: A computer-implemented method includes accessing a plurality of data records, each data record having a plurality of data fields. The method further includes analyzing values for one or more of the data fields for at least some of the plurality of data records and generating a profile of the plurality of data records based on the analyzing. The method further includes formulating at least one subsetting rule based on the profile; and selecting a subset of data records from the plurality of data records based on the at least one subsetting rule.

Abstract: A method of managing components in a processing environment is provided. The method includes monitoring (i) a status of each of one or more computing devices, (ii) a status of each of one or more applications, each application hosted by at least one of the computing devices, and (iii) a status of each of one or more jobs, each job associated with at least one of the applications; determining that one of the status of one of the computing devices, the status of one of the applications, and the status of one of the jobs is indicative of a performance issue associated with the corresponding computing device, application, or job, the determination being made based on a comparison of a performance of the computing device, application, or job and at least one predetermined criterion; and enabling an action to be performed associated with the performance issue.

Abstract: Among other things, we describe a technique for storing data of a data cube in one or more flat files. We also describe a technique for processing a query to access data of a data cube. These techniques can be implemented in a number of ways, including as a method, system, and/or computer program product stored on a computer readable storage device. One of the techniques includes receiving a set of data records having at least two dimensions, generating a set of grouped data records ordered by cardinality, and generating and storing at least one flat file containing the set of grouped data records, wherein a particular data record of the grouped data records includes a primary key that can be used to identify data of the particular data record in response to a request.

Abstract: According to some aspects, a tool is provided that reduces errors made by a data processing system by assisting a user in determining a record format for a dataset by dynamically analyzing contents of the dataset based on real-time feedback provided by the user. The data processing system may apply the determined record format to automatically parse contents of the dataset, with fewer errors. According to some aspects, the tool may generate a user interface that allows a user to identify delimiters based on the content of the dataset, and may generate a provisional record format according to the identified delimiters.

Abstract: One or more mappings each define a correspondence between input attributes of an input entity and output attributes of an output entity, where the input out output entities each include one or more key attributes identified as part of a unique key. Computing result information, displayed in a user interface, includes: processing instances of a first input entity to generate instances of a first output entity; determining one or more mapped input attributes of the first input entity that correspond to each of the key attributes of the first output entity; generating the instances of the first output entity based on the determined one or more mapped input attributes; computing a total number of instances of the first input entity that were processed; and computing a total number of instances of the first output entity that were generated.

Abstract: A computing system processes data units using one of at least two different modes of applying a rule. In a first mode, data units are received in a particular order and are processed including writing an updated value to at least one state variable based on a result of applying the rule to the data unit. In a second mode, a selection of particular data units is processed including determining a first set of data units including an ordered subset of data units that occur before the particular data unit from the number of data units, prior to applying the rule to the particular data unit, updating at least one state variable to a state that would result from processing the first set of data units in the first mode, and applying the rule to the particular data unit including reading the updated value of the state variable.

Abstract: One or more mappings each define a correspondence between one or more input attributes of an input entity and one or more output attributes of an output entity, where the input entity includes one or more key attributes identified as part of a unique key, and the output entity includes one or more key attributes identified as part of a unique key. Generating instances of the output entity includes: determining one or more mapped input attributes of the input entity that correspond to each of the key attributes of the output entity, based on the mappings; and comparing the mapped input attributes with the key attributes of the input entity to determine whether the mapped input attributes include: (1) all of the key attributes of the input entity, or (2) fewer than all of the key attributes of the input entity.

Abstract: In general, metadata is stored in a data storage system. Summary data identifying one or more characteristics of each of multiple metadata objects stored in the data storage system is computed, and the summary data characterizing a given metadata object in association with the given metadata object is stored. A visual representation is generated of a diagram including nodes representing respective metadata objects and relationships among the nodes. Generating the visual representation includes superimposing a representation of a characteristic identified by the summary data characterizing a given metadata object in proximity to the node representing the given metadata object.

Abstract: A method for applying a rule to data includes receiving data from a data source and processing a particular data unit of an ordered number of data units derived at least in part from the received data. The processing includes determining a set of data units including an ordered subset of data units from the ordered number of data units, each data unit of the ordered subset being related to the particular data unit and occurring before the particular data unit in the ordered number of data units. Prior to applying the rule the state variable is updated to a state that would result from iterating in order through the set of data units and, for each data unit of the ordered subset, applying the rule to the data unit. The rule is then applied to the particular data unit including reading the updated value of the state variable.

Abstract: A method includes receiving data indicative of a number of times each of one or more rules was executed by a data processing application during processing of one or more records; based on the number of times each of the rules was executed by the data processing application, determining a content criterion for each of one or more particular fields; generating content for each of the particular fields based on the content criterion; and populating each of the particular fields with the generated content.

Abstract: Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

Abstract: Managing lineage information includes: receiving lineage information representing one or more lineage relationships among two or more data processing programs and two or more logical datasets; receiving one or more runtime artifacts, each runtime artifact including information related to a previous execution of a data processing program of the two or more data processing programs; and analyzing the one or more runtime artifacts and the lineage information to determine one or more candidate modifications to the lineage information.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for impact analysis. One of the methods includes receiving information about at least two logical datasets, the information identifying, for each logical dataset, a field in that logical dataset and format information about that field. The method includes receiving information about a transformation identifying a first logical dataset from which the transformation is to receive data and a second logical dataset to which the transformed data is provided. The method includes receiving one or more proposed changes to at least one of the fields. The method includes analyzing the proposed changes based on information about the transformation and information about the first logical dataset and the second logical dataset. The method includes calculating metrics of the proposed change based on the analysis. The method also includes storing information about the metrics.

Abstract: User input is received specifying components of a graph-based program specification. User input is received specifying links, at least some connecting an output port of an upstream component to an input port of a downstream component. The graph-based program specification is processed to identify one or more subsets of the components, including: identifying one or more subset entry points and one or more subset exit points that occur between components in different subsets based at least in part on data processing characteristics of linked components, and forming the subsets based on the identified subset entry points and exit points. A visual representation of the formed subsets is rendered within a user interface. Prepared code is generated for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.

Abstract: Processing transactions in a distributed computing system that includes multiple processing modules includes: storing data items in a data storage system accessible to multiple processes running in the distributed computing system, where the data items are totally ordered according to an ordering rule, and at least some of the processes are running on different processing modules; and processing transactions using a plurality of the multiple processes. Processing a transaction using one of the plurality of the multiple processes includes: receiving a set of requests for accessing data items stored in the data storage system (where the requests are in a first order), obtaining locks on the data items sequentially in the first order if each of the locks is obtained within a first time interval, and, if any of the locks is not obtained within the first time interval, restarting the transaction being processed.

Abstract: A dependency analyzer for a data processing system comprising at least one computer hardware processor configured to generate dependency information among variables, which may appear in any of multiple programs written in different source languages. The data processing system may parse each program, regardless of the language in which the module was written. Parsed information about each program may be recorded in a first-type data structure and then may be converted to a format representing dependencies among variables. Dependency information for each of the plurality of programs may be expressed as a combination of language independent constructs, which may be processed together, to generate dependency information for the data processing system. The dependency information may be recorded in a dependency data structure and further used for operations, such as data quality checking and change control for the data processing program.