Abstract: A method implemented by a data processing system for enabling a user to browse a data catalog and select fields from multiple data sources to be integrated into a data profile so that, when a request is received for the data profile, data from those fields can be made available efficiently and immediately.

Abstract: A data processing system for identifying one or more portions of executable logic that are executed in processing one or more data items that are each associated with a given value of a key by accessing the specification that represents the executable logic; and generating a log record for the given value of the key, the generating including: for each of the one or more of the data items associated with the given value of the key: identifying a given state of the specification that is maintained for the given value of the key associated with that data item; processing that data item including executing, on that data item, executable logic that are specified in the identified given state of the specification; and generating the log record specifying the execution of those one or more portions of the executable logic for that given value of the key.

Abstract: A method implemented by a data processing system for enabling a user to browse a data catalog and select fields from multiple data sources to be integrated into a data profile so that, when a request is received for the data profile, data from those fields can be made available efficiently and immediately

Abstract: Described are techniques for back-calculating one or more values of a new, real-time aggregate before sufficient data to calculate the new, real-time aggregate has been collected, wherein the back-calculating is based on data collected for one or more aggregates that have been executing prior to start of execution of the new, real-time aggregate.

Abstract: A method for performing real-time segmentation by updating a wide record based on receipt of real-time data, wherein an item of real-time data represents a transaction, detecting that the updated wide record satisfies criteria for performing real-time segmentation, and performing real-time segmentation on the updated, wide record, wherein real-time is relative to when a transaction represented in the updated wide record occurs.

Abstract: A method for enabling a user to generate a complex aggregation on their own by providing the user with a graphical user interface that displays data items in a data catalog and that provides controls for the user to select data items to be used in generating the complex aggregation, and to select a type of aggregation, and based on the user's selections, automatically generating computer instructions to generate a value of the complex aggregation is described.

Abstract: A method is described for processing keyed data items that are each associated with a value of a key, the keyed data items being from a plurality of distinct data streams, the processing including collecting the keyed data items, determining, based on contents of at least one of the keyed data items, satisfaction of one or more specified conditions for execution of one or more actions and causing execution of at least one of the one or more actions responsive to the determining.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for workload automation and job scheduling information. One of the methods includes obtaining job dependency information, the job dependency information specifying an order of execution of a plurality of jobs. The method also includes obtaining data lineage information that identifies dependency relationships between data stores and transformation, wherein at least one transformation accepts data from a first data store and produces data for a second data store. The method also includes creating links between the job dependency information and the data lineage information. The method also includes determining an impact of a change in a planned execution of an application of the plurality of applications based on the job dependency information, the created links, and the data lineage information.

Abstract: A data processing system configured to perform: obtaining a first data lineage representing relationships among physical data elements, the first data lineage being generated at least in part by performing at least one of: (a) analyzing source code of at least one computer program configured to access the physical data elements; and (b) analyzing information obtained during runtime of the at least one computer program; obtaining, based on user input, a second data lineage representing relationships among business data elements; obtaining an association between at least some of the physical data elements of the first data lineage and at least some of the business data elements of the second data lineage; and generating, based on the association between the physical data elements and the business data elements, an indication of agreement or discrepancy between the first data lineage and the second data lineage.

Abstract: A computer-implemented method for integrating client portals of underlying data processing applications through a shared log record, including: storing one or more log records that are each shared by the process management application and the version control application; receiving instructions through a user interface that integrates, through the shared one or more log records, the process management client portal with the version control client portal; in response to the receiving of the instructions, executing the received instructions, the executing of the received instructions including: selecting, by the version control application, a particular version of the rule from the multiple versions of the rule stored in the system storage; and transitioning, by the process management application, the particular version of the rule from the first state of the plurality of states to the second, different state of the plurality of states.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for impact analysis. One of the methods includes receiving information about at least two logical datasets, the information identifying, for each logical dataset, a field in that logical dataset and format information about that field. The method includes receiving information about a transformation identifying a first logical dataset from which the transformation is to receive data and a second logical dataset to which the transformed data is provided. The method includes receiving one or more proposed changes to at least one of the fields. The method includes analyzing the proposed changes based on information about the transformation and information about the first logical dataset and the second logical dataset. The method includes calculating metrics of the proposed change based on the analysis. The method also includes storing information about the metrics.

Abstract: A method is described for processing keyed data items that are each associated with a value of a key, the keyed data items being from a plurality of distinct data streams, the processing including collecting the keyed data items, determining, based on contents of at least one of the keyed data items, satisfaction of one or more specified conditions for execution of one or more actions and causing execution of at least one of the one or more actions responsive to the determining.

Abstract: In general, metadata is stored in a data storage system. Summary data identifying one or more characteristics of each of multiple metadata objects stored in the data storage system is computed, and the summary data characterizing a given metadata object in association with the given metadata object is stored. A visual representation is generated of a diagram including nodes representing respective metadata objects and relationships among the nodes. Generating the visual representation includes superimposing a representation of a characteristic identified by the summary data characterizing a given metadata object in proximity to the node representing the given metadata object.

Abstract: A method implemented by a data processing system for processing data items of a stream of data items, including: accessing a specification that represents the executable logic, wherein a state of the specification for a particular value of the key specifies one or more portions of the executable logic that are executable in that state; receiving, over an input device or port, data items of a stream of data; for a first one of the data items of the stream, identifying a first state of the specification for a value of the key associated with that first one of the data items; processing, by the data processing system, the first one of the data items according to one or more portions of executable logic that are represented in the specification as being associated with the first state.

Abstract: A method includes receiving information indicative of an output dataset generated by a data processing system; identifying, based on data lineage information relating to the output dataset, one or more upstream datasets on which the output dataset depends; analyzing one or more of the identified one or more upstream datasets on which the output dataset depends. The analyzing includes, for each particular upstream dataset of the one or more upstream datasets, applying one or more of: (i) a first rule indicative of an allowable deviation between a profile of the particular upstream dataset and a reference profile for the particular upstream dataset, and (ii) a second rule indicative of one or more allowable values or prohibited values for each of one or more data elements in the particular upstream dataset, and based on the results of applying the one or more rules, selecting one or more of the upstream datasets.

Abstract: A data processing system configured to perform: obtaining a first data lineage representing relationships among physical data elements, the first data lineage being generated at least in part by performing at least one of: (a) analyzing source code of at least one computer program configured to access the physical data elements; and (b) analyzing information obtained during runtime of the at least one computer program; obtaining, based on user input, a second data lineage representing relationships among business data elements; obtaining an association between at least some of the physical data elements of the first data lineage and at least some of the business data elements of the second data lineage; and generating, based on the association between the physical data elements and the business data elements, an indication of agreement or discrepancy between the first data lineage and the second data lineage.

Abstract: A data processing system for identifying one or more portions of executable logic that are executed in processing one or more data items that are each associated with a given value of a key by accessing the specification that represents the executable logic; and generating a log record for the given value of the key, the generating including: for each of the one or more of the data items associated with the given value of the key: identifying a given state of the specification that is maintained for the given value of the key associated with that data item; processing that data item including executing, on that data item, executable logic that are specified in the identified given state of the specification; and generating the log record specifying the execution of those one or more portions of the executable logic for that given value of the key.

Abstract: According to some aspects, techniques for configuring a data processing system are provided that increase flexibility and efficiency of generation of a data profile of a dataset. The data processing system may produce a value census and a validation census of the dataset in separate processing steps. The value census may then be enriched with contents of the validation census by processing the validation census in a manner that allows matching of field-value pairs of the dataset between the two censuses.

Abstract: A data processing system for identifying one or more portions of executable logic that are executed in processing one or more data items that are each associated with a given value of a key by accessing the specification that represents the executable logic; and generating a log record for the given value of the key, the generating including: for each of the one or more of the data items associated with the given value of the key: identifying a given state of the specification that is maintained for the given value of the key associated with that data item; processing that data item including executing, on that data item, executable logic that are specified in the identified given state of the specification; and generating the log record specifying the execution of those one or more portions of the executable logic for that given value of the key.

Abstract: A method is described for processing keyed data items that are each associated with a value of a key, the keyed data items being from a plurality of distinct data streams, the processing including collecting the keyed data items, determining, based on contents of at least one of the keyed data items, satisfaction of one or more specified conditions for execution of one or more actions and causing execution of at least one of the one or more actions responsive to the determining.