Abstract: An electronic system for increasing the speed of preparing data with a specified data quality for storage by automatically identifying for a user, with minimal user input, common contexts among (i) fields in disparate datasets, and (ii) names the user has specified as potentially describing the fields, and by using those common contexts to govern the disparate datasets prior to storage to ensure the specified data quality.

Abstract: A method for enabling a user to generate a complex aggregation on their own by providing the user with a graphical user interface that displays data items in a data catalog and that provides controls for the user to select data items to be used in generating the complex aggregation, and to select a type of aggregation, and based on the user's selections, automatically generating computer instructions to generate a value of the complex aggregation is described.

Abstract: Techniques for executing a data processing application in an environment in which there can be a plurality of data sources including continuous data sources and batch data sources. A data processing application may be representable as a plurality of input nodes and a plurality of processing nodes. The techniques include: for a node of the plurality of processing nodes having a first input configured at the time of execution of the application to receive batch data and a second input configured to receive continuous data: computing first data by executing data processing operations of the data processing application between the first input of the node and one or more data sources of the plurality of data sources on data from the one or more data sources; and storing the first data; and configuring the data processing system to, when executing the data processing application, use the stored first data as the first input to the node.

Abstract: Systems and methods are for executing, by a data processing system, a workflow to process results data indicating an output of a data quality test on data records by generating, responsive to receiving the results data and metadata describing the results data, a data quality issue associated with a state and one or more processing steps of the workflow to resolve a data quality error associated with the data quality test. Operations include generating a workflow for processing results data based a state specified by a data quality issue. Generating the workflow includes: assigning, based on the results data and the state of the data quality issue, an entity responsible for resolving the data quality error; determining, based on the metadata, one or more actions for satisfying the data quality condition specified in the data quality test; and updating the state associated with the data quality issue.

Abstract: A method implemented by data processing systems enables usage of a cloud object storage for storing data accessed based on execution of file operations that are unsupported by the cloud object storage. The method includes executing an application configured to access application data; as the application executes, generating instructions to perform operations, unsupported by cloud object storage, to modify the application data; transmitting the instructions to a first file service system representing a location for the application data to be stored in the cloud object storage; recording an indication of the operations specified by the instructions; and transmitting, to the cloud object storage, the indication of the operations specified by the instructions, whereby transmittal of the indication to cloud object storage enables a second data processing system to access the application data that is modified in accordance with the operations unsupported by cloud object storage.

Abstract: A data processing system defines a unit test for a dataflow graph comprising a plurality of executable components. The system receives an indication of a portion of a dataflow graph for testing. The system receives a parameter set indicating expected output data to be generated by execution of a component of the graph. The system receives input data for the component. The input data are indicated by the parameter set and configured to test a functionality of the component. The system defines a unit test of the component based on the parameter set. The unit test provides the input data to inputs of the dataflow graph, causes processing of the input data by the component of the dataflow graph to generate output data, and generates results data indicating a correspondence between the output data and the expected output data indicated by the parameter.

Abstract: Techniques for determining processing layouts to nodes of a dataflow graph.

Abstract: A data processing system with a dataset multiplexer that enables applications to be written to specify access to datasets as operations on logical datasets. During execution of an application by the data processing system, operations that access a dataset are implemented by accessing an entry in a dataset catalog for the logical dataset. That entry includes information to access the physical data source storing the logical dataset, including conversion of data from the format of the physical data source to the format of the logical dataset. An entry in the catalog may be created based on registration of a data source with the dataset multiplexer and may be updated automatically based on changes in storage of the dataset. This maintenance of the catalog may be partially or totally automated such that the system automatically adjusts to any changes in storage of the dataset without need for modification of any application.

Abstract: A method implemented by a data processing system for automatically detecting a change in entities of a first entity class, and, in response to the detecting, generating a notification specifying that a change relating to a second entity class has been detected, including: storing an entity model for entity classes; determining that a notification is to be generated upon detection of a change to entities of a first entity class of the entity model, the notification specifying that the detected change relates to a second entity class; detecting a change to an entity of the first entity class; responsive to the change, traversing edges of the entity model to identify a node representing an entity of the second entity class related to another node representing the entity of the first entity class; generating the notification for the entity of the second entity class; and storing the notification for transmission.

Abstract: A method for updating a computer program includes receiving a program hosted on and configured to be executed by a first computing system. The program is analyzed to obtain characterization of the program's lineage, architecture, and operation. Lineage includes relationships among elements of the program, architecture includes a characteristic of the data source, the data target, and processors configured to process the data contained in data records, and operation includes processes that are executed to process the data from the data records. A characterization is received of an update to be made to the program; when the program is modified according to the update, at least some of the modified program is configured to be hosted on and executed by a second computing system. The program is modified to implement the update to generate the modified program.

Abstract: A data processing system for discovering a semantic meaning of a field included in one or more data sets is configured to identify a field included in one or more data sets, with the field having an identifier. For that field, the system profiles data values of the field to generate a data profile, accesses a plurality of label proposal tests, and generates a set of label proposals by applying the plurality of label proposal tests to the data profile. The system determines a similarity among the label proposals and selects a classification. The system identifies one of the label proposals as identifying the semantic meaning. The system stores the identifier of the field with the identified one of the label proposals that identifies the semantic meaning.

Abstract: A method for using a metadata model to perform operations on data items, with the metadata model including parent nodes and child nodes connected by edges, with the parent nodes specifying logical metadata and the child nodes specifying physical metadata representing the data items, and with the edges specifying relationships between the nodes. The method includes: identifying a given data item and physical metadata of that given data item, accessing the metadata model, identifying, in the metadata model, a child node representing the physical metadata of the given data item, traversing one or more edges in the metadata model to identify parent nodes of the child node, determining, from logical metadata associated with the identified parent nodes, one or more operations to be performed on the given data item, applying the one or more operations to the given data item to transform the data item, and storing the transformed data item.

Abstract: Among other things, we describe a method of receiving a portion of metadata from a data source, the portion of metadata describing nodes and edges; generating instances of a data structure representing the portion of metadata, at least one instance of the data structure including an identification value that identifies a corresponding node, one or more property values representing respective properties of the corresponding node, and one or more pointers to respective identification values, each pointer representing an edge associated with a node identified by the corresponding respective identification value; storing the instances of the data structure in random access memory; receiving a query that includes an identification of at least one particular element of data; and using at least one instance of the data structure to cause a display of a computer system to display a representation of lineage of the particular element of data.

Abstract: Techniques for obtaining information about data entity instances managed by a data processing system using at least one data store. The techniques include obtaining a query comprising a first portion comprising information for identifying instances of a first data entity stored in at least one data store; and a second portion indicating at least one attribute of the first data entity; generating, from the query, a plurality of executable queries including a first set of one or more executable queries and a second set of one or more executable queries, the generating comprising: generating, using the first portion, the first set of executable queries for identifying instances of the first data entity, and generating, using the second portion, the second set of executable queries for obtaining attribute values for instances of the first data entity; and executing the plurality of executable queries to obtain results for the query.

Abstract: Techniques for converting application programs written in a source structured query language (SQL) dialect to respective computer programs embodied by dataflow graphs, the method comprising using at least one computer hardware processor to perform: obtaining a source SQL dialect (SSD) application program comprising one or more SSD scripts, the one or more SSD scripts comprising a plurality of SSD statements; translating the plurality of SSD statements into a respective plurality of target SQL dialect (TSD) statements; converting the plurality of TSD statements into a respective plurality of dataflow graphs; and merging dataflow graphs in the plurality of dataflow graphs to obtain one or more merged dataflow graphs.

Abstract: In an aspect, a method for migrating data records to a federated database system includes obtaining data records from a data source in a first federated database system; generating a data snapshot file based on the obtained data records and data indicative of a characteristic associated with the obtained data records; generating a hash of the data snapshot file to prevent modification of the data snapshot file; storing the data snapshot file and the generated hash in a data storage; migrating the obtained data records from the data snapshot file to a data target in a second federated database system, the migrating including: retrieving the data records from the data snapshot file stored in the data storage; providing the retrieved data records to the data target according to a mapping between a characteristic of the data source and a characteristic of the data target.

Abstract: A method includes accessing a schema that specifies relationships among datasets, computations on the datasets, or transformations of the datasets, selecting a dataset from among the datasets, and identifying, from the schema, other datasets that are related to the selected dataset. Attributes of the datasets are identified, and logical data representing the identified attributes and relationships among the attributes is generated. The logical data is provided to a development environment, which provides access to portions of the logical data representing the identified attributes. A specification that specifies at least one of the identified attributes in performing an operation is received from the development environment.

Abstract: A method implemented by a data processing system for enabling a system to pipeline or otherwise process data in conformance with specified criteria by providing a graphical user interface for selecting data to be processed, determining metadata of selected data, and, based on the metadata, automatically processing the selected data in conformance with the specified criteria.

Abstract: A data processing system with a dataset multiplexer that enables applications to be written to specify access to datasets as operations on logical datasets. During execution of an application by the data processing system, operations that access a dataset are implemented by accessing an entry in a dataset catalog for the logical dataset. That entry includes information to access the physical data source storing the logical dataset, including conversion of data from the format of the physical data source to the format of the logical dataset. An entry in the catalog may be created based on registration of a data source with the dataset multiplexer and may be updated automatically based on changes in storage of the dataset. This maintenance of the catalog may be partially or totally automated such that the system automatically adjusts to any changes in storage of the dataset without need for modification of any application.

Abstract: A method implemented by a data processing system for enabling a user to browse a data catalog and select fields from multiple data sources to be integrated into a data profile so that, when a request is received for the data profile, data from those fields can be made available efficiently and immediately.