Abstract: A data set subscription tracking and termination system may include a distribute module, a publisher and a plurality of subscribers. The distribute module may receive a publication registration to register a publication. The distribute module may receive a subscription registration to register a subscription to the publication. The publication registration and/or subscription registration may include metadata relating to the publication and/or subscription. A metadata store, included in the distribute module, may store the publication registration and/or the subscription registration. The publisher may change the publication. The change to the publication may include adding and/or deleting rows and/or columns to, or from, the publication. The publisher may notify the distribute module of publication. The distribute module may transmit an alert to all subscribers notifying them of the publication changes.

Abstract: A system includes an execution engine and a processor. The execution engine receives application code that, when executed, is configured to generate a second data element from a first data element that is stored in a first database and store the second element in a second database. The execution engine converts the code into an execution plan and executes it. The execution plan includes a first operation to obtain the first element from the first database, a second operation to apply a transformation to the first element, and a third operation to store the first element as the second element in the second database. The processor accesses the execution plan and determines that the third operation includes storing the second element. In response, the processor generates a data lineage for the second element by extracting, from each operation from the third to the first, a portion of the data lineage.

Abstract: A data set subscription tracking and termination system may include a distribute module, a publisher and a plurality of subscribers. The distribute module may receive a publication registration to register a publication. The distribute module may receive a subscription registration to register a subscription to the publication. The publication registration and/or subscription registration may include metadata relating to the publication and/or subscription. A metadata store, included in the distribute module, may store the publication registration and/or the subscription registration. The publisher may change the publication. The change to the publication may include adding and/or deleting rows and/or columns to, or from, the publication. The publisher may notify the distribute module of publication. The distribute module may transmit an alert to all subscribers notifying them of the publication changes.

Abstract: A data set subscription tracking and termination system may include a distribute module, a publisher and a plurality of subscribers. The distribute module may receive a publication registration to register a publication. The distribute module may receive a subscription registration to register a subscription to the publication. The publication registration and/or subscription registration may include metadata relating to the publication and/or subscription. A metadata store, included in the distribute module, may store the publication registration and/or the subscription registration. The publisher may change the publication. The change to the publication may include adding and/or deleting rows and/or columns to, or from, the publication. The publisher may notify the distribute module of publication. The distribute module may transmit an alert to all subscribers notifying them of the publication changes.

Abstract: A system includes an execution engine and a processor. The execution engine receives application code that, when executed, is configured to generate a second data element from a first data element that is stored in a first database and store the second element in a second database. The execution engine converts the code into an execution plan and executes it. The execution plan includes a first operation to obtain the first element from the first database, a second operation to apply a transformation to the first element, and a third operation to store the first element as the second element in the second database. The processor accesses the execution plan and determines that the third operation includes storing the second element. In response, the processor generates a data lineage for the second element by extracting, from each operation from the third to the first, a portion of the data lineage.

Abstract: A trihybrid data movement, data governance and data provenance system may include a distribute module, a publisher and a plurality of subscribers. The distribute module may include a user interface. The user interface may receive a publication registration from the publisher. The publication registration may register a publication. The user interface may receive a subscription registration from each of the plurality of subscribers. The subscription registration may subscribe to the publication. The publication registration and/or subscription registration may include metadata relating to the publication and/or subscription. A metadata store, included in the distribute module, may store the publication registration and/or the subscription registration. The publication may be published by transmitting a unique identifier from the publisher to an application programming interface within the distribute module.

Abstract: A data set subscription tracking and termination system may include a distribute module, a publisher and a plurality of subscribers. The distribute module may receive a publication registration to register a publication. The distribute module may receive a subscription registration to register a subscription to the publication. The publication registration and/or subscription registration may include metadata relating to the publication and/or subscription. A metadata store, included in the distribute module, may store the publication registration and/or the subscription registration. The publisher may change the publication. The change to the publication may include adding and/or deleting rows and/or columns to, or from, the publication. The publisher may notify the distribute module of publication. The distribute module may transmit an alert to all subscribers notifying them of the publication changes.

Abstract: A trihybrid data movement, data governance and data provenance system may include a distribute module, a publisher and a plurality of subscribers. The distribute module may include a user interface. The user interface may receive a publication registration from the publisher. The publication registration may register a publication. The user interface may receive a subscription registration from each of the plurality of subscribers. The subscription registration may subscribe to the publication. The publication registration and/or subscription registration may include metadata relating to the publication and/or subscription. A metadata store, included in the distribute module, may store the publication registration and/or the subscription registration. The publication may be published by transmitting a unique identifier from the publisher to an application programming interface within the distribute module.

Abstract: Systems for dynamically transforming data are provided. Database data may be received and ingested into a system. Ingesting the data may include executing one or more first data governance functions, such as data quality evaluation functions, data controls, and the like. The ingested data may then be output for further processing as first processed data and first data governance information may be captured and stored. The first processed data may be processed to execute one or more data transformations. Data transformations may include calculations, formatting, derivations, and the like. In some arrangements, second data governance functions may be executed on the transformed data. The transformed data may then be output as second processed data. The system may capture second data governance information as the data is transformed. The second processed data may then be published to one or more downstream databases for use in one or more applications executed by an entity.

Abstract: Systems for dynamically transforming data are provided. Database data may be received and ingested into a system. Ingesting the data may include executing one or more first data governance functions, such as data quality evaluation functions, data controls, and the like. The ingested data may then be output for further processing as first processed data and first data governance information may be captured and stored. The first processed data may be processed to execute one or more data transformations. Data transformations may include calculations, formatting, derivations, and the like. In some arrangements, second data governance functions may be executed on the transformed data. The transformed data may then be output as second processed data. The system may capture second data governance information as the data is transformed. The second processed data may then be published to one or more downstream databases for use in one or more applications executed by an entity.

Abstract: A user interface for interfacing with a trihybrid data movement, data governance and data provenance system is provided. The user interface may include an application selection drop-down field that enables selection of an application. The user interface may include a deliveries tab. When selected, the deliveries tab may display the status of user subscription deliveries. The user interface may include a resources tab. When selected, the resources tab may display publications and/or subscriptions that have been set up. When selected, the resources tab may enable a user to create a new publication. The user interface may include an approvals tab. When selected, the approvals tab may display approvals relating to subscriptions that have been requested. The user interface may include a data catalog tab. When selected, the data catalog tab may display a plurality of available publications, and enable a user to request a subscription to a publication.

Abstract: Systems for providing dynamic lineage validation are provided. A system may receive mapping data from a computing device. The mapping data may be formatted used to generate one or more libraries. In some examples, the libraries may be stored and used in development of future processes. In some examples, one or more controls related to the process, mapping data, and the like, may be received. The system may execute the controls to determine whether data violates the one or more of the controls. If so, the system may prevent the associated process from executing. If the data does not violate one or more controls, the system may cause the process to execute. After executing the process, run time events may be collected. The run time events may be linked to the mapping data and may be used to validate data quality, data lineage, data accuracy, and the like.

Abstract: Systems for providing dynamic lineage validation are provided. A system may receive mapping data from a computing device. The mapping data may be formatted used to generate one or more libraries. In some examples, the libraries may be stored and used in development of future processes. In some examples, one or more controls related to the process, mapping data, and the like, may be received. The system may execute the controls to determine whether data violates the one or more of the controls. If so, the system may prevent the associated process from executing. If the data does not violate one or more controls, the system may cause the process to execute. After executing the process, run time events may be collected. The run time events may be linked to the mapping data and may be used to validate data quality, data lineage, data accuracy, and the like.