Abstract: A system for aggregating dataflow lineage information is disclosed. The system receives one or more input data elements and determines a dataflow path for the one or more input data elements. The dataflow path includes at least a data storage node and a computation node. Then, the system identifies a lineage control value associated with the data storage node and a version control value associated with the computation node. The system generates an output lineage for the one or more input data elements by appending the lineage control value to the version control value.

Abstract: A system for performing an integrated data quality control is disclosed. The system determines a dataflow path for one or more input data elements. The, the system performs a lineage control check, a timeliness control check, and a variation control check on the dataflow path. If the dataflow path integrated scoring of the three controls is sufficient, the system determines that data related to the dataflow path is fit for use. If the dataflow path fails any one of the three checks, the system determines that data related to the dataflow path is not fit for use.

Abstract: A system for performing a timeliness control is disclosed. The system identifies a dataflow path for performing timeliness control and identifies a first network node and a second network node of the dataflow path for determining a latency between the first and the second network node. The system determines an output lineage corresponding to the dataflow path and identifies, from the output lineage, a first control value associated with the first network node and a second control value associated with the second network node. Then, the system extracts a first timestamp from the first control value and a second timestamp from the second control value and determines the latency based on the first timestamp and the second timestamp. Although the intranode latency is described herein with respect to a first and second nodes, the intra-node latency can be determined for up to n nodes using the techniques described herein.

Abstract: A system for validating a dataflow graph is disclosed. The system receives an input dataflow graph and compares the input dataflow graph to each of a plurality of pre-learned dataflow graphs. Each of the pre-learned dataflow graphs is associated with an occurrence rate. Then, the system identifies a pre-learned dataflow graph that matches the input dataflow graph and identifies an occurrence rate associated with the pre-learned dataflow graph. The system compares the occurrence rate with a pre-defined threshold for validating the input dataflow graph. If the occurrence rate exceeds the pre-defined threshold, the system determines that the input dataflow graph is a valid dataflow graph.

Abstract: A system for aggregating dataflow lineage information is disclosed. The system receives one or more input data elements and determines a dataflow path for the one or more input data elements. The dataflow path includes at least a data storage node and a computation node. Then, the system identifies a lineage control value associated with the data storage node and a version control value associated with the computation node. The system generates an output lineage for the one or more input data elements by appending the lineage control value to the version control value.

Abstract: A system for performing an integrated data quality control is disclosed. The system determines a dataflow path for one or more input data elements. The, the system performs a lineage control check, a timeliness control check, and a variation control check on the dataflow path. If the dataflow path integrated scoring of the three controls is sufficient, the system determines that data related to the dataflow path is fit for use. If the dataflow path fails any one of the three checks, the system determines that data related to the dataflow path is not fit for use.

Abstract: A system for aggregating dataflow lineage information is disclosed. The system receives one or more input data elements and determines a dataflow path for the one or more input data elements. The dataflow path includes at least a data storage node and a computation node. Then, the system identifies a lineage control value associated with the data storage node and a version control value associated with the computation node. The system generates an output lineage for the one or more input data elements by appending the lineage control value to the version control value.

Abstract: A system for performing a timeliness control is disclosed. The system identifies a dataflow path for performing timeliness control and identifies a first network node and a second network node of the dataflow path for determining a latency between the first and the second network node. The system determines an output lineage corresponding to the dataflow path and identifies, from the output lineage, a first control value associated with the first network node and a second control value associated with the second network node. Then, the system extracts a first timestamp from the first control value and a second timestamp from the second control value and determines the latency based on the first timestamp and the second timestamp. Although the intranode latency is described herein with respect to a first and second nodes, the intra-node latency can be determined for up to n nodes using the techniques described herein.

Abstract: A system for performing an integrated data quality control is disclosed. The system determines a dataflow path for one or more input data elements. The, the system performs a lineage control check, a timeliness control check, and a variation control check on the dataflow path. If the dataflow path integrated scoring of the three controls is sufficient, the system determines that data related to the dataflow path is fit for use. If the dataflow path fails any one of the three checks, the system determines that data related to the dataflow path is not fit for use.

Abstract: A system for performing a variation control in a data network is disclosed. The system generates a distribution graph for a set of node values associated with a network node. The system determines a statistical confidence level for the distribution graph and determines a statistical confidence interval for the distribution graph based on the statistical confidence level. Then, the system receives a new node value associated with the network node and compares the new node value to the statistical confidence interval. If the new node value falls within the statistical confidence interval, the system determines that determining that the new node value is normal.

Abstract: A system for validating a dataflow graph is disclosed. The system receives an input dataflow graph and compares the input dataflow graph to each of a plurality of pre-learned dataflow graphs. Each of the pre-learned dataflow graphs is associated with an occurrence rate. Then, the system identifies a pre-learned dataflow graph that matches the input dataflow graph and identifies an occurrence rate associated with the pre-learned dataflow graph. The system compares the occurrence rate with a pre-defined threshold for validating the input dataflow graph. If the occurrence rate exceeds the pre-defined threshold, the system determines that the input dataflow graph is a valid dataflow graph.

Abstract: A system for performing an integrated data quality control is disclosed. The system determines a dataflow path for one or more input data elements. The, the system performs a lineage control check, a timeliness control check, and a variation control check on the dataflow path. If the dataflow path integrated scoring of the three controls is sufficient, the system determines that data related to the dataflow path is fit for use. If the dataflow path fails any one of the three checks, the system determines that data related to the dataflow path is not fit for use.

Abstract: A system for aggregating dataflow lineage information is disclosed. The system receives one or more input data elements and determines a dataflow path for the one or more input data elements. The dataflow path includes at least a data storage node and a computation node. Then, the system identifies a lineage control value associated with the data storage node and a version control value associated with the computation node. The system generates an output lineage for the one or more input data elements by appending the lineage control value to the version control value.

Abstract: A system for performing a variation control in a data network is disclosed. The system generates a distribution graph for a set of node values associated with a network node. The system determines a statistical confidence level for the distribution graph and determines a statistical confidence interval for the distribution graph based on the statistical confidence level. Then, the system receives a new node value associated with the network node and compares the new node value to the statistical confidence interval. If the new node value falls within the statistical confidence interval, the system determines that determining that the new node value is normal.