Abstract: In some implementations, a monitoring device may receive configuration information associated with an extract, transform, load (ETL) pipeline that includes one or more data sources and one or more data sinks. The monitoring device may generate, based on the configuration information, lineage data related to a data flow from the one or more data sources to the one or more data sinks in the ETL pipeline. The monitoring device may generate one or more predicted quality metrics associated with the ETL pipeline using a machine learning model. The monitoring device may generate a visualization in which multiple nodes are arranged to indicate the data flow from the one or more data sources to the one or more data sinks and further in which the one or more predicted quality metrics are encoded within the visualization.

Abstract: In some implementations, a device may obtain the relational statement associated with a query of a database. The device may parse the relational statement to identify one or more temporary queries included in the relational statement. The device may modify the relational statement to include segment indicators for respective segments of the relational statement associated with respective temporary queries from the one or more temporary queries. The device may execute the relational statement. The device may determine, based on executing the relational statement and based on the segment indicators, one or more execution performance parameters for each segment included in the relational statement, wherein the one or more execution performance parameters indicate performance levels of the one or more temporary queries. The device may store the one or more execution performance parameters for each segment.

Abstract: In some implementations, a monitoring device may receive configuration information associated with an extract, transform, load (ETL) pipeline that includes one or more data sources and one or more data sinks. The monitoring device may generate, based on the configuration information, lineage data related to a data flow from the one or more data sources to the one or more data sinks in the ETL pipeline. The monitoring device may generate one or more predicted quality metrics associated with the ETL pipeline using a machine learning model. The monitoring device may generate a visualization in which multiple nodes are arranged to indicate the data flow from the one or more data sources to the one or more data sinks and further in which the one or more predicted quality metrics are encoded within the visualization.

Abstract: In some implementations, a monitoring device may receive configuration information associated with an extract, transform, load (ETL) pipeline that includes one or more data sources and one or more data sinks. The monitoring device may generate, based on the configuration information, lineage data related to a data flow from the one or more data sources to the one or more data sinks in the ETL pipeline. The monitoring device may generate one or more predicted quality metrics associated with the ETL pipeline using a machine learning model. The monitoring device may generate a visualization in which multiple nodes are arranged to indicate the data flow from the one or more data sources to the one or more data sinks and further in which the one or more predicted quality metrics are encoded within the visualization.

Abstract: Disclosed herein are system, apparatus, article of manufacture, method, and/or computer program product embodiments for predictive scheduling and execution of data analytics applications based on machine learning techniques. An apparatus may operate by determining a first prediction entry in a predicted execution schedule based at least on a current timestamp. The apparatus may also operate by determining that a first confidence score of the first prediction entry is greater than or equal to a confidence score threshold and determining that an execution prediction of the first prediction entry is greater than or equal to an execution threshold. The apparatus may further operate by transmitting a first data analytics application execution request configured to request a first instance of execution of the data analytics application.

Abstract: In some implementations, a monitoring device may receive configuration information associated with an extract, transform, load (ETL) pipeline that includes one or more data sources and one or more data sinks. The monitoring device may generate, based on the configuration information, lineage data related to a data flow from the one or more data sources to the one or more data sinks in the ETL pipeline. The monitoring device may generate one or more predicted quality metrics associated with the ETL pipeline using a machine learning model. The monitoring device may generate a visualization in which multiple nodes are arranged to indicate the data flow from the one or more data sources to the one or more data sinks and further in which the one or more predicted quality metrics are encoded within the visualization.

Abstract: Disclosed herein are system, apparatus, article of manufacture, method, and/or computer program product embodiments for predictive scheduling and execution of data analytics applications based on machine learning techniques. An apparatus may operate by determining a first prediction entry in a predicted execution schedule based at least on a current timestamp. The apparatus may also operate by determining that a first confidence score of the first prediction entry is greater than or equal to a confidence score threshold and determining that an execution prediction of the first prediction entry is greater than or equal to an execution threshold. The apparatus may further operate by transmitting a first data analytics application execution request configured to request a first instance of execution of the data analytics application.