Abstract: According to some embodiments, a system, method and non-transitory computer-readable medium are provided comprising receiving a node group at an integration module, wherein the node group includes one or more requests for internal data and external data, wherein internal data is data stored in an internal datastore and external data is data stored outside of the internal data store; identifying in a configuration data, one or more meta-data nodes from the node group, wherein each meta-data node indicates a request for external data; retrieving the internal data via execution of an internal data query; determining an order of execution for the one or more meta-data nodes; executing a first meta-data node based on the determined order to generate a first result; ingesting the first result into a semantic datastore; and executing a query to generate a final result, wherein the query includes the retrieved internal data and the ingested first result. Numerous other aspects are provided.

Abstract: According to some embodiments, a system, method and non-transitory computer-readable medium are provided comprising receiving a node group at an integration module, wherein the node group includes one or more requests for internal data and external data, wherein internal data is data stored in an internal datastore and external data is data stored outside of the internal data store; identifying in a configuration data, one or more meta-data nodes from the node group, wherein each meta-data node indicates a request for external data; retrieving the internal data via execution of an internal data query; determining an order of execution for the one or more meta-data nodes; executing a first meta-data node based on the determined order to generate a first result; ingesting the first result into a semantic datastore; and executing a query to generate a final result, wherein the query includes the retrieved internal data and the ingested first result. Numerous other aspects are provided.

Abstract: A plurality of job profiles is received. Each job profile describes a job to be executed, and each job includes map tasks and reduce tasks. An execution duration for a map stage including the map tasks and an execution duration for a reduce stage including the reduce tasks of each job is estimated. The jobs are scheduled for execution based on the estimated execution duration of the map stage and the estimated execution duration of the reduce stage of each job.

Abstract: A plurality of job profiles is received. Each job profile describes a job to be executed, and each job includes map tasks and reduce tasks. An execution duration for a map stage including the map tasks and an execution duration for a reduce stage including the reduce tasks of each job is estimated. The jobs are scheduled for execution based on the estimated execution duration of the map stage and the estimated execution duration of the reduce stage of each job.