Abstract: The disclosed embodiments provide a system for monitoring the performance of periodically scheduled workflows and associated jobs while they are executing a computing cluster. During operation, the system monitors the total execution time for the workflow. While monitoring the total execution time for the workflow, the system also monitors execution times for individual jobs in the set of jobs that comprise the workflow. The system also periodically determines an execution-time threshold for the workflow based on prior executions of the workflow. If the monitored execution time for the workflow exceeds the determined execution-time threshold for the workflow, the system sends an alert about the workflow to a user. The system also enables the user to examine the monitored execution time for the workflow and the monitored execution times for the associated jobs. This helps the user to determine a solution to a performance problem for the workflow.

Abstract: The disclosed embodiments relate to a system that uses a predictive-optimization technique to facilitate distributed computation in a multi-tenant system. During operation, the system receives a job to be executed, wherein the job performs a MapReduce computation. Next, the system uses the predictive-optimization technique to determine resource-allocation parameters for the job based on associated input parameters to optimize an execution performance of the job, wherein the predictive-optimization technique uses a model that was trained running previous MapReduce jobs on the multi-tenant system. Then, the system uses the resource-allocation parameters to allocate resources for the job from the multi-tenant system. Finally, the system executes the job using the allocated resources.