Abstract: A method for tuning workflow settings in a distributed computing workflow comprising sequential interdependent jobs includes pairing a terminal stage of a first job and a leading stage of a second, sequential job to form an optimization pair, in which data segments output by the terminal stage of the first job comprises data input for the leading stage of the second job. The performance of the optimization pair is tuned by determining, with a computational processor, an estimated minimum execution time for the optimization pair and increasing the minimum execution time to generate an increased execution time. The method further includes calculating a minimum number of data segments that still permit execution of the optimization pair within the increased execution time.

Abstract: A method for tuning workflow settings in a distributed computing workflow comprising sequential interdependent jobs includes pairing a terminal stage of a first job and a leading stage of a second, sequential job to form an optimization pair, in which data segments output by the terminal stage of the first job comprises data input for the leading stage of the second job. The performance of the optimization pair is tuned by determining, with a computational processor, an estimated minimum execution time for the optimization pair and increasing the minimum execution time to generate an increased execution time. The method further includes calculating a minimum number of data segments that still permit execution of the optimization pair within the increased execution time.

Abstract: A method for tuning workflow settings in a distributed computing workflow comprising sequential interdependent jobs includes pairing a terminal stage of a first job and a leading stage of a second, sequential job to form an optimization pair, in which data segments output by the terminal stage of the first job comprises data input for the leading stage of the second job. The performance of the optimization pair is tuned by determining, with a computational processor, an estimated minimum execution time for the optimization pair and increasing the minimum execution time to generate an increased execution time. The method further includes calculating a minimum number of data segments that still permit execution of the optimization pair within the increased execution time.

Abstract: Determining a completion time of a job in a distributed network environment, the method includes determining completion times of a map task and a reduce task of a job and executing at least one test to collect a training dataset that characterizes the completion times of the map task and the reduce task.

Abstract: A method for tuning workflow settings in a distributed computing workflow comprising sequential interdependent jobs includes pairing a terminal stage of a first job and a leading stage of a second, sequential job to form an optimization pair, in which data segments output by the terminal stage of the first job comprises data input for the leading stage of the second job. The performance of the optimization pair is tuned by determining, with a computational processor, an estimated minimum execution time for the optimization pair and increasing the minimum execution time to generate an increased execution time. The method further includes calculating a minimum number of data segments that still permit execution of the optimization pair within the increased execution time.

Abstract: A performance model for a collection of jobs that make up a program is used to calculate a performance parameter based on a number of map tasks in the jobs, a number of reduce tasks in the jobs, and an allocation of resources, where the jobs include the map tasks and the reduce tasks, the map tasks producing intermediate results based on segments of input data, and the reduce tasks producing an output based on the intermediate results. The performance model considers overlap of concurrent jobs. Using a value of the performance parameter calculated by the performance model, a particular allocation of resources is determined to assign to the jobs of the program to meet a performance goal of the program.

Abstract: A performance model is used to calculate a performance parameter based on characteristics of a collection of jobs that make up a program, a number of map tasks in the jobs, a number of reduce tasks in the jobs, and an allocation of resources, where the jobs include the map tasks and the reduce tasks, the map tasks producing intermediate results based on segments of input data, and the reduce tasks producing an output based on the intermediate results. Using a value of the performance parameter calculated by the performance model, a particular allocation of resources is determined to assign to the jobs of the program to meet a performance goal of the program.