Abstract: An intelligent data partitioning engine processes instructions to monitor an input queue of a cluster computing framework processing on a distributed computing system. The intelligent data partitioning engine calculates data requirements for processing of one or more program files in the input queue and determines, based on a block size and available processing resources of a plurality of nodes of the distributed computing system, a number of data partitions. Based on the data partitions, the intelligent data partitioning engine triggers execution of the one or more program files by the cluster computing framework, where the cluster computing framework is configured based on the block size and the number of data partitions and updates the data requirements for processing of the one or more program files based on feedback from the cluster computing framework corresponding to one or more previous processing runs of the one or more program files.

Abstract: An intelligent data partitioning engine processes instructions to monitor an input queue of a cluster computing framework processing on a distributed computing system. The intelligent data partitioning engine calculates data requirements for processing of one or more program files in the input queue and determines, based on a block size and available processing resources of a plurality of nodes of the distributed computing system, a number of data partitions. Based on the data partitions, the intelligent data partitioning engine triggers execution of the one or more program files by the cluster computing framework, where the cluster computing framework is configured based on the block size and the number of data partitions and updates the data requirements for processing of the one or more program files based on feedback from the cluster computing framework corresponding to one or more previous processing runs of the one or more program files.

Abstract: A resource allocation module may be configured to monitor an input queue of a cluster computing framework for a batch of one or more programs for processing. The resource allocation module parses commands in each of the one or more programs to determine an input/output (I/O) complexity parameter and at least one operation complexity parameter corresponding to each program of the one or more program files. The resource allocation module triggers execution of the one or more program files by a cluster computing framework via a network communication, wherein the cluster computing framework is configured based on the I/O complexity parameter and the at least one operation complexity parameter. Based on analysis of feedback from the cluster computing framework, the resource allocation module modifies a calculation for determining the I/O complexity parameter and/or a calculation for determining the operation complexity of the program files.

Abstract: A resource allocation module may be configured to monitor an input queue of a cluster computing framework for a batch of one or more programs for processing. The resource allocation module parses commands in each of the one or more programs to determine an input/output (I/O) complexity parameter and at least one operation complexity parameter corresponding to each program of the one or more program files. The resource allocation module triggers execution of the one or more program files by a cluster computing framework via a network communication, wherein the cluster computing framework is configured based on the I/O complexity parameter and the at least one operation complexity parameter. Based on analysis of feedback from the cluster computing framework, the resource allocation module modifies a calculation for determining the I/O complexity parameter and/or a calculation for determining the operation complexity of the program files.