Abstract: In response to receiving a parallel processing job from a customer, a system operated by a computing resource service provider allocates and configures a cluster of computer systems capable of executing the job. In an embodiment, each computer system is configured with a first network stack that allows access to resources of the computing resource service provider and a second network stack that allows access to resources of the customer. In an embodiment, the state of the cluster is monitored by the system via the first network stack. In an embodiment, the system deploys a set of tasks on the cluster for fulfilling the processing job. In an embodiment, the tasks have access to the second network stack so that each task can access resources of the customer.

Abstract: A system operated by a computing resource service provider maintains a pool of computing resources for use in processing job requests submitted by customers. The system allocates computing resources to the pool in anticipation of future resource needs. In various embodiments, future resource needs can be estimated based on scheduled jobs, or historical job information. In an embodiment, the computing resources are virtual computer systems which may be arranged in a cluster. In response to receiving a parallel processing job from a customer, the system reserves computing resources from the pool for performing the job. In an embodiment, the reserved resources are configured with a network namespace that is able to access to a customer's resources.

Abstract: Portions of distributed data processing programs may be dynamically allocated computing resources. When execution of a distributed data processing program is requested, interdependent portions of the resource may be dynamically computing resources according to requirements determined for the portions. A resource manager may, in some embodiments, respond to individual allocation requests for the different portions received from a program manager for the distributed data processing program. The allocated resources may be obtained from pools of computing resources that can be utilized by different distributed data processing programs.