Abstract: The present application relates to a network, apparatus, and method for allocating clusters of computing nodes for programming jobs. A network includes a plurality of datacenters including computing resources configurable to instantiate nodes for executing programming jobs on a cluster. The computing resources at one of the datacenters are configured to: provision a live pool including a number of clusters, each cluster in the live pool including a plurality of nodes imaged with a configuration for executing the programming jobs in parallel on the cluster; receive a request from a user to execute a programming job; allocate a cluster from the live pool to the user for the programming job when the cluster is available; evict the cluster from the live pool; and provision a new cluster within the live pool to meet the number of clusters. The number of clusters may be optimized based on linear programming and machine-learning.

Abstract: Mechanisms to modify a dataflow execution graph that processes a data stream. An intermediate dataflow execution graph is used during modification of the dataflow execution graph from one configuration (the old dataflow execution graph) to the next (the new dataflow execution graph). Data messages of the data stream may continue to feed into the intermediate dataflow execution graph, thereby reducing latency and maintaining throughput during reconfiguration of the dataflow execution graph. Control message(s) that are structured to accomplish the reconfiguration is/are also passed into the intermediate dataflow execution graph during reconfiguration. As the control message(s) are all processed by the intermediate dataflow execution graph, the intermediate dataflow execution graph assumes the topology of the new dataflow execution graph.