Abstract: A method, a computing system, and a non-transitory machine readable storage medium containing instructions for managing a stream processing topology are provided. In an example, the method includes receiving a first topology that communicatively couples a plurality of processing elements via a first arrangement of interconnections to perform an operation on a stream of data. A second topology is defined that communicatively couples the plurality of processing elements via a second arrangement of interconnections that is different from the first arrangement. The second topology assigns the plurality of processing elements a first set of operations. The second topology is provided to a stream processing manager and is modified during processing of the stream of data by assigning a second set of operations to the plurality of processing elements that is different from the first set of operations.

Abstract: A method, a computing system, and a non-transitory machine readable storage medium containing instructions for managing a stream processing topology are provided. In an example, the method includes receiving a first topology that communicatively couples a plurality of processing elements via a first arrangement of interconnections to perform an operation on a stream of data. A second topology is defined that communicatively couples the plurality of processing elements via a second arrangement of interconnections that is different from the first arrangement. The second topology assigns the plurality of processing elements a first set of operations. The second topology is provided to a stream processing manager and is modified during processing of the stream of data by assigning a second set of operations to the plurality of processing elements that is different from the first set of operations.

Abstract: Some examples include a plurality of nodes configured to execute map-reduce jobs by enabling tasks to share processing slots with other tasks. As one example, a job tracker may compare a task profile for a received task with one or more task profiles for one or more respective tasks already assigned for execution on the processing slots of one or more worker nodes. Based at least in part on the comparing, the job tracker may select a particular one of already assigned tasks to be executed concurrently with the received task on a slot. In addition, the job tracker may determine one or more expected future tasks based at least in part on one or more ongoing workflows of map-reduce jobs. The selection of the already assigned task to be executed concurrently with the received task may also be based in part on the expected future tasks.

Abstract: Some examples include a pipeline manager that creates pipeline queues across data nodes in a cluster. The pipeline manager may assign a pipeline queue connection to contiguous map-reduce jobs so that a reduce task of a first map-reduce job sends data to the pipeline queue, and a map task of a second map-reduce job receives the data from the pipeline queue. Thus, the map task of the second job may begin using the data from the reduce task of the first job prior to completion of the first job. Furthermore, in some examples, the data nodes in the cluster may monitor access success to individual pipeline queues. If the access attempts to access successes exceeds a threshold, a data node may request an additional pipeline queue connection for a task. Additionally, if a failure occurs, information maintained at a data node may be used by the pipeline manager for recovery.