Abstract: A dataflow controller technology. A dataflow execution graph (such as a directed acyclic graph) receives or is to receive one or more data streams for processing. The control of the dataflow execution graph is performed by a set of one or more dataflow controllers that are dedicated to that dataflow execution graph. Each dataflow execution graph is instantiated and assigned to the dataflow execution graph such that the assignment results in the dataflow controller executing dataflow control functions for that dataflow execution graph. By dedicating controller(s) to only one dataflow execution graph, the risk of failure of the controller is bound to a single dataflow execution graph.

Abstract: At least some embodiments described herein relate to the automatic tuning of a dataflow execution graph. Such dataflow execution graphs are often used to execute some processing against a stream of data messages. A performance parameter of the dataflow execution graph is monitored, and compared against a service level objective. Based on the comparison, it is automatically decided whether a configuration of the dataflow execution graph should be changed. If a change is decided to be made, the configuration of the dataflow execution graph is altered. Thus, rather than require explicit instructions to change the configuration of a dataflow execution graph, the configuration of a dataflow execution graph is changed (or tuned) depending on compliance of performance with a service level objective.

Abstract: A dataflow controller technology. A dataflow execution graph (such as a directed acyclic graph) receives or is to receive one or more data streams for processing. The control of the dataflow execution graph is performed by a set of one or more dataflow controllers that are dedicated to that dataflow execution graph. Each dataflow execution graph is instantiated and assigned to the dataflow execution graph such that the assignment results in the dataflow controller executing dataflow control functions for that dataflow execution graph. By dedicating controller(s) to only one dataflow execution graph, the risk of failure of the controller is bound to a single dataflow execution graph.

Abstract: At least some embodiments described herein relate to the automatic tuning of a dataflow execution graph. Such dataflow execution graphs are often used to execute some processing against a stream of data messages. A performance parameter of the dataflow execution graph is monitored, and compared against a service level objective. Based on the comparison, it is automatically decided whether a configuration of the dataflow execution graph should be changed. If a change is decided to be made, the configuration of the dataflow execution graph is altered. Thus, rather than require explicit instructions to change the configuration of a dataflow execution graph, the configuration of a dataflow execution graph is changed (or tuned) depending on compliance of performance with a service level objective.

Abstract: Mechanisms to modify a dataflow execution graph that processes a data stream. An intermediate dataflow execution graph is used during modification of the dataflow execution graph from one configuration (the old dataflow execution graph) to the next (the new dataflow execution graph). Data messages of the data stream may continue to feed into the intermediate dataflow execution graph, thereby reducing latency and maintaining throughput during reconfiguration of the dataflow execution graph. Control message(s) that are structured to accomplish the reconfiguration is/are also passed into the intermediate dataflow execution graph during reconfiguration. As the control message(s) are all processed by the intermediate dataflow execution graph, the intermediate dataflow execution graph assumes the topology of the new dataflow execution graph.