Abstract: A system, apparatus, and methods are provided for self-healing and balancing partition distribution across nodes within a message broker cluster. During operation, the system receives a stream of messages at the message brokering cluster, wherein the message stream is divided into partitions and replicas for each partition are distributed among a set of nodes within the message brokering cluster. Responsive to a change in the number of nodes within the message brokering cluster, the system (1) determines a set of replicas to be migrated within the message brokering cluster, (2) divides the set of replicas into multiple chunks, wherein each chunk includes one or more of the replicas to be migrated to a new node, and (3) migrates the set of replicas one chunk at a time, wherein replicas not corresponding to the single chunk do not begin migrating until all replicas within the single chunk finish migrating.

Abstract: A system, apparatus, and methods are provided for balancing partition distribution across nodes within a message broker cluster so as to balance the broker nodes' workloads. During operation, the system receives a stream of messages at the cluster, wherein the message stream is divided into topics, the topics are divided into partitions, and replicas for each partition are distributed among the nodes of the message brokering cluster. Upon detection of an imbalance in the nodes' workloads by a monitor (e.g., as indicated by uneven resource consumption), an analyzer considers various possible remedies (e.g., reassigning/demoting/promoting a replica), estimates their likely impacts on the workload, and determines whether they satisfy hard and/or soft goals of the system. The analyzer generates a plan that satisfies the hard goals and that may satisfy some or all soft goals, and passes it to an executor for implementation.

Abstract: A multi-tenancy storage node is provided. The storage node hosts partitions of multiple databases accessed by multiple applications. Each database is governed by a corresponding service-level agreement (SLA) or policy that specifies a maximum load or level of operation of the database in terms of one or more metrics (e.g., number or frequency of reads/writes, maximum size of reads/writes). To determine whether another database can be hosted by the node, a snapshot of the node's database operations is replayed on a test node, along with operations that exercise the other database. If maximum thresholds of the node for the metrics are not exceeded, the other database can be added to the storage node. An SLA is generated for it automatically, based on the metrics it exhibited during the operations on the test node. The storage node may enforce the SLA by rate-limiting activity of one or more applications.

Abstract: A multi-tenancy storage node is provided. The storage node hosts partitions of multiple databases accessed by multiple applications. Each database is governed by a corresponding service-level agreement (SLA) or policy that specifies a maximum load or level of operation of the database in terms of one or more metrics (e.g., number or frequency of reads/writes, maximum size of reads/writes). To determine whether another database can be hosted by the node, a snapshot of the node's database operations is replayed on a test node, along with operations that exercise the other database. If maximum thresholds of the node for the metrics are not exceeded, the other database can be added to the storage node. An SLA is generated for it automatically, based on the metrics it exhibited during the operations on the test node. The storage node may enforce the SLA by rate-limiting activity of one or more applications.

Abstract: A multi-tenancy storage node is provided. The storage node hosts partitions of multiple databases accessed by multiple applications. Each database is governed by a corresponding service-level agreement (SLA) or policy that specifies a maximum load or level of operation of the database in terms of one or more metrics (e.g., number or frequency of reads/writes, maximum size of reads/writes). To determine whether another database can be hosted by the node, a snapshot of the node's database operations is replayed on a test node, along with operations that exercise the other database. If maximum thresholds of the node for the metrics are not exceeded, the other database can be added to the storage node. An SLA is generated for it automatically, based on the metrics it exhibited during the operations on the test node. The storage node may enforce the SLA by rate-limiting activity of one or more applications.