Abstract: A system, apparatus, and methods are provided for self-healing and balancing partition distribution across nodes within a message broker cluster. During operation, the system receives a stream of messages at the message brokering cluster, wherein the message stream is divided into partitions and replicas for each partition are distributed among a set of nodes within the message brokering cluster. Responsive to a change in the number of nodes within the message brokering cluster, the system (1) determines a set of replicas to be migrated within the message brokering cluster, (2) divides the set of replicas into multiple chunks, wherein each chunk includes one or more of the replicas to be migrated to a new node, and (3) migrates the set of replicas one chunk at a time, wherein replicas not corresponding to the single chunk do not begin migrating until all replicas within the single chunk finish migrating.

Abstract: A system, apparatus, and methods are provided for balancing partition distribution across nodes within a message broker cluster so as to balance the broker nodes' workloads. During operation, the system receives a stream of messages at the cluster, wherein the message stream is divided into topics, the topics are divided into partitions, and replicas for each partition are distributed among the nodes of the message brokering cluster. Upon detection of an imbalance in the nodes' workloads by a monitor (e.g., as indicated by uneven resource consumption), an analyzer considers various possible remedies (e.g., reassigning/demoting/promoting a replica), estimates their likely impacts on the workload, and determines whether they satisfy hard and/or soft goals of the system. The analyzer generates a plan that satisfies the hard goals and that may satisfy some or all soft goals, and passes it to an executor for implementation.

Abstract: A system, apparatus, and methods are provided that support the passing of oversized messages within a publish-subscribe messaging system. During operation, the system subscribes to a message stream brokered by a message brokering cluster that imposes a maximum message size. Responsive to receiving a given message of the message stream, the system (1) determines an offset to commit to the message brokering cluster, wherein the determined offset is based on whether one or more segments of one or more oversized messages are buffered, and (2) commits the determined offset to the message brokering cluster. Responsive to recovering from a crash, wherein the offset at which to resume the receipt of messages from the message stream is unknown, the system then retrieves the determined offset from the message brokering cluster and resumes the receipt of messages from the message brokering cluster based on the determined offset.

Abstract: A system, method, and apparatus are provided for repartitioning a topic of a publish-subscribe message system. The topic is originally configured with N partitions (N>1) hosted by multiple brokers for storing messages to be consumed by multiple consumers. The repartitioning process causes one or more collections of partitions to be created in addition to the original collection of N partitions. Afterward, when a new message is received for the topic and has an associated partition key, the key is processed (e.g., hashed) once to identify a target collection of partitions and is then reprocessed to identify a destination partition within the target collection. Consumers may be automatically subscribed to new partitions. For example, a consumer subscribed to the ith partition of the original N partitions may be subscribed to the ith partition of each additional collection of partitions.

Abstract: A system, apparatus, and methods are provided that support the passing of oversized messages within a publish-subscribe messaging system. During operation, the system subscribes to a message stream brokered by a message brokering cluster that imposes a maximum message size. Responsive to receiving a given message of the message stream, the system (1) determines an offset to commit to the message brokering cluster, wherein the determined offset is based on whether one or more segments of one or more oversized messages are buffered, and (2) commits the determined offset to the message brokering cluster. Responsive to recovering from a crash, wherein the offset at which to resume the receipt of messages from the message stream is unknown, the system then retrieves the determined offset from the message brokering cluster and resumes the receipt of messages from the message brokering cluster based on the determined offset.