Abstract: Distributed processing of a data collection includes receiving information for configuring a distributed processing system. A first configuration of components is formed including sources of data elements and workers configured to process data elements, distributed among computing resources. Each data element includes a partition value that identifies a subset of the workers according to a partition rule. Data elements are accepted from the sources for a first part of the data collection in a first processing epoch and the data elements are routed through the first configuration. After accepting a first part of the data collection, change of configuration is initiated to a second configuration. A succession of two or more transitions between configurations of components is performed to a succession of modified configurations, a last of which corresponds to the second configuration. Further data elements are accepted from sources of the second configuration in a second processing epoch.

Abstract: Distributed processing of a data collection includes receiving information for configuring a distributed processing system. A first configuration of components is formed including sources of data elements and workers configured to process data elements, distributed among computing resources. Each data element includes a partition value that identifies a subset of the workers according to a partition rule. Data elements are accepted from the sources for a first part of the data collection in a first processing epoch and the data elements are routed through the first configuration. After accepting a first part of the data collection, change of configuration is initiated to a second configuration. A succession of two or more transitions between configurations of components is performed to a succession of modified configurations, a last of which corresponds to the second configuration. Further data elements are accepted from sources of the second configuration in a second processing epoch.

Abstract: Work units are received into, and processed from, holding queues. A partition, associated with a unique holding queue, is assigned to each work unit stored in a given holding queue, according to a partitioning rule. Work units are separated from each holding queue into one of two or more other queues stored in the same storage as that holding queue. A ready queue (relocation queue) is for each work unit whose assigned partition is (is not) associated with the holding queue in which it is stored. Each work unit in each relocation queue is relocated to the holding queue associated with the partition assigned to that work unit. The work units are repartitioned in response to a change in the partitioning rule to a new partitioning rule before the holding queues are empty.

Abstract: Distributed processing of a data collection includes receiving information for configuring a distributed processing system. A first configuration of components is formed including sources of data elements and workers configured to process data elements, distributed among computing resources. Each data element includes a partition value that identifies a subset of the workers according to a partition rule. Data elements are accepted from the sources for a first part of the data collection in a first processing epoch and the data elements are routed through the first configuration. After accepting a first part of the data collection, change of configuration is initiated to a second configuration. A succession of two or more transitions between configurations of components is performed to a succession of modified configurations, a last of which corresponds to the second configuration. Further data elements are accepted from sources of the second configuration in a second processing epoch.

Abstract: Work units are received into, and processed from, holding queues. A partition, associated with a unique holding queue, is assigned to each work unit stored in a given holding queue, according to a partitioning rule. Work units are separated from each holding queue into one of two or more other queues stored in the same storage as that holding queue. A ready queue (relocation queue) is for each work unit whose assigned partition is (is not) associated with the holding queue in which it is stored. Each work unit in each relocation queue is relocated to the holding queue associated with the partition assigned to that work unit. The work units are repartitioned in response to a change in the partitioning rule to a new partitioning rule before the holding queues are empty.