Abstract: Efficient distributed indexing of clumpy data is disclosed. For example, a plurality of data stores store a plurality of data objects, where a plurality of data references to each of the plurality of data objects is recorded on a distributed index and an index manager executes a processor to temporarily partition a namespace of the plurality of data references into a plurality of temporary partitions. A respective count of data references in each temporary partition is computed. A trend line approximating a distribution of data references is calculated based on the respective counts. A cumulative representation of the distribution of the respective counts is calculated based on the trend line. The namespace is repartitioned based on the cumulative representation resulting in a repartitioned namespace, where the distributed index is divided into index segments based on the repartitioned namespace.

Abstract: Dynamic distributed work allocation is disclosed. For example, a first work server (WS) stores a first plurality of tasks and a second WS stores a second plurality of tasks. A work client (WC) is configured to send a first lock request (LR) with a first priority value (PV) to the first WS and a second LR with a second PV to the second WS. The WC receives a first lock notice (LN) and a first task from the first WS, and a second LN and a second task from the second WS. Prior to a first lock duration (LD) expiring and completing processing of the first task, the WC sends a third LR to the first WS that extends the first LD. After completing the second task, the WC sends a lock release notice and a fourth LR to the second WS.

Abstract: Dynamic distributed work allocation is disclosed. For example, a first work server (WS) stores a first plurality of tasks and a second WS stores a second plurality of tasks. A work client (WC) is configured to send a first lock request (LR) with a first priority value (PV) to the first WS and a second LR with a second PV to the second WS. The WC receives a first lock notice (LN) and a first task from the first WS, and a second LN and a second task from the second WS. Prior to a first lock duration (LD) expiring and completing processing of the first task, the WC sends a third LR to the first WS that extends the first LD. After completing the second task, the WC sends a lock release notice and a fourth LR to the second WS.

Abstract: Efficient distributed indexing of clumpy data is disclosed. For example, a plurality of data stores store a plurality of data objects, where a plurality of data references to each of the plurality of data objects is recorded on a distributed index and an index manager executes a processor to temporarily partition a namespace of the plurality of data references into a plurality of temporary partitions. A respective count of data references in each temporary partition is computed. A trend line approximating a distribution of data references is calculated based on the respective counts. A cumulative representation of the distribution of the respective counts is calculated based on the trend line. The namespace is repartitioned based on the cumulative representation resulting in a repartitioned namespace, where the distributed index is divided into index segments based on the repartitioned namespace.