Abstract: A collector for distributed data collection includes input and output queues employed for priority based queuing and dispatch of data received from endpoints and downstream collector nodes. Collection Table of Contents (CTOC) data structures for collection data are received by the collector from the endpoints or downstream collectors and are placed in the input queue, then sorted by the priority within the CTOC. Within a given priority level, collection of the data is scheduled based on the activation time window within the CTOC, which specifies the period during which the endpoint or downstream collector node will be available to service data transfer requests. The collected data, in the form of data packs and constituent data segments, is stored in persistent storage (depot). A CTOC is then transmitted to the next upstream collector node. Network bandwidth utilization is managed by adjusting the activation time window specified within a CTOC and the route employed between source and recipient.

Abstract: The “scan” phase of a distributed data collection process is decoupled from upload of the return collection data, with the “scan” consisting merely of an infrequent profile push to configure autonomous scanners at the data collection endpoints. Distributed data collection is initiated by endpoints within the distributed network, which autonomously perform a scan and transmit a Collection Table of Contents (CTOC) data structure to a nearest available collector, then await a ready message from the collector. When ready to receive the return collection data, the collector signals the endpoint, which transfer the data collection in small packets to the collector. The collector stores the received data collection in persistent storage, then initiates collection to a higher collector or recipient in substantially the same manner as the endpoint. A routing manager controls the routing of data from endpoints through one or more collectors to the recipient.