Abstract: A method of comprises managing versions of a cell of a collection of cells in a first database and metadata entries being associated with one or more computing nodes through distributed storage with each indicating a timestamp of a write transaction leading to a new version in a second database; selecting a current timestamp for a current sweep; dynamically determining a number of threads to run simultaneously on each computing node to perform deletion for the current sweep based on a performance update related to a computing node or a resource availability update related to the computing node; and causing the number of threads to run simultaneously on each computing node, a first thread running on a specific computing node performing: identifying, a first set of cells with versions written prior to the current timestamp; deleting corresponding metadata entries having timestamps prior to the current timestamp in one database transaction.

Abstract: A system for scaling parallelism in sweeping stale data associated with multi-version data management is disclosed. The system is programmed to manage a targeted sweep of stale data stored on distributed storage associated with a set of computing nodes, where the stale data includes metadata for database transactions. The system is programmed to dynamically determine, on a computing node, how many threads to simultaneously perform deletion for the targeted sweep based on historical performance of the computing nodes, historical performance of the distributed storage associated with the computing nodes, or other factors. The system is programmed to then cause a certain number of threads to run on the computing node, where each thread identifies and deletes specific metadata for data transactions within the scope of the targeted sweep.