Abstract: A system for enhanced data pre-aggregation is provided. In one embodiment, a method is provided that includes receiving data formatted in a key/subkey format and distributing a data batch of the data to a plurality of processing threads. Each processing thread performs operations of: performing a first pass on the data batch to determine subkey rollup data; performing a second pass on the data batch to determine key rollup data; and storing the subkey rollup data and the key rollup data into data blocks. The method also includes outputting the data blocks to form a pre-aggregated data cube.

Abstract: A system comprises a computing system, an Enterprise Resource Planning (ERP) system, and a relational database management system (RDBMS). Ledger data is received onto the computing system and stored in a first data structure. Aggregated values over one or more time periods are determined and stored in the first data structure. Next, stored ledger data stored in a database is retrieved and stored in a second data structure. The first and second data structures are stored in continuous blocks of memory and are highly amenable to caching, thereby improving system performance. One or more bit arrays are generated by comparing values stored in the first and second data structures. These bit arrays indicate data change state between stored and received ledger data. Bit counting yields bit count sums that scalably identify costly database operations for received data across many accounts. Database operations are scheduled intelligently using bit count sums.