Abstract: A generic mechanism to persist data objects into databases without a fundamental dependence on the underlying data store is provided. Rather than relying on the expertise of a programmer to know what the structure of a database should look like, data types are defined and adorned by the programmer with attributes that suggest what the corresponding data will be used for, and without the programmer defining the structure of the database that will be used to store the data. Thereafter, the database is dynamically created to satisfy the needs suggested by the adorned attributes. In particular, a number of different tables are created according to the intended needs for accessing the data. By doing this an optimized database can be created to provide the desired results without requiring the programmer to have any specific knowledge regarding databases and corresponding database schemas.

Abstract: Described herein is a data collection and warehousing system that collects log files from a plurality of servers, and that transforms the entries of the log files prior to using them to populate a database. Nightly, the system provides a pre-processing program to each of the servers. The servers executing the pre-processing component to filter the logged data before sending it on to the data collection and warehousing system. The data collection and warehousing system pre-parses each received log file to create dimension and fact tables. During this process, strings are replaced by keys. These keys are calculated on-the-fly by taking a hash of each string. The resulting tables are then further parsed to create further dimension and fact tables that correspond to each of the dimension and fact tables of a relational database schema. After further processing, these tables are loaded into a data warehousing system.