Abstract: An example embodiment may involve a main database; a main memory; and one or more processors configured to: retrieve, by a data collector application, records from the main database, wherein the data collector application includes an embedded database; aggregate, by the data collector application, values in the records relating to a key performance indicator (KPI) to form partial KPI data stored in one or more blocks of the main memory; determine, by the data collector application, that utilization of the main memory exceeds a pre-defined threshold; in response to the utilization of the main memory exceeding the pre-defined threshold, write, by the data collector application, the partial KPI data to a row of the embedded database; and release, by the data collector application, the one or more blocks of the main memory used to store the partial KPI data.

Abstract: A system and method for a secure and scalable data collection and ingestion data pipeline is discussed. The data pipeline provides secure ingestion of data into internet-of-things (IoT) platform, while providing high throughput, replication, and fault-tolerance, which makes it suitable for large scale systems where data can be received from millions or billions of devices, sensors, or clients. In addition, the data pipeline provides a platform for real time event generation and data analysis. Further the data pipeline reduces operational costs and platform-as-a-service (PAAS) vendor dependencies.

Abstract: A system and method for a secure and scalable data collection and ingestion data pipeline is discussed. The data pipeline provides secure ingestion of data into internet-of-things (IoT) platform, while providing high throughput, replication, and fault-tolerance, which makes it suitable for large scale systems where data can be received from millions or billions of devices, sensors, or clients. In addition, the data pipeline provides a platform for real time event generation and data analysis. Further the data pipeline reduces operational costs and platform-as-a-service (PAAS) vendor dependencies.