Abstract: The disclosed embodiments provide a system for managing a data store. During operation, the system stores a set of pending updates to a data store in a registry. Next, the system executes an asynchronous process that applies a first subset of updates from the registry as writes to records in the data store without blocking processing of read queries of the data store. Upon completing a write by the asynchronous process at a second portion of the data store, the system updates the registry with an indication of the completed write at the second portion of the data store. During processing of a read query of the data store, the system applies a second subset of updates from the registry to a result of the read query. Finally, the system returns the result in a response to the read query.

Abstract: In order to ingest data from an arbitrary source in a set of sources, a computer system accesses predefined configuration instructions. Then, the computer system generates a dynamic data-ingestion pipeline that is compatible with a Hadoop file system based on the predefined configuration instructions. This dynamic data-ingestion pipeline includes a modular arrangement of operators from a set of operators that includes: an extraction operator for extracting the data of interest from the source, a converter operator for transforming the data, and a quality-checker operator for checking the transformed data. Moreover, the computer system receives the data from the source. Next, the computer system processes the data using the dynamic data-ingestion pipeline as the data is received without storing the data in memory for the purpose of subsequent ingestion processing.

Abstract: Computer systems and methods for estimating the number of client computers actively coupled to a server computer system in real-time are discussed herein. Specifically, systems and methods are discussed for a server computer system receiving heartbeat messages from a plurality of client computers and generating an estimate of the number of client computers actively coupled to the server computer system in real-time without locks, such as a database table lock. A heartbeat message from a client computer need not include a client or user identifier. In an embodiment, the memory footprint/overhead is O(l), and may be a single whole number greater than zero, such as a 64-bit unsigned integer. Systems and methods are also discussed herein to calculate and reduce the expected error of the estimated number of active clients.

Abstract: In order to ingest data from an arbitrary source in a set of sources, a computer system accesses predefined configuration instructions. Then, the computer system generates a dynamic data-ingestion pipeline that is compatible with a Hadoop file system based on the predefined configuration instructions. This dynamic data-ingestion pipeline includes a modular arrangement of operators from a set of operators that includes: an extraction operator for extracting the data of interest from the source, a converter operator for transforming the data, and a quality-checker operator for checking the transformed data. Moreover, the computer system receives the data from the source. Next, the computer system processes the data using the dynamic data-ingestion pipeline as the data is received without storing the data in memory for the purpose of subsequent ingestion processing.

Abstract: Computer systems and methods for estimating the number of client computers actively coupled to a server computer system in real-time are discussed herein. Specifically, systems and methods are discussed for a server computer system receiving heartbeat messages from a plurality of client computers and generating an estimate of the number of client computers actively coupled to the server computer system in real-time without locks, such as a database table lock. A heartbeat message from a client computer need not include a client or user identifier. In an embodiment, the memory footprint/overhead is O(l), and may be a single whole number greater than zero, such as a 64-bit unsigned integer. Systems and methods are also discussed herein to calculate and reduce the expected error of the estimated number of active clients.