Abstract: Data processing can be managed in a distributed computing environment (DCE). In one example, the DCE can receive a sequence of computing operations to be consecutively executed in the DCE. For each computing operation in the sequence, the DCE can receive input data for the computing operation, partition the input data into subsets, and determine whether the computing operation is linear or non-linear. The DCE can then apply different processing techniques to the subsets depending on whether the computing operation is linear or non-linear.

Abstract: A compressed data structure is disclosed for storing collected data and delivering ad-hoc data analysis. The compressed data structure can reduce hardware requirements, such as data storage requirements, and decrease processing requirements during ad-hoc data analysis. Raw data can be pre-aggregated to a base level of aggregation (base cell level) that is determined by the unique combination of attributes available for aggregation. Rather than storing the raw data, the entire distribution is stored at the base cell level along with a selection of pre-computed statistics of the raw data. At the time of ad-hoc analysis, pre-computed results based on the raw data can be retrieved or computed from the pre-computed statistics, or analysis of higher-level data can be computed on-demand based on the pre-aggregated base level data.

Abstract: A compressed data structure is disclosed for storing collected data and delivering ad-hoc data analysis. The compressed data structure can reduce hardware requirements, such as data storage requirements, and decrease processing requirements during ad-hoc data analysis. Raw data can be pre-aggregated to a base level of aggregation (base cell level) that is determined by the unique combination of attributes available for aggregation. Rather than storing the raw data, the entire distribution is stored at the base cell level along with a selection of pre-computed statistics of the raw data. At the time of ad-hoc analysis, pre-computed results based on the raw data can be retrieved or computed from the pre-computed statistics, or analysis of higher-level data can be computed on-demand based on the pre-aggregated base level data.

Abstract: Streaming data, such as streaming records transmitted from entities, can be managed. For example, a new record associated with an entity can be received. There can be an existing record for the entity within a group of records. The group of records can form a block. A new block for the new record can be generated. A datastore can be updated to indicate that the new block has the most current record for the entity. Entries in the datastore can be filtered to identify a subgroup of blocks that has the most current record for each entity of multiple entities. A combined group of blocks can be generated by joining the new block with the subgroup of blocks. The combined group of blocks can be processed as a batch of data by a processing engine.

Abstract: Streaming data, such as streaming records transmitted from entities, can be managed. For example, a new record associated with an entity can be received. There can be an existing record for the entity within a group of records. The group of records can form a block. A new block for the new record can be generated. A datastore can be updated to indicate that the new block has the most current record for the entity. Entries in the datastore can be filtered to identify a subgroup of blocks that has the most current record for each entity of multiple entities. A combined group of blocks can be generated by joining the new block with the subgroup of blocks. The combined group of blocks can be processed as a batch of data by a processing engine.

Abstract: Real-time and historical data can be managed. For example, a processing engine can be selectively operated in a historical mode. In the historical mode, the processing engine can analyze data associated with a previous date. For example, the processing device can access a database that includes multiple entries. Each entry in the database can be associated with a respective record and including (i) an entity associated with the respective record, (ii) an identifier of a block of records that comprises the respective record, and (iii) a timestamp indicating when the block of records was generated. The processing engine can filter entries in the database to determine blocks of records that include a most current record for each entity as of the previous date. The processing engine can process the blocks of records to determine an output corresponding to the previous date.