Abstract: Methods and apparatus are provided for evaluating combinatorial processes using simulation techniques and multiple parallel statistical analyses of real-world data. A simulation model is generated that simulates one or more steps of a combinatorial process. The simulation model comprises key features of the combinatorial process. A plurality of first data mining tasks are performed in parallel over real data of the combinatorial process to obtain key feature prediction models that estimate the key features. The key feature prediction models are bound to the simulation model. Query types to be supported are identified and a plurality of simulation runs are generated in parallel, comprising simulated data for the supported query types. A plurality of second data mining tasks are performed in parallel over the plurality of simulation runs to build global prediction models to answer queries of each supported query type. An answer to a user query is determined using the global prediction models.

Abstract: Methods and apparatus are provided for process monitoring based on large-scale combinations of time series data. An exemplary method comprises generating a model from time series data for a given target time series; determining whether a first difference between measured values and predicted values based on the model exceeds a predefined threshold indicating a target prediction error; in response to a detected target prediction error, performing evaluations of (i) a neighborhood coherence comprising an average of variables of the model weighted by corresponding coefficients on a predefined neighborhood time window, and/or (ii) a second difference between a given value of at least one variable in the model and an average value of the at least one variable based on a training dataset; providing notifications when first predefined criteria based on the evaluations are satisfied; and updating the model when second predefined criteria based on the evaluations are satisfied.

Abstract: Product planning techniques are provided that recommend compositions of product features for weighted heterogeneous consumer segments using regression trees. An exemplary method comprises obtaining historical consumer data comprising product preferences for existing product items for multiple consumer segments; obtaining product features indicating characteristics for each existing product item; prioritizing the consumer segments by obtaining a weight indicating an interest in each consumer segment; computing a total performance metric, for each product item, by calculating a dot product between the consumer segment weights and respective preferences of the consumer segments regarding a given product item; obtaining a regression tree from the existing product items to predict the total performance metric in terms of corresponding product features; and selecting a combination of the product features to be used in future product items based on identified paths in the regression tree.

Abstract: Techniques are provided for simulation-based online workflow optimization.

Abstract: Methods and apparatus are provided for taking snapshots to train prediction models and improve workflow execution.

Abstract: Techniques are provided for dataflow execution time estimation for distributed processing frameworks. An exemplary method comprises: obtaining an input dataset for a dataflow for execution; determining a substantially minimal data unit for a given operation of the dataflow processed by the given operation; estimating a number of rounds required to execute a number of data units in the input dataset using nodes assigned to execute the given operation; determining an execution time spent by the given operation to process one data unit; estimating the execution time for the given operation based on the execution time spent by the given operation to process one data unit and the number of rounds required to execute the number of data units in the input dataset; and executing the given operation with the input dataset. A persistent cost model is optionally employed to record the execution times of known dataflow operations.

Abstract: Simulation data is summarized and queried. A user provides an indication of simulation data that will be subsequently queried. The queried simulation data comprises (i) a set of key attributes, (ii) a set of events, and/or (iii) a set of causality relationships between a plurality of the events. First level summaries summarize simulation executions of scenarios of a combinatorial process and comprise (i) a summary of the frequency distribution of key attribute values, (ii) a timestamp for each event, and (iii) an indication of causality between events observed during the simulation. Second level summaries summarize executions of the given scenario and comprise (i) a consolidated distribution probability for the key attributes, (ii) a frequency distribution of occurrences of the events in a single execution, and (iii) a frequency of observations of the causality between pairs of events.

Abstract: Methods and apparatus are provided for data operation pre-processing with a probabilistic estimation of operation value. An exemplary method comprises extracting feature values from a data set; identifying a set of operations that previously processed data sets comprising the extracted feature values; determining whether to execute an operation from the set of operations before an explicit request for the execution of the operation based on a probabilistic evaluation of a value of pre-processing the operation; and executing a set of instructions for the operation when it is determined that the operation is to be executed. The set of instructions comprises, e.g., stopping execution of other operations being executed, freeing resources required by the operation, and/or allocating resources required by the operation.

Abstract: Techniques are provided for dataflow execution time estimation for distributed processing frameworks. An exemplary method comprises: obtaining an input dataset for a dataflow for execution; determining a substantially minimal data unit for a given operation of the dataflow processed by the given operation; estimating a number of rounds required to execute a number of data units in the input dataset using nodes assigned to execute the given operation; determining an execution time spent by the given operation to process one data unit; estimating the execution time for the given operation based on the execution time spent by the given operation to process one data unit and the number of rounds required to execute the number of data units in the input dataset; and executing the given operation with the input dataset. A persistent cost model is optionally employed to record the execution times of known dataflow operations.

Abstract: Techniques are provided for automatic combination of sub-process simulation results with dataset selection based on a fitness under one or more specific scenarios. An exemplary method comprises obtaining an execution map for each sub-process in a sequence that stores results of a given sub-process execution. The results comprise a scenario, a distribution and a distribution fitness value. In response to a user query regarding a target feature and an initial dataset, initial dataset are combined with results selected from the execution map for a first sub-process in the sequence; each available dataset from the previous sub-processes in the sequence is combined with results selected from the execution map for the next sub-process; a probability distribution function (pdf) for the target feature is composed from a combined dataset that represents a simulation of the process and combines results of each of sub-process in the sequence; and the pdf is processed to answer the user query for the target feature.

Abstract: Methods and apparatus are provided for automatic combination of sub-process simulation results and heterogeneous data sources. An exemplary method comprises obtaining, for a process comprised of a sequence of a plurality of sub-processes, an identification of relevant input and output features for each sub-process; obtaining an execution map for each sub-process, wherein each execution map stores results of an execution of a given sub-process; and, in response to a user query regarding a target feature and a user-provided initial scenario: composing a probability distribution function for the target feature representing a simulation of the process based on a sequence of the execution maps, by matching input features of each execution map with features from the initial scenario or the output of previous execution maps; and processing the probability distribution function to answer the user query.

Abstract: Pattern queries are evaluated in parallel over large N-dimensional datasets to identify features of interest.

Abstract: Methods and apparatus are provided for performing massively parallel processing (MPP) large-scale combinations of time series data.

Abstract: Techniques are provided for automatic placement of cache operations in a dataflow. An exemplary method obtains a graph representation of a dataflow of operations; determines a number of executions and a computational cost of the operations, and a computational cost of a caching operation to cache a dataset generated by an operation; establishes a dataflow state structure recording values for properties of the dataflow operations for a number of variations of caching various dataflow operations; determines a cache gain factor for dataflow operations as an estimated reduction in the accumulated cost of the dataflow by caching an output dataset of a given operation; determines changes in the dataflow state structure by caching an output dataset of a different operation in the dataflow; and searches the dataflow state structures to determine the output datasets to cache based on a total dataflow execution cost.

Abstract: Lossless content-aware compression and decompression techniques are provided for floating point data, such as seismic data. A minimum-length compression technique exploits an association between an exponent and a length of the significand, which corresponds to the position of the least significant bit of the significand. A reduced number of bits from the significand can then be stored. A prediction method is also optionally previously applied, so that residual values with shorter lengths are compressed instead of the original values. An alignment compression technique exploits repetition patterns in the floating point numbers when they are aligned to the same exponent. Floating point numbers are then split into integral and fractional parts. The fractional part is separately encoded using a dictionary-based compression method, while the integral part is compressed using a delta-encoding method.

Abstract: Distributed content-aware compression and decompression techniques are provided for data, such as seismic data. Data is compressed by obtaining a file chunk of a plurality of file chunks of a larger file, a start offset of the file chunk, and a data descriptor indicating how the file chunk should be processed based on characteristics of the larger file. Headers in the file chunk are compressed using a substantially lossless header compression technique to obtain compressed headers. Samples in the file chunk are compressed using a substantially lossless sample compression technique to obtain compressed samples. The compressed headers and compressed samples are packed into a single bit stream comprising a compressed version of the file chunk. The compression can be performed in a physical or logical layer of storage nodes of a file system or in compute nodes of a computing cluster. The compression can be executed on demand by an external agent and/or in a background mode by a storage operating system.

Abstract: Methods and apparatus are provided for content-aware compression of data using multiple prediction functions.

Abstract: Methods and apparatus are provided for automatically identifying possible faults in large seismic datasets.

Abstract: Multiple parallel prediction functions are employed for content-aware data compression. An exemplary method comprises obtaining a floating point number comprising a sign, an exponent at a given base and a significand having a length; applying a plurality of distinct prediction algorithms to the floating point number to generate a corresponding plurality of predictions; selecting a given one of the plurality of distinct prediction algorithms for the floating point number by evaluating a compression metric applied to the plurality of predictions; and encoding the floating point number by encoding the exponent and the length as a single code using a residual generated by the selected prediction algorithm. A disambiguation index optionally identifies the selected prediction algorithm among a set of prediction algorithms that potentially generated the selected prediction.

Abstract: Methods and apparatus are provided for content-aware compression of data. An exemplary method comprises obtaining a plurality of floating point numbers each comprising a sign, an exponent at a given base and a significand; grouping a plurality of exponents of the plurality of floating point numbers using a transformation that provides a transformed exponent to reduce a number of distinct exponents in the plurality of floating point numbers to be encoded; and encoding the given floating point number by encoding the transformed exponent and the length of the given floating point number as a single class code. A substantially optimal number of output class codes that need to be encoded is optionally automatically selected. A linear prediction algorithm, such as a first derivative or a second derivative, is optionally applied to the floating point numbers to generate a prediction, wherein the first or second derivative is selected based on an analysis of the data to be compressed.