Abstract: Techniques are provided for anomaly detection in time-series data using state inference and machine learning. An exemplary method comprises: obtaining detected states of a plurality of data samples in temporal data, wherein each data sample in the temporal data has a corresponding detected state; obtaining a likelihood that each of the data samples belongs to the corresponding detected state; obtaining a distribution of likelihoods of the data samples indicating a number of observations of each of a plurality of likelihood values; training, using a supervised learning technique, an anomaly detection model that, given the distribution of likelihoods and one or more anomaly thresholds, generates a quality score for each of the anomaly thresholds; and selecting at least one anomaly threshold based on the quality score, wherein the trained anomaly detection model is applied to detect anomalies in new temporal data samples using the selected at least one anomaly threshold.

Abstract: Methods and apparatus are provided for real-time ad hoc querying of data records, such as call detail records.

Abstract: Scale-out network attached storage (NAS) file systems can employ an Ingest, Transform, Store (ITS) framework for data processing. In one aspect, the ITS-NAS file systems comprise NAS nodes and high performance computing (HPC) nodes that operate under a common operating system and that are coupled to each other via a common high-bandwidth, low-latency private network infrastructure. The NAS nodes can present data to the HPC nodes as well as dispatch the execution of transform services to the HPC nodes. The ITS-NAS file systems enable massive parallelization of operations on files, for example, complex distributed operations on large files and/or simple parallel operations on large collections of small files, all within the same hardware and software architecture.

Abstract: A hot folder mechanism is employed to provide a truly integrated architecture for easy-to-use, easy-to-deploy scale-out computation and scale-out storage. A folder of an Ingest, Transform, Store (ITS)-Network attached storage (NAS) system can be configured as “hot.” The configured hot folder can then detect changes on its content, analyze such content, perform transform services on the content, and output the computation results as files on other specified output folders. In one aspect, file system nodes of the ITS-NAS can present the content to high performance computing (HPC) compute nodes of the ITS-NAS as well as to dispatch the execution of transform services to the HPC compute nodes.

Abstract: Lossless content-aware compression and decompression techniques are provided for floating point data, such as seismic data. A minimum-length compression technique exploits an association between an exponent and a length of the significand, which corresponds to the position of the least significant bit of the significand. A reduced number of bits from the significand can then be stored. A prediction method is also optionally previously applied, so that residual values with shorter lengths are compressed instead of the original values. An alignment compression technique exploits repetition patterns in the floating point numbers when they are aligned to the same exponent. Floating point numbers are then split into integral and fractional parts. The fractional part is separately encoded using a dictionary-based compression method, while the integral part is compressed using a delta-encoding method.

Abstract: Distributed content-aware compression and decompression techniques are provided for data, such as seismic data. Data is compressed by obtaining a file chunk of a plurality of file chunks of a larger file, a start offset of the file chunk, and a data descriptor indicating how the file chunk should be processed based on characteristics of the larger file. Headers in the file chunk are compressed using a substantially lossless header compression technique to obtain compressed headers. Samples in the file chunk are compressed using a substantially lossless sample compression technique to obtain compressed samples. The compressed headers and compressed samples are packed into a single bit stream comprising a compressed version of the file chunk. The compression can be performed in a physical or logical layer of storage nodes of a file system or in compute nodes of a computing cluster. The compression can be executed on demand by an external agent and/or in a background mode by a storage operating system.

Abstract: Methods and apparatus are provided for content-aware compression of data using multiple prediction functions.

Abstract: Multiple parallel prediction functions are employed for content-aware data compression. An exemplary method comprises obtaining a floating point number comprising a sign, an exponent at a given base and a significand having a length; applying a plurality of distinct prediction algorithms to the floating point number to generate a corresponding plurality of predictions; selecting a given one of the plurality of distinct prediction algorithms for the floating point number by evaluating a compression metric applied to the plurality of predictions; and encoding the floating point number by encoding the exponent and the length as a single code using a residual generated by the selected prediction algorithm. A disambiguation index optionally identifies the selected prediction algorithm among a set of prediction algorithms that potentially generated the selected prediction.

Abstract: Methods and apparatus are provided for content-aware compression of data. An exemplary method comprises obtaining a plurality of floating point numbers each comprising a sign, an exponent at a given base and a significand; grouping a plurality of exponents of the plurality of floating point numbers using a transformation that provides a transformed exponent to reduce a number of distinct exponents in the plurality of floating point numbers to be encoded; and encoding the given floating point number by encoding the transformed exponent and the length of the given floating point number as a single class code. A substantially optimal number of output class codes that need to be encoded is optionally automatically selected. A linear prediction algorithm, such as a first derivative or a second derivative, is optionally applied to the floating point numbers to generate a prediction, wherein the first or second derivative is selected based on an analysis of the data to be compressed.

Abstract: Methods and apparatus are provided for content-aware compression of data using selection from multiple weighted prediction functions. An exemplary method comprises obtaining a floating point number; applying a plurality of distinct prediction algorithms to the floating point number to generate a corresponding plurality of predictions; selecting at least one of the predictions for the floating point number based on combination weights for the plurality of distinct prediction algorithms, wherein the combination weights are based only on prior samples of the floating point numbers; and encoding the floating point number by encoding, as a single code, the exponent and the length of a residual generated by the distinct prediction algorithm associated with the selected at least one prediction. The combination weights are optionally updated for subsequent floating point numbers.

Abstract: Data compression with window-based selection from multiple prediction functions is provided. A predefined default predictor and a plurality of other predictors are applied to a floating point number to generate a plurality of predictions. A compression metric over a collection of floating point numbers is evaluated for the default predictor and the plurality of other predictors. Based on the compression metric, (i) the floating point number is encoded using the predefined default predictor, or (ii) the collection of floating point numbers is encoded using one of the other predictors. Stored indexes indicate which predictor was used for the encoding. A set of predictors out of a larger set of predictors can be determined for a specific data set based on a performance-based ranking. The default predictor and the alternate predictors can be represented as ensembles of predictors.

Abstract: Lossless content-aware compression and decompression techniques are provided for floating point data, such as seismic data. A minimum-length compression technique exploits an association between an exponent and a length of the significand, which corresponds to the position of the least significant bit of the significand. A reduced number of bits from the significand can then be stored. A prediction method is also optionally previously applied, so that residual values with shorter lengths are compressed instead of the original values. An alignment compression technique exploits repetition patterns in the floating point numbers when they are aligned to the same exponent. Floating point numbers are then split into integral and fractional parts. The fractional part is separately encoded using a dictionary-based compression method, while the integral part is compressed using a delta-encoding method.

Abstract: Methods and apparatus are provided for random access to compressed data using bitwise indices, enabling interaction with compressed data as if interaction were with an uncompressed version thereof. A compressed file is decompressed using an index table comprising a bitwise mapping between individual bits in the compressed file and corresponding portions of an uncompressed version of the compressed file; and decompressing at least a portion of the compressed file using the index table. Different data types within a file are optionally managed by separate index tables. A block-based file system can process index tables to provide transparent access to the compressed file. The index tables support dynamic index granularities, without decompressing the compressed file and recompressing it. The decompressed portion of the compressed file is optionally stored in a cache, possibly with neighbor portions pre-fetched using the index tables.