Abstract: A computer-implemented method for automatically analyzing log contents received via a network and detecting content-level anomalies is presented. The computer-implemented method includes building a statistical model based on contents of a set of training logs and detecting, based on the set of training logs, content-level anomalies for a set of testing logs. The method further includes maintaining an index and metadata, generating attributes for fields, editing model capability to incorporate user domain knowledge, detecting anomalies using field attributes, and improving anomaly quality by using user feedback.

Abstract: A computer-implemented method, computer program product, and computer processing system are provided. The method includes preprocessing, by a processor, a set of heterogeneous logs by splitting each of the logs into tokens to obtain preprocessed logs. Each of the logs in the set is associated with a timestamp and textual content in one or more fields. The method further includes generating, by the processor, a set of regular expressions from the preprocessed logs. The method also includes performing, by the processor, an unsupervised parsing operation by applying the regular expressions to the preprocessed logs to obtain a set of parsed logs and a set of unparsed logs, if any. The method additionally includes storing, by the processor, the set of parsed logs in a log analytics database and the set of unparsed logs in a debugging database.

Abstract: A method is provided that includes transforming training data into a neural network based learning model using a set of temporal graphs derived from the training data. The method includes performing model learning on the learning model by automatically adjusting learning model parameters based on the set of the temporal graphs to minimize differences between a predetermined ground-truth ranking list and a learning model output ranking list. The method includes transforming testing data into a neural network based inference model using another set of temporal graphs derived from the testing data. The method includes performing model inference by applying the inference and learning models to test data to extract context features for alerts in the test data and calculate a ranking list for the alerts based on the extracted context features. Top-ranked alerts are identified as critical alerts. Each alert represents an anomaly in the test data.

Abstract: Systems and methods are disclosed for processing a stream of logged data by: creating one or more models from a set of training logs during a training phase; receiving testing data in real-time and generating anomalies using the models created during the training phase; updating the one or more models during real-time processing of a live stream of logs; and detecting a log anomaly from the live stream of logs.

Abstract: Systems and methods are disclosed for handling log data from one or more applications, sensors or instruments by receiving heterogeneous logs from arbitrary/unknown systems or applications; generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom; generating models and profiles from training logs based on different conditions and updating a global model database storing all models generated over time; tokenizing raw log messages from one or more applications, sensors or instruments running a production system; transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and generating an anomaly alert from the one or more applications, sensors or instruments running a production system.

Abstract: A heterogeneous log pattern editing recommendation system and computer-implemented method are provided. The system has a processor configured to identify, from heterogeneous logs, patterns including variable fields and constant fields. The processor is also configured to extract a category feature, a cardinality feature, and a before-after n-gram feature by tokenizing the variable fields in the identified patterns. The processor is additionally configured to generate target similarity scores between target fields to be potentially edited and other fields from among the variable fields in the heterogeneous logs using pattern editing operations based on the extracted category feature, the extracted cardinality feature, and the extracted before-after n-gram feature. The processor is further configured to recommend, to a user, log pattern edits for at least one of the target fields based on the target similarity scores between the target fields in the heterogeneous logs.

Abstract: Methods and systems for log management include pre-processing heterogeneous logs and performing a log management action on the pre-processed plurality of heterogeneous logs. Pre-processing the logs includes performing a fixed tokenization of the heterogeneous logs based on a predefined set of symbols, performing a flexible tokenization of the heterogeneous logs based on a user-defined set of rules, converting timestamps in the heterogeneous logs to a single target timestamp format, and performing structural log tokenization of the heterogeneous logs based on user-defined structural information.

Abstract: Systems and methods are disclosed for parsing logs from arbitrary or unknown systems or applications by capturing heterogeneous logs from the arbitrary or unknown systems or applications; generating one pattern for every unique log message; building a pattern hierarchy tree by grouping patterns based on similarity metrics, and for every group it generates one pattern by combing all constituting patterns of that group; and selecting a set of patterns from the pattern hierarchy tree.

Abstract: Systems and methods are disclosed for handling log data from one or more applications, sensors or instruments by receiving heterogeneous logs from arbitrary/unknown systems or applications; generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom; generating models and profiles from training logs based on different conditions and updating a global model database storing all models generated over time; tokenizing raw log messages from one or more applications, sensors or instruments running a production system; transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and generating an anomaly alert from the one or more applications, sensors or instruments running a production system.

Abstract: The subject disclosure is directed towards a data deduplication technology in which a hash index service's index maintains a hash index in a secondary storage device such as a hard drive, along with a compact index table and look-ahead cache in RAM that operate to reduce the I/O to access the secondary storage device during deduplication operations. Also described is a session cache for maintaining data during a deduplication session, and encoding of a read-only compact index table for efficiency.

Abstract: The subject disclosure is directed towards a data deduplication technology in which a hash index service's index maintains a hash index in a secondary storage device such as a hard drive, along with a compact index table and look-ahead cache in RAM that operate to reduce the I/O to access the secondary storage device during deduplication operations. Also described is a session cache for maintaining data during a deduplication session, and encoding of a read-only compact index table for efficiency.

Abstract: The subject disclosure is directed towards a data deduplication technology in which a hash index service's index maintains a hash index in a secondary storage device such as a hard drive, along with a compact index table and look-ahead cache in RAM that operate to reduce the I/O to access the secondary storage device during deduplication operations. Also described is a session cache for maintaining data during a deduplication session, and encoding of a read-only compact index table for efficiency.

Abstract: The subject disclosure is directed towards a data deduplication technology in which a hash index service's index maintains a hash index in a secondary storage device such as a hard drive, along with a compact index table and look-ahead cache in RAM that operate to reduce the I/O to access the secondary storage device during deduplication operations. Also described is a session cache for maintaining data during a deduplication session, and encoding of a read-only compact index table for efficiency.