Abstract: A system, program, and method for anomaly detection in heterogeneous logs. The system having a processor configured to identify pattern fields comprised of a plurality of event identifiers. The processor is further configured to generate an automata model by profiling event behaviors of the plurality of event sequences, the plurality of event sequences grouped in the automata model by combinations of one or more pattern fields and one or more event identifiers from among the plurality of event identifiers, wherein for a given combination, the one or more event identifiers therein must be respectively comprised in a same one of the one or more pattern fields with which it is combined. The processor is additionally configured to detect an anomaly in one of the plurality of event sequences using the automata model. The processor is also configured to control an anomaly-initiating one of the network devices based on the anomaly.

Abstract: A computer-implemented method provides an early warning of an impending failure in a monitored system. The method includes performing, by a processor, an offline model learning process that generates a model of expected log rates in the monitored system from historical log data. The model represents a normal behavior of the monitored system. The method further includes performing an online detection process that detects the impending failure in the monitored system prior to an actual occurrence thereof based on (i) the model of expected log rates and (ii) observed log rates. The method also includes displaying, by a display device based on (i) the model of expected log rates and (ii) observed log rates in the monitored system, information relating to the impending failure prior to the actual occurrence of the impending failure. The online detection process identifies short term and long term failures and long term failures.

Abstract: A computer-implemented method for real-time detecting of abnormal network connections is presented. The computer-implemented method includes collecting network connection events from at least one agent connected to a network, recording, via a topology graph, normal states of network connections among hosts in the network, and recording, via a port graph, relationships established between host and destination ports of all network connections.

Abstract: Methods and systems for intrusion detection include determining a causality trace for a flagged event. Determining the causality trace includes identifying a hot process that generates bursts of events with interleaved dependencies, aggregating events related to the hot process according to a process-centric dependency approximation that ignores dependencies between the events related to the hot process, and tracking causality in a reduced event stream that comprises the aggregated events. It is determined whether an intrusion has occurred based on the causality trace. One or more mitigation actions is performed if it is determined that an intrusion has occurred.

Abstract: Methods and systems for dependency tracking include identifying a hot process that generates bursts of events with interleaved dependencies. Events related to the hot process are aggregated according to a process-centric dependency approximation that ignores dependencies between the events related to the hot process. Causality in a reduced event stream that comprises the aggregated events is tracked.

Abstract: Systems and methods are disclosed for parsing logs from arbitrary or unknown systems or applications by capturing heterogeneous logs from the arbitrary or unknown systems or applications; generating one pattern for every unique log message; building a pattern hierarchy tree by grouping patterns based on similarity metrics, and for every group it generates one pattern by combing all constituting patterns of that group; and selecting a set of patterns from the pattern hierarchy tree.

Abstract: An exemplary method for detecting one or more anomalies in a system includes building a temporal causality graph describing functional relationship among local components in normal period; applying the causality graph as a propagation template to predict a system status by iteratively applying current system event signatures; and detecting the one or more anomalies of the system by examining related patterns on the template causality graph that specifies normal system behaviors. The system can aligning event patterns on the causality graph to determine an anomaly score.

Abstract: A computer implemented method for network monitoring includes providing network packet event characterization and analysis for network monitoring that includes supporting summarization and characterization of network packet traces collected across multiple processing elements of different types in a virtual network, including a trace slicing to organize individual packet events into path-based trace slices, a trace characterization to extract at least 2 types of feature matrix describing those trace slices, and a trace analysis to cluster, rank and query packet traces based on metrics of the feature matrix.

Abstract: Systems and methods for network management, including adaptively installing one or more monitoring rules in one or more network devices on a network using an intelligent network middleware, detecting application traffic on the network transparently using an application demand monitor, and predicting future network demands of the network by analyzing historical and current demands. The one or more monitoring rules are updated once counters are collected; and network paths are determined and optimized to meet network demands and maximize utilization and application performance with minimal congestion on the network.

Abstract: Methods and systems for intrusion attack recovery include monitoring two or more hosts in a network to generate audit logs of system events. One or more dependency graphs (DGraphs) is generated based on the audit logs. A relevancy score for each edge of the DGraphs is determined. Irrelevant events from the DGraphs are pruned to generate a condensed backtracking graph. An origin is located by backtracking from an attack detection point in the condensed backtracking graph.

Abstract: Methods and systems for finding a packet's routing path in a network includes intercepting control messages sent by a controller to one or more switches in a software defined network (SDN). A state of the SDN at a requested time is emulated and one or more possible routing paths through the emulated SDN is identified by replaying the intercepted control messages to one or more emulated switches in the emulated SDN. The one or more possible routing paths correspond to a requested packet injected into the SDN at the requested time.

Abstract: A method is provided for root cause anomaly detection in an invariant network having a plurality of nodes that generate time series data. The method includes modeling anomaly propagation in the network. The method includes reconstructing broken invariant links in an invariant graph based on causal anomaly ranking vectors. Each broken invariant link involves a respective node pair formed from the plurality of nodes such that one of the nodes in the respective node pair has an anomaly. Each causal anomaly ranking vector is for indicating a respective node anomaly status for a given one of the plurality of nodes when paired. The method includes calculating a sparse penalty of the casual anomaly ranking vectors to obtain a set of time-dependent anomaly rankings. The method includes performing temporal smoothing of the set of rankings, and controlling an anomaly-initiating one of the plurality of nodes based on the set of rankings.

Abstract: Systems and a method are provided. A system includes a Temporal Behavior Query Language (TBQL) server having a processor and a memory operably coupled to the processor. The TBQL server configured to construct a TBQL query using a grammar inference technique based on syntactic sugar to expedite query construction. The TBQL server is further configured to execute the TBQL query to generate TBQL query results.

Abstract: Systems and methods are disclosed for detecting error in a cloud infrastructure by running a plurality of training tasks on the cloud infrastructure and generating training execution logs; generating a model miner with the training execution logs to represent one or more correct task executions in the cloud infrastructure; after training, running a plurality of tasks on the cloud infrastructure and capturing live execution logs; and from the live execution logs, if a current task deviates from the correct task execution, indicating an execution error for correction in real-time.

Abstract: Methods and systems for detecting anomalous network activity include determining whether a network event exists within an existing topology graph and port graph. A connection probability for the network event is determined if the network does not exist within the existing topology graph and port graph. The network event is identified as abnormal if the connection probability is below a threshold.

Abstract: Systems and methods are disclosed for handling log data from one or more applications, sensors or instruments by receiving heterogeneous logs from arbitrary/unknown systems or applications; generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom; generating models and profiles from training logs based on different conditions and updating a global model database storing all models generated over time; tokenizing raw log messages from one or more applications, sensors or instruments running a production system; transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and generating an anomaly alert from the one or more applications, sensors or instruments running a production system.

Abstract: A method and system are provided. The method includes extracting shapelets from each of a plurality of time series dimensions of multi-dimensional time series data. The method further includes building a plurality of decision-tree classifiers, one for each time series dimension, responsive to the shapelets extracted therefrom. The method also includes generating a pairwise similarity matrix between respective different ones of the plurality of time series dimensions using the shapelets as intermediaries for determining similarity. The method additionally includes applying a feature selection technique to the matrix to determine respective feature weights for each of shapelet features of the shapelets and respective classifier weights for each of the decision-tree classifiers that uses the shapelet features.

Abstract: In a software defined network having switches including first and last switches and intermediate switches, wherein a default routing path exists between the first and last switches, a system and method are provided for computing path latency. The method includes inserting a respective monitoring rule(s) in each switch, mandating for each switch, forwarding a received rule matching packet to a next switch, and further mandating for the first switch and the last switch, sending a PacketIn message to a controller. The method includes inserting, in each switch, a respective monitoring probe(s) matching the respective monitoring rule(s) in a same switch to initiate mandates specified by the respective monitoring rule(s) in the same switch responsive to an arrival of the packet thereat. The method includes time-stamping the PacketIn messages to generate PacketIn timestamps, aggregating the PacketIn timestamps, and estimating the path latency from an aggregation of PacketIn timestamps.

Abstract: Systems and methods are disclosed for analyzing logs generated by a machine by analyzing a log and identifying one or more abstract landmark delimiters (ALDs) representing delimiters for log tokenization; from the log and ALDs, tokenizing the log and generating an increasingly tokenized format by separating the patterns with the ALD to form an intermediate tokenized log; iteratively repeating the tokenizing of the logs until a last intermediate tokenized log is processed as a final tokenized log; and applying the tokenized logs in applications.

Abstract: Systems and methods are disclosed for detecting periodic event behaviors from machine generated logging by: capturing heterogeneous log messages, each log message including a time stamp and text content with one or more fields; recognizing log formats from log messages; transforming the text content into a set of time series data, one time series for each log format; during a training phase, analyzing the set of time series data and building a category model for each periodic event type in heterogeneous logs; and during live operation, applying the category model to a stream of time series data from live heterogeneous log messages and generating a flag on a time series data point violating the category model and generating an alarm report for the corresponding log message.