Abstract: Systems and methods are described for processing ingested data, detecting anomalies in the ingested data, and providing explanations of a possible cause of the detected anomalies as the data is being ingested. For example, a token or field in the ingested data may have an anomalous value. Tokens or fields from another portion of the ingested data can be extracted and analyzed to determine whether there is any correlation between the values of the extracted tokens or fields and the anomalous token or field having an anomalous value. If a correlation is detected, this information can be surfaced to a user.

Abstract: Systems and methods are described for processing ingested data, detecting anomalies in the ingested data, and providing explanations of a possible cause of the detected anomalies as the data is being ingested. For example, a token or field in the ingested data may have an anomalous value. Tokens or fields from another portion of the ingested data can be extracted and analyzed to determine whether there is any correlation between the values of the extracted tokens or fields and the anomalous token or field having an anomalous value. If a correlation is detected, this information can be surfaced to a user.

Abstract: A computerized method is disclosed that includes accessing domain name server (DNS) record data including a plurality of DNS records spanning a first time period, performing a time-to-live (TTL) analysis to determine a TTL run length distribution for the DNS record data, wherein the TTL analysis includes: generating a vector of the TTL values of each DNS record ordered sequentially in time, parsing the vector of the TTL values into segments, where a segment consists of one or more TTL values where a current TTL value is less than an immediately preceding TTL value, and determining the TTL run length distribution, determining whether DNS beaconing is present based on a result of the TTL analysis and in response to determining that DNS beaconing is present, generating an alert for a system administrator.

Abstract: Systems and methods are described for training an artificial intelligence model to extract one or more data fields from a log. For example, the artificial intelligence model may be a neural network. The neural network may be trained using training data obtained by iterating through a plurality of logs using active learning, and selecting a subset of the logs in the plurality to be labeled by a user. For example, the selected subset of logs may be logs that are not similar to other logs already labeled by a user. The user may be prompted to label the selected subset of logs to identify one or more data fields to extract. Once the selected subset of logs are labeled, these labeled logs can be used as the training data to train the neural network.

Abstract: Systems and methods are described for training an artificial intelligence model to infer a log sourcetype of a log. For example, logs may have different log sourcetypes, and logs having the same log sourcetypes may have different messagetypes. The artificial intelligence model may be a machine learning model, and can be trained using training data that includes logs with known log sourcetypes. Each log can be tokenized, filtered, converted into a vector, and applied to a machine learning model as an input to perform the training. The machine learning model may output an inferred log sourcetype, which can be compared with the known log sourcetype to update model parameters to improve the machine learning model accuracy. The trained machine learning model may be trained to infer a log sourcetype of a log regardless of the messagetype of the log.

Abstract: Systems and methods are described for extracting data fields from logs ingested in a data processing pipeline or otherwise stored. For example, a log can be applied as an input to an artificial intelligence model trained to infer a log sourcetype of logs, and the artificial intelligence model can output an inferred log sourcetype of the log. The inferred log sourcetype can be used to select another artificial intelligence model trained to extract data fields from logs having the inferred log sourcetype, and the log can then be applied as an input to the other artificial intelligence model. The other artificial intelligence model may then output one or more data fields extracted from the log.

Abstract: Systems and methods are described for processing ingested data using an online machine learning algorithm as the data is being ingested. For example, the online machine learning algorithm can be an adaptive thresholding algorithm used to identify outliers in a moving window of data. As another example, the online machine learning algorithm can be a sequential outlier detector that detects anomalous sequences of logs or events. As another example, the online machine learning algorithm can be a sentiment analyzer that determines whether text has a positive, negative, or neutral sentiment. As another example, the online machine learning algorithm can be a drift detector that detects whether ingested data marks the start of a change in the distribution of a time-series.

Abstract: Systems and methods are described for processing ingested pipeline metrics and ingested logs in an asynchronous manner as the data is being ingested to explain anomalies detected in the pipeline metrics using the ingested logs. For example, one or more streaming data processors can convert data as the data is ingested into a comparable data structure, determine whether the comparable data structure should be assigned to an existing data pattern or a new data pattern, and determine whether the logs corresponding to the comparable data structure is anomalous. Separately, the streaming data processor(s) can perform an outlier detection on the pipeline metrics to detect outliers. The streaming data processor(s) can then window the anomalous logs and the pipeline metric outliers to surface explanations for the pipeline metric outliers using the anomalous logs.

Abstract: Systems and methods are described for providing a user interface through which a user can program operation of a data processing pipeline by specifying a graph of nodes that transform data and interconnections that designate routing of data between individual nodes within the graph. In response to a user request, a preview mode can be activated that causes the data processing pipeline to retrieve data from at least one source specified by the graph, transform the data according to the nodes of the graph, sample the transformed data, and display the sampling of the transformed data to at least one node without writing the transformed data to at least one destination specified by the graph.

Abstract: Systems and methods are described for processing ingested data in an asynchronous manner as the data is being ingested to detect potential anomalies. For example, one or more streaming data processors can convert data as the data is ingested into a comparable data structure, determine whether the comparable data structure should be assigned to an existing data pattern or a new data pattern, and optionally update a characteristic of the data pattern to which the comparable data structure is assigned. The streaming data processor(s) can perform these operations automatically in real-time or in periodic batches. Once one or more comparable data structures have been assigned to one or more data patterns, the streaming data processor(s) can analyze the comparable data structures assigned to a particular data pattern to determine whether any of the comparable data structures appear to be anomalous.

Abstract: Systems and methods are described for processing ingested data, detecting anomalies in the ingested data, and providing explanations of a possible cause of the detected anomalies as the data is being ingested. For example, a token or field in the ingested data may have an anomalous value. Tokens or fields from another portion of the ingested data can be extracted and analyzed to determine whether there is any correlation between the values of the extracted tokens or fields and the anomalous token or field having an anomalous value. If a correlation is detected, this information can be surfaced to a user.

Abstract: Systems and methods are described for testing one or more machine learning algorithms in parallel with an existing machine learning algorithm implemented within a data processing pipeline. Each machine learning algorithm can train a machine learning model that receives a live stream of raw machine data. The output of the machine learning model trained by the existing machine learning algorithm may be written to an external storage system, but the output of the machine learning model(s) trained by the test machine learning algorithm(s) may not be written to an external storage system. After some time, performance of the test machine learning algorithm(s) and the existing machine learning algorithm is evaluated. If the test machine learning algorithm performs better than the existing machine learning algorithm, then the machine learning algorithms can be swapped without any downtime and without needed to re-train a machine learning model using previously seen raw machine data.

Abstract: An advertising and data analysis platform may need to mine through vast amounts of data to come up with insights into advertising effectiveness, and measure and improve the effectiveness of advertising reach. Distributed network data analytics may be applied to ad matching/targeting, such that an in-memory cluster computing environment may be used with advertising data. For example, HADOOP may be utilized for distributed processing of the vast amounts of data and the HADOOP distributed file system (HDFS) is used for organizing communications and storage of that data. Satellite clusters or nodes may be generated that also utilize HDFS. For example, a SPARK or SHARK satellite cluster may be arranged to further utilize the HDFS of the HADOOP clusters.

Abstract: Example methods, apparatuses, and/or articles of manufacture are disclosed that may be implemented, in whole or in part, using one or more computing devices to provide analysis of the distribution of overlaps of logs of values.

Abstract: Described herein are solutions for determining quality of online ads and matching the ads to content so that the content is not devalued by the ads. Such solutions may also identify relationships between ads and their influence on user engagement with host content. The solutions may also define and provide the relationships to advertisers, in forms of historical scores and projected scores. The historical scores may include historical elasticity scores and the projected scores may include projected elasticity scores. The scores may be determined per ad and content pair. The solutions can use the scores to influence ad pricing.

Abstract: An advertising and data analysis platform may need to mine through vast amounts of data to come up with insights into advertising effectiveness, and measure and improve the effectiveness of advertising reach. Distributed network data analytics may be applied to ad matching/targeting, such that an in-memory cluster computing environment may be used with advertising data. For example, HADOOP may be utilized for distributed processing of the vast amounts of data and the HADOOP distributed file system (HDFS) is used for organizing communications and storage of that data. Satellite clusters or nodes may be generated that also utilize HDFS. For example, a SPARK or SHARK satellite cluster may be arranged to further utilize the HDFS of the HADOOP clusters.