Abstract: Embodiments of the present disclosure provide techniques for using an inverted index in a pipelined search query. A field searchable data store is provided that comprises a plurality of event records, each event record comprising a time-stamped portion of raw machine data. Responsive to the reciept of an incoming search query, the search engine accesses an inverted index, wherein each entry in the inverted index comprises at least one field name, a corresponding at least one field value and a reference value associated with each field name and value pair that identifies a location in the data store where an associated event record is stored. Once the inverted index is accessed, it can be used to identify and search a subset of the plurality of event records, wherein the subset comprises one or more event records with corresponding reference values in the inverted index.

Abstract: Embodiments of the present invention are directed to facilitating performing data extraction via efficient extraction rule matching. Generally, an extraction rule can be determined to match an event based on a two-step process. In particular, initially, a determination that a set of fixed substrings associated with the extraction rule matches fixed substrings of the event can be made. Based on fixed substring match, a determination can be made that a set of fields associated with the extraction rule matches fields of the event. In such a case, the extraction rule can be deemed to match the event and used to extract values from the event.

Abstract: The disclosed embodiments provide a system for extracting custom content from network packets. During operation, the system receives a stream of packets. The system then parses packets in the stream to determine a protocol for each packet. Next, the system applies a custom-content-extraction rule to each packet associated with a target protocol to obtain the extracted content. Then, the system stores the extracted content in events in a data store to facilitate subsequent queries involving the extracted content.

Abstract: The technology disclosed relates to formulating and refining field extraction rules that are used at query time on raw data with a late-binding schema. The field extraction rules identify portions of the raw data, as well as their data types and hierarchical relationships. These extraction rules are executed against very large data sets not organized into relational structures that have not been processed by standard extraction or transformation methods. By using sample events, a focus on primary and secondary example events help formulate either a single extraction rule spanning multiple data formats, or multiple rules directed to distinct formats. Selection tools mark up the example events to indicate positive examples for the extraction rules, and to identify negative examples to avoid mistaken value selection. The extraction rules can be saved for query-time use, and can be incorporated into a data model for sets and subsets of event data.

Abstract: A continuous anomaly detection service receives data stream and performs continuous anomaly detection on the incoming data streams. This continuous anomaly detection is performed based on anomaly detection definitions, which define a signal used for anomaly detection and an anomaly detection configuration. These anomaly detection definitions can be modified, such that continuous anomaly detection continues to be performed for the data stream and the signal, based on the new anomaly detection definition.

Abstract: Systems and methods are disclosed for executing a query that includes an indication to process data managed by an external data system. The system identifies the external data system that manages the data to be processed and generates a subquery for the external data system indicating that the results of the subquery are to be sent to one worker node of multiple worker nodes. The system instructs the one worker node to distribute the results received from the external data system to multiple worker nodes for processing.

Abstract: Techniques are described for enabling a cloud-based IT and security operations application to execute playbooks containing custom code in a manner that mitigates types of risk related to the misuse of cloud-based resources and security of user data. Users use a client application to create and modify playbooks and, upon receiving input to save a playbook, the client application determines whether the playbook includes custom code. If the client application determines that the playbook includes custom code, the client application establishes a connection with a proxy application (also referred to as an “automation broker”) running in the user's own on-premises network and sends a representation of the playbook to the proxy application. The client application further sends to the IT and security operations application an identifier of the playbook and an indication that the playbook (or the custom code portions of the playbook) is stored within the user's on-premises network.

Abstract: The disclosed embodiments relate to systems and methods that provides a dashboard that includes multiple independent panels where each independent panel functions independently and is associated with a respective search query that when executed generates data that may populate and/or configure the associated panel. The systems and methods further permits generation of a filter condition based on user input provided through a single panel and automatically apply the filter condition to the queries of some or all of the queries of the independent panels of the dashboard and execute the updated queries to update some or all of the independent panels.

Abstract: As an indexer indexes and groups events, it can generate data slices that include events. Based on a slice rollover policy, the indexer can add a particular slice to an aggregate slice. Based on an aggregate slice backup policy, the indexer can store a copy of the aggregate slice to a shared storage system. The aggregate slice can be used for restore purposes in the event the indexer fails or becomes unresponsive.

Abstract: Systems and methods for querying and obtaining results from an external data source that operates with a different querying language is provided. The system activates a datasource connector of the system. The system receives attributes of a query in a native language of the system, and the datasource connector formats the attributes of the query into a query language statement in a native language of the external source. The datasource connector then makes an application programming interface (API) call to the external source. The API call includes a transmission of the query language statement to the external source, which causes the external source to perform a query using the query language statement. The datasource connector receives results of the query performed at the external source, whereby the results are in a non-tabular format. The datasource connector then reformats the results into a tabular format.

Abstract: Embodiments of the present disclosure provide techniques for performing searches of event records by leveraging reference values in an inverted index. A method of searching comprises accessing a query associated with a first set of event records in a field searchable data store, each event record comprising a time-stamped portion of raw machine data. The method further comprises evaluating the query and generating results for the query by accessing an inverted index, wherein each entry in the inverted index comprises at least one field, a corresponding at least one field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored. The method further comprises performing a search to filter out a second set of event records and retrieving the second set of event records from the field searchable data store using reference values in the inverted index.

Abstract: Embodiments of the present invention are directed to facilitating data preprocessing for machine learning. In accordance with aspects of the present disclosure, a training set of data is accessed. A preprocessing query specifying a set of preprocessing parameter values that indicate a manner in which to preprocess the training set of data is received. Based on the preprocessing query, a preprocessing operation is performed to preprocess the training set of data in accordance with the set of preprocessing parameter values to obtain a set of preprocessed data. The set of preprocessed data can be provided for presentation as a preview. Based on an acceptance of the set of preprocessed data, the set of preprocessed data is used to train a machine learning model that can be subsequently used to predict data.

Abstract: Techniques are described for providing a highly available data ingestion system for ingesting machine data sent from remote data sources across potentially unreliable networks. To provide for highly available delivery of such data, a data intake and query system provides users with redundant sets of ingestion endpoints to which messages sent from users' computing environments can be delivered to the data intake and query system. Users' data sources, or data forwarding components configured to obtain and send data from one or more data sources, are then configured to encapsulate obtained machine data into discrete messages and to send copies of each message to two or more of the ingestion endpoints provisioned for a user. The ingestion endpoints receiving the messages implement a deduplication technique and provide only one copy of each message to a subsequent processing component (e.g., to an indexing subsystem for event generation, event indexing, etc.).

Abstract: Systems and methods are disclosed for associating summarizations of visualizations of a data set based on affinities between the summarizations. For a data set, a number of summarizations may be created that summarizes the data set in different ways. The summarizations may be linked, such that selecting a data element of a first summarization causes display of a second summarization. To assist in linking of summarizations, suggested linkings between summarizations can be determined based on affinities of the two summarizations. Affinities can reflect similarities in the data content of the two summarizations, such as an output of a first summarization being a valid input to the second summarization.

Abstract: Described are techniques for accelerating streaming analytics jobs, which may be used for generating dashboards. The disclosed techniques can reduce overhead, such as in the form of processor usage, network usage, or the like, due to duplicative or overlapping requests for streaming analytics data by implementing a caching process in which analytics data is evaluated to determine if it is likely to be requested multiple times or by multiple users, caching the analytics data, and serving future requests for the same analytics data from the cache instead of requiring separate analytics jobs for each request.

Abstract: An instrumentation analysis system processes data streams received from servers executing instrumented software. The system determines a set of servers that satisfy a given criteria, for example, a set of servers with high resource utilization. The set of servers may be determined by the system based on triggers or specified by a user. The system analyzes properties of servers to determine a property that characterizes the set of servers. The property characterizing the servers is provided to users via a user interface or alerts for further analysis, for example, to analyze the cause of high resource utilization.

Abstract: A method comprises acquiring anomaly data including a plurality of anomalies detected from streaming data, wherein each of the anomalies relates to an entity on or associated with a computer network. The method determines a risk score of each of the anomalies, and adjusts the risk score of an anomaly according to a set of factors. The method further determines, for each of a plurality of sliding time windows of different lengths, an entity score of the entity in relation to the sliding time window, based on an aggregation of risk scores of all anomalies related to the entity that were detected within the sliding time window, where the entity score corresponds to a risk level associated with the entity. An action to prevent the entity from performing an operation can be determined and caused to occur based on the entity score.

Abstract: Techniques are described for automating the configuration of a simple network management protocol (SNMP) manager device for enabling collection of SNMP data from one or more SNMP-enabled devices. Based upon SNMP object identifiers (OIDs) received from an SNMP-enabled device, processing is performed to map the OIDs to one or more SNMP management information bases (MIBs) corresponding to the OIDs. The identification of the OIDs and mapping the OIDs to one or more MIBs is performed in an automated manner and substantially free of any human or manual intervention. The identified one or more MIBs are then used to configure the SNMP manager to enable SNMP communications between the SNMP-enabled device and the SNMP manager. In certain implementations, the identified one or more MIBs are loaded into system memory by the SNMP manager.

Abstract: The disclosure includes methods and systems that perform operations of identifying a behavior of a metric, where the metric is associated with a node of included within a nodal graph displayed on a graphical user interface. Additionally, a root cause of the behavior is determined through automated, computerized analytics, which may include execution of a search query associated with the node, and a notification of the root cause may be provided via the graphical user interface. Additionally, the graphical user interface may be configured to receive user input that results in the generation of a nodal graph, where the user input includes placement of nodes on a display screen and edges representing a connection between two nodes, where the edges may represent a dependency between the nodes.

Abstract: Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is stored as discrete events time stamps. A search is received and relevant event information is retrieved based in whole or in part on the time stamp, a keyword indexing mechanism, or statistical indices calculated at the time of the search.