Abstract: In accordance with various embodiments of the present disclosure, a first instance of a data intake and query system (DIQS) may receive latency data that indicates latency states of second instances of the DIQS, the latency states indicative of latencies associated with processing of event data by the plurality of second instances. The first instance may then determine overall latency state of the first instance based, at least in part, on determining number or percentage of the first instance and the second instances of the DIQS having one or more particular latency states, and determining whether the number or percentage of the first instance and the f second instances of the DIQS having the one or more particular latency states is equal to or exceeds a threshold. The first instance may then present the overall latency state of the first instance.

Abstract: Techniques and mechanisms are disclosed to optimize the size of index files to improve use of storage space available to indexers and other components of a data intake and query system. Index files of a data intake and query system may include, among other data, a keyword portion containing mappings between keywords and location references to event data containing the keywords. Optimizing an amount of storage space used by index files may include removing, modifying and/or recreating various components of index files in response to detecting one or more storage conditions related to the event data indexed by the index files. The optimization of index files generally may attempt to manage a tradeoff between an efficiency with which search requests can be processed using the index files and an amount of storage space occupied by the index files.

Abstract: This technology is directed to facilitating scalable and secure data collection. In particular, scalability of data collection is enabled in a secure manner by, among other things, abstracting a connector(s) to a pod(s) and/or container(s) that executes separate from other data-collecting functionality. For example, an execution manager can initiate deployment of a collect coordinator on a first pod associated with a first job and deployment of a first connector on a second pod associated with a second job separate from the first job of a container-managed platform. The collect coordinator can provide a data collection task to the first connector deployed on the second pod of the second job. The first connector can then obtain the set of data from the data source and provide the set of data to the collect coordinator for providing the set of data to a remote source.

Abstract: An analysis system receives data streams generated by instances of instrumented software executing on external systems. The analysis system evaluates an expression using data values of the data streams over a plurality of time intervals. For example, the analysis system may aggregate data values of data streams for each time interval. The analysis system determines whether or not a data stream is considered for a time interval based on when the data value arrives during the time interval. The analysis system determines a maximum expected delay value for each data stream being processed. The analysis system evaluates the expression using data values that arrive before their maximum expected delay values. The analysis system also determines a failure threshold value for a data stream. If a data value of a data stream fails to arrive before the failure threshold value, the analysis system marks the data stream as dead.

Abstract: A method of tracking errors in a system comprising microservices comprises ingesting a plurality of spans generated by the microservices during a given duration of time. The method further comprises consolidating the plurality of spans associated with the given duration of time into a plurality of traces, wherein each trace comprises a subset of the plurality of spans that comprise a common trace identifier. For each trace, the method comprises: a) mapping a respective trace to one or more error stacks computed for the respective trace and to one or more attributes determined for the respective trace; and b) emitting each error stack computed from the respective trace with an associated pair of attributes. The method then comprises reducing duplicate pairs of error stack and associated attributes and maintaining a count for each pair of error stack and associated attributes.

Abstract: Implementations include receiving a user provided example value of personally identifiable information (PII). Occurrences of the received example value are automatically identified in a dataset of events, wherein each occurrence is identified in a portion of raw machine data of a respective event of the events. For each occurrence of the identified occurrences, an extraction rule is generated, which defines a pattern of the occurrence of the example value and is executable to identify PII values in portions of raw machine data of the events using the pattern. Values of the PII are identified in a set of events using a set of extraction rules comprising the extraction rule of a plurality of the occurrences.

Abstract: Systems and methods for assigning scores to objects based on evaluating triggering conditions applied to datasets produced by search queries in data aggregation and analysis systems. An example method includes causing display of a user interface for generating a correlation search, the correlation search comprising a search query, a triggering condition to be applied to a dataset produced by the search query, and one or more actions to be performed when the dataset produced by the search query satisfies the triggering condition, wherein the one or more actions comprise at least modifying a score assigned to an object to which the dataset produced by the search query pertains.

Abstract: Techniques are described for providing an IT and security operations mobile application for managing IT and security operations instances of an IT and security operations application via a mobile device. The IT and security operations mobile application can be linked to the IT and security operations application to enable the IT and security operations application to send messages (e.g., notifications, alerts, action requests, etc.) related the occurrences of incidents/events in an IT environment, such as security-related incident, that can impact the operation of the IT environment. The IT and security operations mobile application enables a user to respond to the messages by initiating actions that are sent to the IT and security operations application for executing within the IT environment.

Abstract: Systems and methods are disclosed for authenticating a chunk of data identified in a query received by a data intake and query system. The data intake and query system receives a query that identifies a set of data and manner for processing the set of data, and identifies a chunk of data that is part of the set of data. The system generates a content identifier, such as a hash, of the chunk of data. The system further authenticates the chunk of data based on the generated content identifier and a content identifier stored by a distributed ledger system.

Abstract: A data intake and query system receives a message including raw machine via an internet protocol (IP) such as the hypertext transfer protocol (HTTP). The message includes a distinct payload portion and a distinct custom field portion. The payload portion includes raw machine data, while the custom field portion includes values for fields. An event that includes the raw machine data and the values is generated from the payload portion and the values are extracted from the custom field portion. The event is then stored such that the values are associated with the event.

Abstract: Systems and methods are described for executing a query of raw machine data that is stored at a remote data store that may store heterogeneous data. The system can determine the directories or file types that may store event data and may instruct one or more worker nodes to access files that may store events based on the determined directories of file types. Further, the system may exclude files at the remote data store that may not be identified as potentially storing events enabling a query that implicates a heterogeneous data store to be efficiently executed.

Abstract: An actionable event collector in a server cluster receives information specifying an actionable event instance regarding an actionable event occurrence in the server cluster. The actionable event collector transmits a representation of the actionable event instance to an actionable event queue builder. The actionable event queue builder inserts the representation as an entry into an actionable event queue. The event action dispatcher processes the entry from the actionable event queue, wherein processing the entry comprises determining a responsive action for the entry and causing performance of the responsive action.

Abstract: Operational machine components of an information technology (IT) or other microprocessor- or microcontroller-permeated environment generate disparate forms of machine data. Network connections are established between these components and processors of an automatic data intake and query system (DIQS). The DIQS conducts network transactions on a periodic and/or continuous basis with the machine components to receive the disparate data and ingest certain of the data as measurement entries of a DIQS metrics datastore that is searchable for DIQS query processing. The DIQS may receive search queries to process against the received and ingested data via an exposed network interface. In one example embodiment, a query building component conducts a user interface using a network attached client device. The query building component may elicit search criteria via the user interface using a natural language interface, construct a proper query therefrom, and present new information based on results returned from the DIQS.

Abstract: Described herein are techniques are provided for enabling a security orchestration, automation, and response (SOAR) service to automatically manage apps used to interface with an integrated security operations service and other related devices and services. Further described herein is a SOAR app generator service or application used to automate the creation of apps for a SOAR service based on application programming interfaces (API) specifications for related devices or services, as well as visual playbook editor interfaces for a SOAR service that enable the configuration of complex action input parameters including arrays and objects.

Abstract: A computer-implemented method is disclosed that includes operations of parsing a query comprised of a sequence of operators to detect each operator of the sequence of operators, where the sequence of operators includes a machine learning (ML) operator representing a trained ML model. Additionally, a schema of the ML operator is determined through metadata. A filter or a projection is generated based on the schema of the ML operator, where the filter or projection is configured to reduce an amount of data retrieved upon application of the filter of the projection to an operator of the sequence of operators comprising the query. The schema of the ML operator indicates a schema of input data to be provided to the ML operator and a schema of output data to be provided by the ML operator following processing.

Abstract: A control plane system can be used to manage or generated components in a shared computing resource environment. To generate a modified components, the control plane system can receive receiving configurations of a component. The configurations can include software versions and/or parameters for the component. Using the configurations, the control plane system can generate an image of a modified component, and communicate the image to a master node in the shared computing resource environment. The master node can provides one or more instances of the modified component for use based on the received image.

Abstract: Embodiments of the present invention are directed to facilitating event forecasting. In accordance with aspects of the present disclosure, a set of events determined from raw machine data is obtained. The events are analyzed to identify leading indicators that indicate a future occurrence of a target event, wherein the leading indicators occur during a search period of time the precedes a warning period of time, thereby providing time for an action to be performed prior to an occurrence of a predicted target event. At least one of the leading indicators is used to predict a target event. An event notification is provided indicating the prediction of the target event.

Abstract: Embodiments are directed towards managing and tracking item identification of a plurality of items to determine if an item is a new or existing item, where an existing item has been previously processed. In some embodiments, two or more item identifiers may be generated. In one embodiment, generating the two or more item identifiers may include analyzing the item using a small item size characteristic, a compressed item, or for an identifier collision. The two or more item identifiers may be employed to determine if the item is a new or existing item. In one embodiment, the two or more item identifiers may be compared to a record about an existing item to determine if the item is a new or existing item. If the item is an existing item, then the item may be further processed to determine if the existing item has actually changed.

Abstract: In various embodiments, a natural language (NL) application implements functionality that enables users to more effectively access various data storage systems based on NL requests. As described, the operations of the NL application are guided by, at least in part, on one or more templates and/or machine-learning models. Advantageously, the templates and/or machine-learning models provide a flexible framework that may be readily tailored to reduce the amount of time and user effort associated with processing NL requests and to increase the overall accuracy of NL application implementations.

Abstract: A processing node selects a first task from a task list and sends, to a task assignment repository, a first write operation with a first task identifier of the first task to assign the first task to the processing node. The processing node detects failure of the first write operation based on the first task already being assigned and selects a second task from the task list. The processing node sends, to the task assignment repository, a second write operation with a second task identifier of the second task to assign the second task to the processing node. The processing node detects success of the second write operation and executes the second task.