Abstract: Techniques and mechanisms are disclosed to optimize the size of index files to improve use of storage space available to indexers and other components of a data intake and query system. Index files of a data intake and query system may include, among other data, a keyword portion containing mappings between keywords and location references to event data containing the keywords. Optimizing an amount of storage space used by index files may include removing, modifying and/or recreating various components of index files in response to detecting one or more storage conditions related to the event data indexed by the index files. The optimization of index files generally may attempt to manage a tradeoff between an efficiency with which search requests can be processed using the index files and an amount of storage space occupied by the index files.

Abstract: Systems and methods are described for monitoring indexing nodes, populating and maintaining a resource catalog with relevant information, receiving requests for indexing node availability or assignments, identifying indexing nodes that are available to process data, and/or communicating information relating to available indexing nodes. The system can maintain the resource catalog based on communications with each of the containerized indexing nodes. The system can receive, from a partition manager of a data intake and query system, a request for a containerized indexing node that the partition manager can assign to process data received by the partition manager. The system can identify an available containerized indexing node to process the data. The system can communicate, to the partition manager, an indexing node identifier associated with the available containerized indexing node.

Abstract: Systems and methods are described for improving data availability and/or resiliency of indexers of a data intake and query system. Due to a lag between the time at which data is received and the time at which the data is available for searching, the data intake and query system may receive a query indicating that received (but unavailable for search) data is to be included as part of the query. A cluster master can dynamically track what data is available for searching by different indexers and map the data to filter criteria using a bucket map identifier. When a search head receives a query, it can request a bucket map identifier from the cluster master and send the bucket map identifier to the indexers that will be executing the query. The indexers can use the bucket map identifier to request the individual buckets that they are assigned to search.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Abstract: Systems and methods are disclosed for making space available in a local storage of a data intake and query system. A cache manager of the data intake and query system may determine an amount of storage space of a local data store that is available for use to perform a query. The cache manager may then use one or more eviction policies associated with content stored at the local data store to purge content items to evict from the local storage. The system may then retrieve content for performing the query from a remote storage and store the retrieved content at the local storage.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets. Based on a determination that the size of multiple buckets satisfies a threshold size, the data intake and query system converts the buckets to non-editable buckets and stores the data in a remote shared storage system.

Abstract: A data intake and query system can manage the search of large amounts of data using one or more processing nodes. When a new processing node is added or becomes available, the node coordinator can reassign duties from one or more processing nodes to the new processing node. The node coordinator can initially assign the new processing node one or more groups of data for backup purposes. At a later time, the node coordinator can reassign the new processing node to the one or more groups of data for searching purposes.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. An indexing system of the data intake and query system receives data and stores at least a portion of it in buckets, which are then stored in a shared storage system. The indexing system merges multiple buckets to generate merged buckets and uploads the merged buckets to the shared storage system.

Abstract: Systems and methods are disclosed for making space available in a local storage of a data intake and query system. A cache manager of the data intake and query system may determine an amount of storage space of a local data store that is available for use to perform a query. The cache manager may then use one or more eviction policies associated with content stored at the local data store to purge content items to evict from the local storage. The system may then retrieve content for performing the query from a remote storage and store the retrieved content at the local storage.

Abstract: Systems and methods are described for improving data availability and/or resiliency of indexers of a data intake and query system. A data intake and query system can index large amounts of data using one or more indexers. An indexer can store a copy of the data that the indexer is assigned to process in the shared storage system, and a cluster master can track the storage of the data and the indexer assigned to process the data. In the event an indexer fails or is otherwise unable to index data that it has been assigned to index, the cluster master can assign one or more second indexers to process the data. The second indexer can download the data from the shared storage system.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Abstract: Systems and methods are described for processing incoming data. The system can receive, from a first partition manager of a data intake and query system, first data that is associated with a first identifier, and can receive, from a second partition manager of the data intake and query system, second data that is associated with a second identifier. The system can process the first data and store first results of said processing the first data in one or more first buckets associated with the first tenant identifier. The system can process the second data and store second results of said processing the second data in one or more second buckets associated with the second tenant identifier.

Abstract: A search request received at a computer of a search support system is processed by analyzing the received search request to identify request parameters and connecting to a system index of the search support system that is referenced in the request parameters. An external result provider (ERP) process is initiated that establishes communication between the search support system and a data source external to the search support system, for a virtual index referenced in the request parameters. Thus, the ERP process provides an interface between the search support system and external data sources, such as by third parties. The ERP process can operate in a streaming mode (providing real-time search results with minimal processing) and/or a reporting mode (providing results with a greater delay and processing extent) and can switch between modes. The search request results are received from the connected system indexes and the referenced virtual indexes.

Abstract: A search request received at a computer of a search support system is processed by analyzing the received search request to identify request parameters and connecting to a system index of the search support system that is referenced in the request parameters. An external result provider (ERP) process is initiated that establishes communication between the search support system and a data source external to the search support system, for a virtual index referenced in the request parameters. Thus, the ERP process provides an interface between the search support system and external data sources, such as by third parties. The ERP process can operate in a streaming mode (providing realtime search results with minimal processing) and/or a reporting mode (providing results with a greater delay and processing extent) and can switch between modes. The search request results are received from the connected system indexes and the referenced virtual indexes.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. An indexing system of the data intake and query system receives data from an ingestion buffer that includes a marker that indicates data that is made available to the indexing system. The data intake and query system stores at least a portion of the data in buckets and stores the buckets in a shared storage system. Based on the storage of the buckets in the shared storage system, the indexing system indicates to the ingestion buffer that the marker can be updated.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Abstract: Techniques and mechanisms are disclosed to optimize the size of index files to improve use of storage space available to indexers and other components of a data intake and query system. Index files of a data intake and query system may include, among other data, a keyword portion containing mappings between keywords and location references to event data containing the keywords. Optimizing an amount of storage space used by index files may include removing, modifying and/or recreating various components of index files in response to detecting one or more storage conditions related to the event data indexed by the index files. The optimization of index files generally may attempt to manage a tradeoff between an efficiency with which search requests can be processed using the index files and an amount of storage space occupied by the index files.

Abstract: The disclosed embodiments include a method performed by a data intake and query system. The method includes ingesting metrics including respective key values and respective measured values, where the respective key values include a primary key value of a selected primary key. The method further includes generating a hash value for each metric by processing each primary key value with a hashing function, and indexing each metric in association with an existing hash bucket having a matching hash value.

Abstract: The disclosed embodiments include a method performed by a data intake and query system. The method includes ingesting each metric including at least one key value and a measured value taken of a computing resource, and storing each metric in an index of a metrics store, where the index defines at least one dimension populated with the at least one key value and a measure populated with the measured value. The method further includes cataloging metadata in a metrics catalog, where the metadata is related to the metrics stored in the metrics store, performing an analysis of metrics data included in the metrics store and/or the metrics catalog to obtain results, and causing display of the results or an indication of the results on a display device.

Abstract: Techniques and mechanisms are disclosed to optimize the size of index files to improve use of storage space available to indexers and other components of a data intake and query system. Index files of a data intake and query system may include, among other data, a keyword portion containing mappings between keywords and location references to event data containing the keywords. Optimizing an amount of storage space used by index files may include removing, modifying and/or recreating various components of index files in response to detecting one or more storage conditions related to the event data indexed by the index files. The optimization of index files generally may attempt to manage a tradeoff between an efficiency with which search requests can be processed using the index files and an amount of storage space occupied by the index files.