Abstract: Systems and methods are disclosed for mapping search nodes to a search head in a data intake and query system based on a tenant identifier in order to execute a query received by the data intake and query system. The mapping may allow same or similar search nodes to be used to execute queries that are associated with a particular tenant identifier, in order to take advantage of caching and local data stored with those search nodes. In some cases, search nodes can be mapped based on the tenant identifier using a hashing algorithm, such as a consistent hashing algorithm.

Abstract: Systems and methods are disclosed for making space available in a local storage of a data intake and query system. A cache manager of the data intake and query system may determine an amount of storage space of a local data store that is available for use to perform a query. The cache manager may then use one or more eviction policies associated with content stored at the local data store to purge content items to evict from the local storage. The system may then retrieve content for performing the query from a remote storage and store the retrieved content at the local storage.

Abstract: Systems and methods are described for establishing and managing components of a distributed computing framework implemented in a data intake and query system. The distributed computing framework may include a master and a plurality of worker nodes. The master may selectively operate on a search head captain that is chosen from the search heads of the data intake and query system. The search head captain may distribute configuration information for the master and the distributed computing framework to the other search heads, which in turn, may distribute that configuration information to indexers of the data intake and query system. Worker nodes may be selectively activated for operation on the indexers based on the configuration information, and the worker nodes may additionally use the configuration information to contact the master and join the distributed computing framework.

Abstract: Embodiments are disclosed for performing cache aware searching. In response to a search query, a first bucket and a second bucket in remote storage for processing the search query. A determination is made that a first file in the first bucket is present in a cache when the search query is received. In response to the search query, a search is performed using the first file based on the determination that the first file is present in the cache when the search query is received, and the search is performed using a second file from the second bucket once the second file is stored in the cache.

Abstract: Systems and methods are described for improving data availability and/or resiliency of indexers of a data intake and query system. A data intake and query system can index large amounts of data using one or more indexers. An indexer can store a copy of the data that the indexer is assigned to process in the shared storage system, and a cluster master can track the storage of the data and the indexer assigned to process the data. In the event an indexer fails or is otherwise unable to index data that it has been assigned to index, the cluster master can assign one or more second indexers to process the data. The second indexer can download the data from the shared storage system.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Abstract: Systems and methods are disclosed for scalable bucket merging in a data intake and query system. Various components of a bucket manager can be used to monitor recently-created buckets of data in common storage that are associated with a particular tenant and a particular index, apply a comprehensive bucket merge policy to determine groups of buckets that qualify for merging, merge those group of buckets into merged buckets to be stored in the common storage, and update any information associated with the merged buckets and pre-merged buckets. These components may be shared across multiple tenants, and some of these components may be dynamically scalable based on need. This approach may also provide many additional benefits, including improved search performance from merged buckets, efficient resource utilization associated with discriminate merging, and redundancy in case of component failure.

Abstract: Systems and methods are described for processing incoming data. The system can receive, from a first partition manager of a data intake and query system, first data that is associated with a first identifier, and can receive, from a second partition manager of the data intake and query system, second data that is associated with a second identifier. The system can process the first data and store first results of said processing the first data in one or more first buckets associated with the first tenant identifier. The system can process the second data and store second results of said processing the second data in one or more second buckets associated with the second tenant identifier.

Abstract: Embodiments described herein facilitate enhancement of data model acceleration, including generating data model summaries and performing searches in an accelerated manner. In one implementation, a set of events are indexed, each of the events having a corresponding index time representing a time at which the event was indexed in an indexer. Index time parameters including an index earliest time indicating a first index time at which to begin generating a data model summary and an index latest time indicating a second index time at which to complete generating the data model summary are obtained. Thereafter, a data model summary is generated. Such a data model summary summarizes events having corresponding index times between the index earliest time and the index latest time. The data model summary is provided to a remote data store that is separate from the indexer at which at least a portion of the events were indexed.

Abstract: Embodiments described herein facilitate enhancement of data model acceleration, including generating data model summaries and performing searches in an accelerated manner. In one implementation, obtaining a search query from a user device. A determination may be made to execute a search, in association with the search query, via an external computing service. As such, the search query, or a variant thereof, can be provided to the external computing service, wherein the external computing service executes the search using data model summaries stored in a remote data store that is separate from a set of events from which the data model summaries were generated. A set of search results are received from the external computing service, and such search results are provided to the user device.

Abstract: Achieving search and ingest isolation via resource management in a search and indexing system includes receiving a search query associated with at least one data store, assigning, in response to the search query being associated with the at least one data store, the search query to a first workload pool in a set of query workload pools, and processing the search query using a first hardware resource in the first workload pool. Achieving search and ingest isolation further includes receiving an ingest request comprising data associated with the at least one data store. The ingest request is assigned to a second workload pool in a set of ingest workload pools. The set of query workload pools and the set of ingest workload pools are disjoint. Achieving search and ingest isolation further includes processing the ingest request using a second hardware resource in the second workload pool.

Abstract: Systems and methods are disclosed for scalable bucket merging in a data intake and query system. Various components of a bucket manager can be used to monitor recently-created buckets of data in common storage that are associated with a particular tenant and a particular index, apply a comprehensive bucket merge policy to determine groups of buckets that qualify for merging, merge those group of buckets into merged buckets to be stored in the common storage, and update any information associated with the merged buckets and pre-merged buckets. These components may be shared across multiple tenants, and some of these components may be dynamically scalable based on need. This approach may also provide many additional benefits, including improved search performance from merged buckets, efficient resource utilization associated with discriminate merging, and redundancy in case of component failure.

Abstract: Systems and methods are disclosed for authenticating a user to use one or more components of a data intake and query system. The data intake and query system enables the generation or searching of events that include raw machine data associated with a timestamp. The data intake and query system receives a request for access via an application programming interface (API). Based on the request, the data intake and query system authenticates the user. The data intake and query system can receive a second request via the API for a component of the data intake and query system. Based on a determination that the user is authenticated, the data intake and query system can communicate the request to the component.

Abstract: Systems and methods are disclosed for mapping search nodes to a search head in a data intake and query system based on a tenant identifier in order to execute a query received by the data intake and query system. The mapping may allow same or similar search nodes to be used to execute queries that are associated with a particular tenant identifier, in order to take advantage of caching and local data stored with those search nodes. In some cases, search nodes can be mapped based on the tenant identifier using a hashing algorithm, such as a consistent hashing algorithm.

Abstract: Embodiments are disclosed for performing cache aware searching. In response to a search query, a first bucket and a second bucket in remote storage for processing the search query. A determination is made that a first file in the first bucket is present in a cache when the search query is received. In response to the search query, a search is performed using the first file based on the determination that the first file is present in the cache when the search query is received, and the search is performed using a second file from the second bucket once the second file is stored in the cache.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. An indexing system of the data intake and query system receives data from an ingestion buffer that includes a marker that indicates data that is made available to the indexing system. The data intake and query system stores at least a portion of the data in buckets and stores the buckets in a shared storage system. Based on the storage of the buckets in the shared storage system, the indexing system indicates to the ingestion buffer that the marker can be updated.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Abstract: Systems and methods are disclosed for dynamically assigning a search head or search nodes in a data intake and query system for a query received by the data intake and query system. Existing search heads and search nodes can periodically report their status to the data intake and query system, which can use that information to help determine the need to provision additional search heads and search nodes. The data intake and query system can receive a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system can use the status information for existing search heads and search nodes to dynamically assign a search head and search nodes for the query. Dynamically assigning the search head and search nodes in this manner may provide many benefits, including improved load balancing and resource utilization.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The query identifies a set of data to be processed and a manner of processing the set of data. The data intake and query system dynamically identifies a plurality of containerized search nodes instantiated on one or more computing devices in a containerized environment to execute the query. The data intake and query system executes the query using the containerized search nodes.

Abstract: Systems and methods are disclosed for interfacing with one or more components of a data intake and query system. The data intake and query system includes a gateway that interfaces between one or more computer-executable applications and one or more components of the data intake and query system. The data intake and query system can include an intake system configured to ingest data, an indexing system configured to generate and store one or more events based on the data, and a query system configured to execute one or more queries. The intake system can include a streaming data processor and at least one ingestion buffer. The indexing system can include at least one containerized indexing node, and the query system can include at least one containerized search node.