Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Abstract: The disclosed embodiments include a method performed by a data intake and query system. The method includes receiving a search query by a search head, defining a search process for applying the search query to indexers, delegating a first portion of the search process to indexers and a second portion of the search process to intermediary node(s) communicatively coupled to the search head and the indexers. The first portion can define a search scope for obtaining partial search results of the indexers and the second portion can define operations for combining the partial search results by the intermediary node(s) to produce a combination of the partial search results. The search head then receives the combination of the partial search results, and outputs final search results for the search query, where the final search results are based on the combination of the partial search results.

Abstract: A data intake and query system can manage the search of large amounts of data using one or more processing nodes. When a new processing node is added or becomes available, the node coordinator can reassign duties from one or more processing nodes to the new processing node. The node coordinator can initially assign the new processing node one or more groups of data for backup purposes. At a later time, the node coordinator can reassign the new processing node to the one or more groups of data for searching purposes.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Abstract: Techniques and mechanisms are disclosed to optimize the size of index files to improve use of storage space available to indexers and other components of a data intake and query system. Index files of a data intake and query system may include, among other data, a keyword portion containing mappings between keywords and location references to event data containing the keywords. Optimizing an amount of storage space used by index files may include removing, modifying and/or recreating various components of index files in response to detecting one or more storage conditions related to the event data indexed by the index files. The optimization of index files generally may attempt to manage a tradeoff between an efficiency with which search requests can be processed using the index files and an amount of storage space occupied by the index files.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched and search nodes to execute the query. The data intake and query system maps the identified buckets to the search nodes and executes the query using the identified bucket and search nodes.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system uses a search node catalog to identify search nodes that are available to execute the query and uses a bucket catalog to identify buckets to be searched. The data intake and query system executes the query using the identified bucket and search nodes.

Abstract: Systems and methods are described for improving data availability and/or resiliency of indexers of a data intake and query system. Due to a lag between the time at which data is received and the time at which the data is available for searching, the data intake and query system may receive a query indicating that received (but unavailable for search) data is to be included as part of the query. A cluster master can dynamically track what data is available for searching by different indexers and map the data to filter criteria using a bucket map identifier. When a search head receives a query, it can request a bucket map identifier from the cluster master and send the bucket map identifier to the indexers that will be executing the query. The indexers can use the bucket map identifier to request the individual buckets that they are assigned to search.

Abstract: Embodiments of the present disclosure provide a method for performing search queries. The method comprises transmitting a list of active indexers in an indexer cluster from a cluster master for receipt by a first search head, wherein the cluster master is communicatively coupled with an indexer cluster comprising a plurality of indexers and the first search head. The method further comprises receiving a first slot request at the cluster master in response to a query from the first search head, wherein the first search head is operable to transmit the query to the active indexers for execution if granted the slot request. Further, the method comprises evaluating a plurality of policies to determine if the first slot request can be granted and responsive to a positive determination, transmitting an authorization token for a slot to the first search head.

Abstract: The disclosed embodiments include a method performed by a data intake and query system. The method includes receiving a search query by a search head, defining a search process for applying the search query to indexers, delegating a first portion of the search process to indexers and a second portion of the search process to intermediary node(s) communicatively coupled to the search head and the indexers. The first portion can define a search scope for obtaining partial search results of the indexers and the second portion can define operations for combining the partial search results by the intermediary node(s) to produce a combination of the partial search results. The search head then receives the combination of the partial search results, and outputs final search results for the search query, where the final search results are based on the combination of the partial search results.

Abstract: Systems and methods are disclosed for mapping search nodes to a search head in a data intake and query system based on a tenant identifier in order to execute a query received by the data intake and query system. The mapping may allow same or similar search nodes to be used to execute queries that are associated with a particular tenant identifier, in order to take advantage of caching and local data stored with those search nodes. In some cases, search nodes can be mapped based on the tenant identifier using a hashing algorithm, such as a consistent hashing algorithm.

Abstract: A data intake and query system can manage the search of large amounts of data using one or more processing nodes. When a new processing node is added or becomes available, the node coordinator can reassign duties from one or more processing nodes to the new processing node. The node coordinator can initially assign the new processing node one or more groups of data for backup purposes. At a later time, the node coordinator can reassign the new processing node to the one or more groups of data for searching purposes.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched and stores a copy of buckets in memory associated with one or more search nodes. A search node performs a search on buckets residing in its memory.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched. The data intake and query system performs a hash on bucket identifiers of the identified buckets to identify search nodes to search the buckets.

Abstract: Systems and methods are disclosed for mapping search nodes to a search head in a data intake and query system based on a tenant identifier in order to execute a query received by the data intake and query system. The mapping may allow same or similar search nodes to be used to execute queries that are associated with a particular tenant identifier, in order to take advantage of caching and local data stored with those search nodes. In some cases, search nodes can be mapped based on the tenant identifier using a hashing algorithm, such as a consistent hashing algorithm.

Abstract: Embodiments of the present disclosure provide a method for performing search queries in a manner that avoids overloading an indexer cluster or indexers with an unwanted or unauthorized high levels of concurrent searches. The method comprises transmitting a slot request from a search head to a cluster master in response to a query, wherein the cluster master is communicatively coupled with an indexer cluster comprising a plurality of indexers. The method further comprises receiving addresses of active indexers in the indexer cluster and a response to the slot request from the cluster master. Responsive to a grant of a slot by the cluster master, the method comprises using the addresses to transmit the query to the active indexers and receiving results of the query from the active indexers. Subsequently, the method comprises releasing the slot to the cluster master.

Abstract: Systems and methods are described for improving data availability and/or resiliency of indexers of a data intake and query system. A data intake and query system can index large amounts of data using one or more indexers. An indexer can store a copy of the data that the indexer is assigned to process in the shared storage system, and a cluster master can track the storage of the data and the indexer assigned to process the data. In the event an indexer fails or is otherwise unable to index data that it has been assigned to index, the cluster master can assign one or more second indexers to process the data. The second indexer can download the data from the shared storage system.

Abstract: An electronic mortgage portfolio asset auction system includes at least one host computing device that assembles a mortgage asset inventory from received mortgage asset data of a plurality of financial institutions for bidding. The host computing device filters the mortgage asset inventory in view of a bidder's mortgage asset purchase criteria to identify mortgage assets corresponding to the bid order, generates at least one ranked list of the identified mortgage assets by applying financial institution preference criteria, and presents the at least one ranked list to the first one of the plurality of financial institutions.

Abstract: Embodiments of the present disclosure provide a method for performing search queries. The method comprises transmitting a list of active indexers in an indexer cluster from a cluster master for receipt by a first search head, wherein the cluster master is communicatively coupled with an indexer cluster comprising a plurality of indexers and the first search head. The method further comprises receiving a first slot request at the cluster master in response to a query from the first search head, wherein the first search head is operable to transmit the query to the active indexers for execution if granted the slot request. Further, the method comprises evaluating a plurality of policies to determine if the first slot request can be granted and responsive to a positive determination, transmitting an authorization token for a slot to the first search head.

Abstract: Systems and methods are disclosed for mapping search nodes to a search head in a data intake and query system based on a tenant identifier in order to execute a query received by the data intake and query system. The mapping may allow same or similar search nodes to be used to execute queries that are associated with a particular tenant identifier, in order to take advantage of caching and local data stored with those search nodes. In some cases, search nodes can be mapped based on the tenant identifier using a hashing algorithm, such as a consistent hashing algorithm.