Abstract: Systems and methods are disclosed for receiving, at a data intake and query system, a query that includes an indication to process data managed by a third-party data storage and processing system that supports a different query language than the data intake and query system. The data intake and query system identifies a third-party data storage and processing system that manages the data to be processed and generates a subquery for execution by the third-party data storage and processing system, generates instructions for one or more worker nodes to receive and process results of the subquery from the third-party data storage and processing system, and instructs the worker nodes to provide results of the processing to the data intake and query system.

Abstract: Multi-threaded processing of search responses returned by search peers is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system; receiving, by a first processing thread, a plurality of data packets from the plurality of search peers; parsing, by a second processing thread operating asynchronously with respect to the first processing thread, one or more data packets of the plurality of data packets, to produce a partial response to the search request; splitting the partial response into two or more fields; and generating, based on the two or more fields of the partial response, an aggregated response to the search request.

Abstract: Systems and methods are disclosed for processing and executing queries against one or more dataset. As part of processing the query, the system determines whether the query is susceptible to a significantly imbalanced partition. In the event, the query is susceptible to an imbalanced partition, the system monitors the query and determines whether to perform a multi-partitioning determination to avoid a significantly imbalanced partition.

Abstract: Systems and methods are disclosed for processing and executing queries against one or more dataset. As part of processing the query, the system determines whether the query is susceptible to a significantly imbalanced partition. In the event, the query is susceptible to an imbalanced partition, the system monitors the query and determines whether to perform a multi-partitioning determination to avoid a significantly imbalanced partition.

Abstract: Systems and methods are described for unified processing of indexed and streaming data. A system enables users to query indexed data or to apply such queries to streaming data. In some instances, a user may specify a query in the same manner as if the query were intended to be run against indexed data, but may specify criteria that includes not-yet-indexed data (e.g., a future time frame). The system may convert the query into a processing graph to be applied to not-yet-indexed data, thus increasing the efficiency of the system.

Abstract: Systems and methods are described for determining a query execution time in a data intake and query system. The system parses a query to identify different portions of the query that are executed by different components of the data intake and query system. The system determines a query execution time for the different portions of the query based on the corresponding components. Based on the query execution time of the different portions for the query, the system determines a query execution time for the query.

Abstract: Systems and methods are disclosed for processing queries against a common storage utilizing dynamically allocated partitions operating on one or more worker nodes. The common storage can include one or more data stores, which collectively contain a data set divided across multiple buckets of data. To query the common storage, a query coordinator can retrieve metadata regarding the multiple buckets, in order to determine a subset of buckets that are potentially relevant to a query. The query coordinator can then dynamically allocate partitions operating on worker nodes to retrieve and intake individual buckets of the subset into a phased search process. The dynamic allocation can be selected to maximize parallelization of the buckets across partitions, thus increasing a speed at which the common storage can be searched.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system uses one or more containerized search nodes to execute the query and stores the results in a data store for combination with additional query results.

Abstract: Processing of search responses returned by search peers is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system; receiving a plurality of data packets from the plurality of search peers; parsing one or more data packets of the plurality of data packets, to produce a response to the search request; and splitting the response into two or more fields based on at least one of: a defined set of bit positions or a defined separator.

Abstract: Systems and methods are disclosed for generating a distributed execution model with untrusted commands. The system can receive a query, and process the query to identify the untrusted commands. The system can use data associated with the untrusted command to identify one or more files associated with the untrusted command. Based on the files, the system can generate a data structure and include one or more identifiers associated with the data structure in the distributed execution model. The system can distribute the distributed execution model to one or more nodes in a distributed computing environment for execution.

Abstract: Systems and methods are disclosed for processing queries against one or more dataset sources. The system tracks query resource data and resource utilization data. The query-resource usage data can indicate resources used to execute queries. The node resource utilization data can indicate current utilization of nodes in the system. Upon receipt of a query that identifies a set of data to be processed and a manner of processing the set of data, the system can use the query-resource usage data and the resource utilization data to define a query processing scheme. The query can then be executed using the query processing scheme. In some cases, the query coordinator can dynamically allocate partitions operating on worker nodes to execute the query.

Abstract: Systems and methods are disclosed for executing a distributed execution model with untrusted commands. The distributed execution model can be distributed to multiple nodes in a distributed computing environment. At least one node can process the distributed execution model to identify an untrusted command. The node can use data associated with the untrusted command to identify one or more files associated with the untrusted command. Based on the files, the node can generate a data structure, and execute at least a portion of the data structure.

Abstract: Systems and methods are disclosed for generating a distributed execution model with untrusted commands. The system can receive a query, and process the query to identify the untrusted commands. The system can use data associated with the untrusted command to identify one or more files associated with the untrusted command. Based on the files, the system can generate a data structure and include one or more identifiers associated with the data structure in the distributed execution model. The system can distribute the distributed execution model to one or more nodes in a distributed computing environment for execution.

Abstract: Disclosed is a data fabric service system that can be implemented in a distributed computer network, such as a data intake and query system. The data index and query system can receive a search query and define a search scheme for applying the search query on distributed data storage systems including internal data storage and external data storage. The data index and query system may provide a portion of the search scheme to a search service of the data fabric service system, which can cause worker nodes of the data fabric service system to perform various functions—including applying the search query to the external data storage based on the portion of the search scheme in order to obtain search results.

Abstract: A content saliency network is a machine-learned neural network that predicts the saliency of elements of a content item. The content saliency network may be used in a method that includes determining a set of elements in a UI and computing a first context vector for the content. The method may also include, for each element in the set of elements, computing a vector of simple features for the element, the simple features being computed from attributes of the element, computing a second context vector for the element, computing a third context vector for an intermediate context of the element, and providing the vectors to the content saliency network. The content saliency network provides a saliency score for the element. The method further includes generating an element-level saliency map of the content using the respective saliency scores for the set of elements and providing the saliency map to a requestor.

Abstract: The disclosed embodiments include techniques for organizing and presenting search results obtained from within a big data ecosystem via a data intake and query system. In particular, a data intake and query system may cause output of the search results or data indicative of the search results on a display device.

Abstract: The disclosed embodiments include techniques for exporting partial search results in parallel from peer indexers of a data intake and query system to the worker nodes. In particular, partial search results (e.g., time-indexed events) obtained from peer indexers can be exported in parallel from the peer indexers to worker nodes. Exporting the partial search results from the peer indexers in parallel can improve the rate at which the partial search results are transferred to the worker nodes for subsequent combination with partial search results of the external data systems. As such, the rate at which the search results of a search query can be obtained from the distributed data system can be improved by implementing parallel export techniques.

Abstract: The capabilities of a data intake and query system can be improved by implementing the data fabric service (DFS) system in a co-located deployment with the data intake and query system. The DFS system can extend the capabilities of a data intake and query system by leveraging computing assets from anywhere in a big data ecosystem to collectively execute search queries on diverse data systems regardless of whether data stores are internal of the data intake and query system and/or external data stores that are communicatively coupled to the data intake and query system over a network.

Abstract: The disclosed embodiments include a technique to obtain search results from the application of transformation operations on partial search results obtained from across internal and/or external data sources. Examples of transformation operations include arithmetic operations such as an average, mean, count, or the like. Examples of reporting transformations include join operations, statistics, sort, top head. Hence, the search results of a search query can be derived from partial search result rather than include the actual partial search results. In this case, the ordering of the search results may be nonessential. An example of a search query that requires a transformation operation is a “batch” or “reporting” search query. The related disclosed techniques involve obtaining data stored in the bid data ecosystem, and returning that data or data derived from that data.

Abstract: The performance and flexibility of a data intake and query system having capabilities extended by a fabric service (DFS) system can be improved with deployment on a cloud computing platform. The DFS system can extend the capabilities of a data intake and query system by leveraging computing assets from anywhere in a big data ecosystem to collectively execute search queries on diverse data systems regardless of whether data stores are internal of the data intake and query system and/or external data stores that are communicatively coupled to the data intake and query system over a network.