Abstract: Systems and methods are disclosed for processing and executing queries against one or more dataset. As part of processing the query, the system determines whether the query is susceptible to a significantly imbalanced partition. In the event, the query is susceptible to an imbalanced partition, the system monitors the query and determines whether to perform a multi-partitioning determination to avoid a significantly imbalanced partition.

Abstract: Systems and methods are disclosed for receiving, at a data intake and query system, a query that includes an indication to process data managed by a third-party data storage and processing system that supports a different query language than the data intake and query system. The data intake and query system identifies a third-party data storage and processing system that manages the data to be processed and generates a subquery for execution by the third-party data storage and processing system, generates instructions for one or more worker nodes to receive and process results of the subquery from the third-party data storage and processing system, and instructs the worker nodes to provide results of the processing to the data intake and query system.

Abstract: The disclosed embodiments include a method performed by a data intake and query system. The method includes receiving a search query by a search head, defining a search process for applying the search query to indexers, delegating a first portion of the search process to indexers and a second portion of the search process to intermediary node(s) communicatively coupled to the search head and the indexers. The first portion can define a search scope for obtaining partial search results of the indexers and the second portion can define operations for combining the partial search results by the intermediary node(s) to produce a combination of the partial search results. The search head then receives the combination of the partial search results, and outputs final search results for the search query, where the final search results are based on the combination of the partial search results.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets. Based on a determination that the size of multiple buckets satisfies a threshold size, the data intake and query system converts the buckets to non-editable buckets and stores the data in a remote shared storage system.

Abstract: Systems and methods are described for determining a query execution time in a data intake and query system. The system parses a query to identify different portions of the query that are executed by different components of the data intake and query system. The system determines a query execution time for the different portions of the query based on the corresponding components. Based on the query execution time of the different portions for the query, the system determines a query execution time for the query.

Abstract: Systems and methods are disclosed for executing a query that includes an indication to process data managed by an external data system. The system identifies the external data system that manages the data to be processed and generates a subquery for the external data system. The system determines a data ingest estimate and uses the data ingest estimate to generate instructions for one or more worker nodes to receive and process results of the subquery from the external data system.

Abstract: Systems and methods are disclosed for executing a query that includes an indication to process data managed by an external data system. The system identifies the external data system that manages the data to be processed and generates a subquery for the external data system indicating that the results of the subquery are to be sent to one worker node of multiple worker nodes. The system instructs the one worker node to distribute the results received from the external data system to multiple worker nodes for processing.

Abstract: Systems and methods are described for determining a quantity of records generated by a processing task of a query executed in a data intake and query. The system receives a query and identifies a processing task of the query and a quantity of records to be processed according to the query. The system determines the number of records generated by the processing task based on the number of records to be processed and a record generation estimate. The system can allocate compute resources or determine a query execution time for at least a portion of the query based on the determined quantity of records generated.

Abstract: Systems and methods are described for assigning a processing task from one component of a data intake and query system to a different component of the data intake and query system. As part of processing a query, the system can determine that a particular processing task is to be executed by a particular component of the data intake and query system. Based on the characteristics of the component that is to execute the processing task, the system can assign the task or a supplemental task to one or more other components of the data intake and query system.

Abstract: Systems and methods are disclosed for processing streaming data. The data can come from various sources. Worker nodes can be configured to process the streaming data, without delays that may be caused by indexing the data. The data can be filtered and/or transformed as it is processed. In some cases, data can be stored in a data store without transformation. The data in the data store can be accessed and processed at a later time.

Abstract: Systems and methods are described for partitioning and reducing records at ingest of a worker node. The worker node receives chunks of data from one or more indexers of a data intake and query system based on the execution of a query by the data intake and query system. The worker node assigns records to different record groups based on the content of the records. The system also assigns the record to a partition of a group of partitions. Record data of the records in a particular partition is combined. The system processes the partitions based on the query.

Abstract: Systems and methods are disclosed for generating a distributed execution model with untrusted commands. The system can receive a query, and process the query to identify the untrusted commands. The system can use data associated with the untrusted command to identify one or more files associated with the untrusted command. Based on the files, the system can generate a data structure and include one or more identifiers associated with the data structure in the distributed execution model. The system can distribute the distributed execution model to one or more nodes in a distributed computing environment for execution.

Abstract: Systems and methods are described for exporting bucket data from one or more buckets to one or more worker nodes. The system can identify data from different bucket data from buckets stored in a data intake and query system that is to be processed by one or more worker nodes. The system can allocate one or more execution resources, such as a processing pipeline, to process and export the bucket data from the buckets. The system can assign bucket data corresponding to individual buckets to the execution resource based on a bucket distribution policy. The indexer can export the bucket data to the worker nodes for further processing based on the bucket data-execution resource assignment.

Abstract: A device has a light guide portion and a light collector portion. The light guide portion is cup shaped. The wall of the cup has a cross section defined by sections of two ellipses disposed in a predefined manner with each other. The light collector portion is also cup-shaped, inverted with reference to the light guide portion, by a section of an ellipse and straight lines defined with reference to the light guide portion. The device radiates an annular illumination at the rim of the cup through total internal reflection of light from an LED, collected by the light collector portion. The device is made of a clear, colourless, substantially transparent material by injection moulding, one example being Polycarbonate. An annular light source system and a fundus camera using such a system are also disclosed.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched and stores a copy of buckets in memory associated with one or more search nodes. A search node performs a search on buckets residing in its memory.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. An indexing system of the data intake and query system receives data and stores at least a portion of it in buckets, which are then stored in a shared storage system. The indexing system merges multiple buckets to generate merged buckets and uploads the merged buckets to the shared storage system.

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched. The data intake and query system performs a hash on bucket identifiers of the identified buckets to identify search nodes to search the buckets.

Abstract: Disclosed is a data fabric service system that can be implemented in a distributed computer network, such as a data intake and query system. The data index and query system can receive a search query and define a search scheme for applying the search query on distributed data storage systems including internal data storage and external data storage. The data index and query system may provide a portion of the search scheme to a search service of the data fabric service system, which can cause worker nodes of the data fabric service system to perform various functions—including applying the search query to the external data storage based on the portion of the search scheme in order to obtain search results.

Abstract: Systems and methods are disclosed for processing and executing queries against one or more dataset. As part of processing the query, the system determines whether the query is susceptible to a significantly imbalanced partition. In the event, the query is susceptible to an imbalanced partition, the system monitors the query and determines whether to perform a multi-partitioning determination to avoid a significantly imbalanced partition.

Abstract: Systems and methods are described for establishing and managing components of a distributed computing framework implemented in a data intake and query system. The distributed computing framework may include a master and a plurality of worker nodes. The master may selectively operate on a search head captain that is chosen from the search heads of the data intake and query system. The search head captain may distribute configuration information for the master and the distributed computing framework to the other search heads, which in turn, may distribute that configuration information to indexers of the data intake and query system. Worker nodes may be selectively activated for operation on the indexers based on the configuration information, and the worker nodes may additionally use the configuration information to contact the master and join the distributed computing framework.