Abstract: Provided are systems and methods for verifying user credentials for performing a search. Verifying user credentials include receiving a search request at a search server, determining, at the search server, whether a set of user credentials of a user has been updated within a threshold period of time. The set of user credentials are received from an identity provider server and cached at the search server. Responsive to determining that the cached set of user credentials have not been updated within the threshold period of time, the identity provider server is queried for a current set of user credentials associated with the user. The current set of user credentials from the identity provider server, and used to determine that the user is authorized to perform the search. The search of the datastore is launched responsive to determining that the user is authorized.

Abstract: Disclosed herein is a data estimation technique for a data intake and query system. The system receives user inputs indicative that a first data source is to be the subject of a storage related estimate. The system receives a first plurality of events generated by the first data source. The system indexes only a sample of the received first plurality of events, based on a sampling criterion, where the sample is fewer than all of the first plurality of events. The system generates the storage related estimate based on at least some of the first plurality of events, and causes an indication of the estimate to be output to a user.

Abstract: The present invention is related to a method for providing dynamic indexer discovery. The method comprises receiving, from an index manager, a status indication associated with a plurality of indexers, wherein each of the plurality of indexers indexes events of raw machine-generated data received from a plurality of data collectors. The method further comprises determining a weight associated with each of the plurality of indexers and selecting an indexer from the plurality of indexers. Subsequently, the method comprises allocating data to the indexer in accordance with a respective weight assigned to the indexer and transmitting the allocated data to the indexer.

Abstract: Provided are systems and methods for verifying user credentials for performing a search. In one embodiment, a method can be provided that includes receiving a request to perform a search of machine generated data comprising time stamped events that is associated with a user, determining whether a set of cached user credentials has been updated within a period of time, querying, in response to determining that the credentials for the user have not been updated within the period of time, an identity provider server for a current set of user credentials associated with the user, receiving the current set of user credentials, determining whether the user has privileges to perform the search based at least in part on the set of user credentials, and causing, in response to determining that the user has privileges to perform the search, the search to be performed to identify one or more of the events that are responsive to the search.

Abstract: Disclosed herein is a data estimation technique for a data intake and query system. The system receives user inputs indicative that a first data source is to be the subject of a storage related estimate. The system receives a first plurality of events generated by the first data source. The system indexes only a sample of the received first plurality of events, based on a sampling criterion, where the sample is fewer than all of the first plurality of events. The system generates the storage related estimate based on at least some of the first plurality of events, and causes an indication of the estimate to be output to a user.

Abstract: The present invention is related to a method for providing dynamic indexer discovery. The method comprises receiving, from an index manager, a status indication associated with a plurality of indexers, wherein each of the plurality of indexers indexes events of raw machine-generated data received from a plurality of data collectors. The method further comprises determining a weight associated with each of the plurality of indexers and selecting an indexer from the plurality of indexers. Subsequently, the method comprises allocating data to the indexer in accordance with a respective weight assigned to the indexer and transmitting the allocated data to the indexer.

Abstract: Disclosed herein is a data estimation technique for a data intake and query system. The system receives user inputs indicative that a first data source is to be the subject of a storage related estimate. The system receives a first plurality of events generated by the first data source. The system indexes only a sample of the received first plurality of events, based on a sampling criterion, where the sample is fewer than all of the first plurality of events. The system generates the storage related estimate based on at least some of the first plurality of events, and causes an indication of the estimate to be output to a user.

Abstract: The present disclosure is directed to providing dynamic indexer discovery. An index manager, which may also be known as a cluster master, is configured to track the statuses and capabilities of indexers and provide the statuses and capabilities obtained from the indexers to data collectors, such as forwarders. The data collectors may use the statuses and capabilities associated with the indexers to load balance transmission of data to the indexers. Dynamic indexer discovery may eliminate the need to manually reconfigure data collectors when the status of an indexer changes because the information may be obtained from the index manager without the need to reinitialize the data collectors.

Abstract: Disclosed herein is a data estimation technique for a data intake and query system. The system receives user inputs indicative that a first data source is to be the subject of a storage related estimate. The system receives a first plurality of events generated by the first data source. The system indexes only a sample of the received first plurality of events, based on a sampling criterion, where the sample is fewer than all of the first plurality of events. The system generates the storage related estimate based on at least some of the first plurality of events, and causes an indication of the estimate to be output to a user.

Abstract: Disclosed herein are a method, apparatus and system that authenticate a first data forwarder, of a distributed machine data acquisition and search system (MDASS), to a node that regulates traversal of a firewall that protects a protected environment within which the data forwarder operates. The authentication may be performed by using a SOCKS5 authentication process. The method further includes, only after successful completion of the SOCKS5 authentication process, establishing a first connection, through a network, between the first data forwarder and a first indexer of the distributed MDASS, where the first indexer operates outside the protected environment, and sending machine data acquired by the first data forwarder from a machine data source, to the first indexer via the first connection.

Abstract: Disclosed herein are a method, apparatus and system that authenticate a first data forwarder, of a distributed machine data acquisition and search system (MDASS), to a node that regulates traversal of a firewall that protects a protected environment within which the data forwarder operates. The authentication may be performed by using a SOCKS5 authentication process. The method further includes, only after successful completion of the SOCKS5 authentication process, establishing a first connection, through a network, between the first data forwarder and a first indexer of the distributed MDASS, where the first indexer operates outside the protected environment, and sending machine data acquired by the first data forwarder from a machine data source, to the first indexer via the first connection.

Abstract: The present disclosure is directed to providing dynamic indexer discovery. An index manager, which may also be known as a cluster master, is configured to track the statuses and capabilities of indexers and provide the statuses and capabilities obtained from the indexers to data collectors, such as forwarders. The data collectors may use the statuses and capabilities associated with the indexers to load balance transmission of data to the indexers. Dynamic indexer discovery may eliminate the need to manually reconfigure data collectors when the status of an indexer changes because the information may be obtained from the index manager without the need to reinitialize the data collectors.

Abstract: Provided are systems and methods for verifying user credentials for performing a search. In one embodiment, a method can be provided that includes receiving a request to perform a search of machine generated data comprising time stamped events that is associated with a user, determining whether a set of cached user credentials has been updated within a period of time, querying, in response to determining that the credentials for the user have not been updated within the period of time, an identity provider server for a current set of user credentials associated with the user, receiving the current set of user credentials, determining whether the user has privileges to perform the search based at least in part on the set of user credentials, and causing, in response to determining that the user has privileges to perform the search, the search to be performed to identify one or more of the events that are responsive to the search.

Abstract: According to various embodiments, techniques are described for managing data within a multi-site clustered data intake and query system. A data intake and query system as described herein generally refers to a system for collecting, retrieving, and analyzing data. In this context, a clustered data intake and query system generally refers to a system environment that is configured to provide data redundancy and other features that improve the availability of data stored by the system. For example, a clustered data intake and query system may be configured to store multiple copies of data stored by the system across multiple components such that recovery from a failure of one or more of the components is possible by using copies of the data stored elsewhere in the cluster.

Abstract: According to various embodiments, techniques are described for managing data within a multi-site clustered data intake and query system. A data intake and query system as described herein generally refers to a system for collecting, retrieving, and analyzing data. In this context, a clustered data intake and query system generally refers to a system environment that is configured to provide data redundancy and other features that improve the availability of data stored by the system. For example, a clustered data intake and query system may be configured to store multiple copies of data stored by the system across multiple components such that recovery from a failure of one or more of the components is possible by using copies of the data stored elsewhere in the cluster.