Abstract: Embodiments are directed towards a method for searching data. The method comprises generating an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name. Furthermore, the method comprises generating results to the incoming search query based on the field searchable datastore, wherein the field searchable datastore is directly searchable by the field name.

Abstract: Embodiments are directed towards a method for searching data. The method comprises providing an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name and evaluating the incoming search query. Furthermore, responsive to the evaluating, the method comprises determining results for the incoming search query using both of the field searchable datastore and the inverted index.

Abstract: Embodiments are directed towards a method for searching data. The method comprises providing an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name and evaluating the incoming search query. Furthermore, responsive to the evaluating, the method comprises determining results for the incoming search query using the field searchable datastore or the inverted index.

Abstract: Embodiments are directed towards a method for searching data. The method comprises generating an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name. Furthermore, the method comprises generating results to the incoming search query based on the field searchable datastore, wherein the field searchable datastore is directly searchable by the field name.

Abstract: Embodiments are directed towards a method for searching data. The method comprises providing an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name and evaluating the incoming search query. Furthermore, responsive to the evaluating, the method comprises determining results for the incoming search query using the field searchable datastore or the inverted index.

Abstract: Embodiments are directed towards managing within a cluster environment having a plurality of indexers for data storage using redundancy the data being managed using a generation identifier, such that a primary indexer is designated for a given generation of data. When a master device for the cluster fails, data may continue to be stored using redundancy, and data searches performed may still be performed.

Abstract: Techniques are described for managing data within a multi-site clustered data intake and query system. A data intake and query system as described herein generally refers to a system for collecting, retrieving, and analyzing data. In this context, a clustered data intake and query system generally refers to a system environment that is configured to provide data redundancy and other features that improve the availability of data stored by the system. For example, a clustered data intake and query system may be configured to store multiple copies of data stored by the system across multiple components such that recovery from a failure of one or more of the components is possible by using copies of the data stored elsewhere in the cluster.

Abstract: Embodiments are directed towards managing within a cluster environment having a plurality of indexers for data storage using redundancy the data being managed using a generation identifier, such that a primary indexer is designated for a given generation of data. When a master device for the cluster fails, data may continue to be stored using redundancy, and data searches performed may still be performed.

Abstract: Embodiments are directed towards managing and tracking item identification of a plurality of items to determine if an item is a new or existing item, where an existing item has been previously processed. In some embodiments, two or more item identifiers may be generated. In one embodiment, generating the two or more item identifiers may include analyzing the item using a small item size characteristic, a compressed item, or for an identifier collision. The two or more item identifiers may be employed to determine if the item is a new or existing item. In one embodiment, the two or more item identifiers may be compared to a record about an existing item to determine if the item is a new or existing item. If the item is an existing item, then the item may be further processed to determine if the existing item has actually changed.

Abstract: Embodiments are directed towards previewing results generated from indexing data raw data before the corresponding index data is added to an index store. Raw data may be received from a preview data source. After an initial set of configuration information may be established, the preview data may be submitted to an index processing pipeline. A previewing application may generate preview results based on the preview index data and the configuration information. The preview results may enable previewing how the data is being processed by the indexing application. If the preview results are not acceptable, the configuration information may be modified. The preview application enables modification of the configuration information until the generated preview results may be acceptable. If the configuration information is acceptable, the preview data may be processed and indexed in one or more index stores.

Abstract: Embodiments are directed towards managing and tracking item identification of a plurality of items to determine if an item is a new or existing item, where an existing item has been previously processed. In some embodiments, two or more item identifiers may be generated. In one embodiment, generating the two or more item identifiers may include analyzing the item using a small item size characteristic, a compressed item, or for an identifier collision. The two or more item identifiers may be employed to determine if the item is a new or existing item. In one embodiment, the two or more item identifiers may be compared to a record about an existing item to determine if the item is a new or existing item. If the item is an existing item, then the item may be further processed to determine if the existing item has actually changed.

Abstract: Embodiments are directed towards receiving and processing search queries directed towards relatively large sets of data. The data is stored in a record based datastore. From the stored data, field names, corresponding field values, and posting values may be determined. Posting values may be employed to locate records in the datastore that include the field names and field values. The field names, field values, and posting values may be employed to generate a lexicon. If queries are received, a lexicon query processor may employ the lexicon separate from the datastore to generate responses to the received queries. Queries may include clauses that may be processed using the lexicon separate from the datastore, such as, where clause expressions, group-by clause expressions, aggregation functions, or the like. A time values array may be used to enable queries to process group-by-time expressions that may return results grouped into sub-sets based on time ranges.

Abstract: Embodiments are directed towards previewing results generated from indexing data raw data before the corresponding index data is added to an index store. Raw data may be received from a preview data source. After an initial set of configuration information may be established, the preview data may be submitted to an index processing pipeline. A previewing application may generate preview results based on the preview index data and the configuration information. The preview results may enable previewing how the data is being processed by the indexing application. If the preview results are not acceptable, the configuration information may be modified. The preview application enables modification of the configuration information until the generated preview results may be acceptable. If the configuration information is acceptable, the preview data may be processed and indexed in one or more index stores.

Abstract: Embodiments are directed towards a method for searching data. The method comprises generating an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name. Furthermore, the method comprises generating results to the incoming search query based on the field searchable datastore, wherein the field searchable datastore is directly searchable by the field name.

Abstract: Embodiments are directed towards a method for searching data. The method comprises providing an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name and evaluating the incoming search query. Furthermore, responsive to the evaluating, the method comprises determining results for the incoming search query using the field searchable datastore or the inverted index.

Abstract: Embodiments are directed towards a method for searching data. The method comprises providing an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name and evaluating the incoming search query. Furthermore, responsive to the evaluating, the method comprises determining results for the incoming search query using both of the field searchable datastore and the inverted index.

Abstract: Embodiments are directed towards previewing results generated from indexing data raw data before the corresponding index data is added to an index store. Raw data may be received from a preview data source. After an initial set of configuration information may be established, the preview data may be submitted to an index processing pipeline. A previewing application may generate preview results based on the preview index data and the configuration information. The preview results may enable previewing how the data is being processed by the indexing application. If the preview results are not acceptable, the configuration information may be modified. The preview application enables modification of the configuration information until the generated preview results may be acceptable. If the configuration information is acceptable, the preview data may be processed and indexed in one or more index stores.

Abstract: Embodiments are directed towards previewing results generated from indexing data raw data before the corresponding index data is added to an index store. Raw data may be received from a preview data source. After an initial set of configuration information may be established, the preview data may be submitted to an index processing pipeline. A previewing application may generate preview results based on the preview index data and the configuration information. The preview results may enable previewing how the data is being processed by the indexing application. If the preview results are not acceptable, the configuration information may be modified. The preview application enables modification of the configuration information until the generated preview results may be acceptable. If the configuration information is acceptable, the preview data may be processed and indexed in one or more index stores.

Abstract: Embodiments are directed towards previewing results generated from indexing data raw data before the corresponding index data is added to an index store. Raw data may be received from a preview data source. After an initial set of configuration information may be established, the preview data may be submitted to an index processing pipeline. A previewing application may generate preview results based on the preview index data and the configuration information. The preview results may enable previewing how the data is being processed by the indexing application. If the preview results are not acceptable, the configuration information may be modified. The preview application enables modification of the configuration information until the generated preview results may be acceptable. If the configuration information is acceptable, the preview data may be processed and indexed in one or more index stores.

Abstract: Embodiments are directed towards previewing results generated from indexing data raw data before the corresponding index data is added to an index store. Raw data may be received from a preview data source. After an initial set of configuration information may be established, the preview data may be submitted to an index processing pipeline. A previewing application may generate preview results based on the preview index data and the configuration information. The preview results may enable previewing how the data is being processed by the indexing application. If the preview results are not acceptable, the configuration information may be modified. The preview application enables modification of the configuration information until the generated preview results may be acceptable. If the configuration information is acceptable, the preview data may be processed and indexed in one or more index stores.