Abstract: In a general aspect, an observability pipeline system includes a pack data processing engine. In some aspects, an observability pipeline system includes data processing engines that are configured according to system default configuration settings and system local configuration settings. A pack file received from a remote computer system contains routes, pipelines, and pack default configuration settings. A pack data processing engine includes the routes and pipelines from the pack file. Pack local configuration settings, defined for the pack data processing engine, inherit at least one of the system default configuration settings and at least one of the pack default configuration settings. The pack local configuration settings are isolated from the system local configuration settings. When pipeline data is processed in the observability pipeline system on the computer system, the pack data processing engine is applied to the pipeline data.

Abstract: In some aspects, search functionality is provided in an observability analysis system. In some implementations, a request is received to search a dataset of an observability analysis system, the request including a search query. A hash is generated that represents the request, wherein generating the hash includes applying a hash function to a query execution plan determined from the search query and at least one parameter corresponding to the request. A determination is made whether the request satisfies a set of one or more query reuse criteria that includes a criterion that is satisfied when the hash matches a previous hash associated with a previous request to search the dataset. In accordance with determining that the set of one or more query reuse criteria is satisfied, a set of query results of the previous request to search the dataset is reused.

Abstract: In some aspects, an edge-based data collection system discovers, collects, processes, and forwards data in an observability pipeline system. In some implementations, an edge agent of the observability pipeline system runs on a computer node. The edge agent receives a process discovery filter configured to identify target processes running on computer nodes, selects one or more matching target processes by applying the process discovery filter, monitors activity of the one or more matching target processes, and processes the activity data to generate output data representing the activity of the one or more matching target processes.

Abstract: In some aspects, search functionality is provided in an observability pipeline system. In some implementations, a method of searching remotely-stored data includes receiving a search query at a computer node residing at a node geolocation, the search query representing a request to search data stored at a storage geolocation; identifying compute geolocations each including computing resources, each of the compute geolocations being distinct from the node geolocation and the storage geolocation; obtaining latency data including latency values for the compute geolocations; selecting one of the compute geolocations based on the latency data; by operation of a coordinator agent on the computer node, initiating a dynamic computing resource at the selected compute geolocation, and receiving search results identified by the dynamic computing resource.

Abstract: In some aspects, search functionality is provided in an observability pipeline system. In some implementations, a search query is received at a computer node from a leader role of an observability pipeline system. The search query represents a request to search event data at the computer node and includes a first search operator that specifies a system state context criterion and a second search operator that specifies an event criterion. An observability pipeline process is configured according to the search query. Search results are generated based on applying the observability pipeline process at the computer node. A current system state of the computer node that matches the first search operator is determined; and a subset of event data on the computer node that matches the second search operator is identified. The search results including the subset of event data are sent to the leader role.

Abstract: In some aspects, search functionality is provided in an observability pipeline system. In some implementations, a search method includes receiving a search query from a leader role in an observability pipeline system. The search query represents a request to search event data at a computer resource. An observability pipeline process is configured to perform a search according to the search query, and search results are obtained based on applying the observability pipeline process to the event data. The search results include events from the event data. Provenance information is obtained for each of the events. The provenance information for each event includes an identification of the computer resource and a link to the computer resource. Augmented search results are generated by associating the provenance information with the respective events, and the augmented search results are communicated to the leader role.

Abstract: In some aspects, search functionality is provided in an observability analysis system. In some implementations, search results are obtained from prior search queries, the search results include a subset of event data generated in a computing environment. Metadata are generated by analyzing the event data to identify a plurality of fields and values for each of the plurality of fields. A partial search query is received through a user interface at a first computer node; the partial search query represents at least a portion of a request to search the event data. Autocomplete suggestions are generated at the first computer node based on the partial search query and the metadata. Each of the autocomplete suggestions includes a suggested field selected from the plurality of fields, a suggested value selected from the values, or both. At least one of the autocomplete suggestions is displayed in the user interface.

Abstract: Data payloads from an external data storage system are processed in an observability pipeline system. In some aspects, the observability pipeline system defines a leader role and worker roles. The leader role generates a data discovery task based on configuration information for a data collection task. A worker role executes the data discovery task, which includes communicating with an external data storage system to identify a data payload that is stored on the external data storage system and contains event data that meet event filter criteria. The leader role generates data collection tasks based on the data payload. Worker roles execute the data collection tasks. Executing a data collection task includes: communicating with the external data storage system to obtain a subset of filtered event data from the data payload; and streaming the subset of filtered event data to an observability pipeline process.

Abstract: In some aspects, search functionality is provided in an observability pipeline system. In some implementations, a search method includes receiving a search query from a leader role in an observability pipeline system. The search query represents a request to search event data at a computer resource. An observability pipeline process is configured to perform a search according to the search query, and search results are obtained based on applying the observability pipeline process to the event data. The search results include events from the event data. Provenance information is obtained for each of the events. The provenance information for each event includes an identification of the computer resource and a link to the computer resource. Augmented search results are generated by associating the provenance information with the respective events, and the augmented search results are communicated to the leader role.

Abstract: Load balancing processes are performed in an observability pipeline system comprising a plurality of computing resources. In some aspects, the observability pipeline system defines a leader role and worker roles. A plurality of computing jobs each include computing tasks associated with event data. The leader role dispatches the computing tasks to the worker roles according to a least in-flight task dispatch criteria, which includes iteratively: identifying an available worker role; identifying one or more incomplete computing jobs; selecting, from the one or more incomplete computing jobs, a computing job that has the least number of in-flight computing tasks currently being executed in the observability pipeline system; identifying a next computing task from the selected computing job; and dispatching the next computing task to the available worker role. The worker roles execute the computing tasks by applying an observability pipeline process to the event data associated with the respective computing task.

Abstract: In some aspects, data collection functions are interposed to generate input data for an observability pipeline system. In some aspects, a data collection function is made available to an application running on a computer system, with the data collection function having the same name as an original function referenced by the application. In response to a call to the original function, the data collection function is executed and data is extracted from the application. The original function is then executed. A reporting thread of the application is executed; executing the reporting thread generates observability pipeline input data by formatting the extracted data and sends the observability pipeline input data from the computer system to an observability pipeline system.

Abstract: In a general aspect, an observability pipeline system includes a pack data processing engine. In some aspects, an observability pipeline system includes data processing engines that are configured according to system default configuration settings and system local configuration settings. A pack file received from a remote computer system contains routes, pipelines, and pack default configuration settings. A pack data processing engine includes the routes and pipelines from the pack file. Pack local configuration settings, defined for the pack data processing engine, inherit at least one of the system default configuration settings and at least one of the pack default configuration settings. The pack local configuration settings are isolated from the system local configuration settings. When pipeline data is processed in the observability pipeline system on the computer system, the pack data processing engine is applied to the pipeline data.

Abstract: In some aspects, an edge-based data collection system discovers, collects, processes, and forwards data in an observability pipeline system. In some implementations, an edge agent of the observability pipeline system runs on a computer node. The edge agent identifies processes running on the computer node; identifies files on the computer node that the processes have opened for writing; accesses log discovery parameters of the observability pipeline system; selects a plurality of files from the identified files according to the log discovery parameters; generates a list of discovered log files that includes a path and a name for each of the plurality of files; adds the list of discovered log files to a list of monitored log files to be monitored by the observability pipeline system; and then monitors the plurality of files to generate input for the observability pipeline system.

Abstract: In some aspects, search functionality is provided in an observability pipeline system. In some implementations, a search method includes receiving a search query from a leader role in an observability pipeline system. The search query represents a request to search event data at a computer resource. An observability pipeline process is configured to perform a search according to the search query, and search results are obtained based on applying the observability pipeline process to the event data. The search results include events from the event data. Provenance information is obtained for each of the events. The provenance information for each event includes an identification of the computer resource and a link to the computer resource. Augmented search results are generated by associating the provenance information with the respective events, and the augmented search results are communicated to the leader role.

Abstract: In some aspects, search functionality is provided in an observability pipeline system. In some implementations, a method of searching remotely-stored data includes receiving a search query at a computer node residing at a node geolocation, the search query representing a request to search data stored at a storage geolocation; identifying compute geolocations each including computing resources, each of the compute geolocations being distinct from the node geolocation and the storage geolocation; obtaining latency data including latency values for the compute geolocations; selecting one of the compute geolocations based on the latency data; by operation of a coordinator agent on the computer node, initiating a dynamic computing resource at the selected compute geolocation, and receiving search results identified by the dynamic computing resource.

Abstract: In some aspects, an interactive graphical user interface displays search data for an observability pipeline system. In some aspects, a method includes obtaining search results including events identified based on searching data generated by an observability pipeline system. The method includes identifying time bins based on the search results; generating first histogram data based on the time bins and the events; and generating second histogram data based on the time bins and the events. The method includes generating a graphical user interface including a first histogram representing the first histogram data and including a first set of bins, and a second histogram representing the second histogram data and including a second set of bins. The method includes updating the graphical user interface to include a visual indication of a selected bin in the first histogram and a visual indication of a corresponding bin in the second histogram.

Abstract: Load balancing processes are performed in an observability pipeline system comprising a plurality of computing resources. In some aspects, the observability pipeline system defines a leader role and worker roles. A plurality of computing jobs each include computing tasks associated with event data. The leader role dispatches the computing tasks to the worker roles according to a least in-flight task dispatch criteria, which includes iteratively: identifying an available worker role; identifying one or more incomplete computing jobs; selecting, from the one or more incomplete computing jobs, a computing job that has the least number of in-flight computing tasks currently being executed in the observability pipeline system; identifying a next computing task from the selected computing job; and dispatching the next computing task to the available worker role. The worker roles execute the computing tasks by applying an observability pipeline process to the event data associated with the respective computing task.

Abstract: In some aspects, an edge-based data collection system discovers, collects, processes, and forwards data in an observability pipeline system. In some implementations, an edge agent of the observability pipeline system runs on a computer node. The edge agent identifies processes running on the computer node; identifies files on the computer node that the processes have opened for writing; accesses log discovery parameters of the observability pipeline system; selects a plurality of files from the identified files according to the log discovery parameters; generates a list of discovered log files that includes a path and a name for each of the plurality of files; adds the list of discovered log files to a list of monitored log files to be monitored by the observability pipeline system; and then monitors the plurality of files to generate input for the observability pipeline system.