Interactive Graphical User Interface for Displaying Search Data for an Observability Pipeline System

Abstract

In some aspects, an interactive graphical user interface displays search data for an observability pipeline system. In some aspects, a method includes obtaining search results including events identified based on searching data generated by an observability pipeline system. The method includes identifying time bins based on the search results; generating first histogram data based on the time bins and the events; and generating second histogram data based on the time bins and the events. The method includes generating a graphical user interface including a first histogram representing the first histogram data and including a first set of bins, and a second histogram representing the second histogram data and including a second set of bins. The method includes updating the graphical user interface to include a visual indication of a selected bin in the first histogram and a visual indication of a corresponding bin in the second histogram.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 63/344,864, filed May 23, 2022, entitled “Observability Platform Search;” U.S. Provisional Patent Application No. 63/414,762, filed Oct. 10, 2022, entitled “Observability Platform Search;” U.S. Provisional Patent Application No. 63/419,632, filed Oct. 26, 2022, entitled “Observability Platform Search;” and U.S. Provisional Application No. 63/423,264, filed Nov. 7, 2022, entitled “Observability Platform Search.” Each of the above-referenced priority documents are incorporated herein by reference.

BACKGROUND

The following description relates to displaying search data for an observability pipeline system in an interactive graphical user interface.

Observability pipelines are used to route and process data in a number of contexts. For example, observability pipelines can provide unified routing of various types of machine data to multiple destinations, while adapting data shapes and controlling data volumes. In some implementations, observability pipelines allow an organization to interrogate machine data from its environment without knowing in advance the questions that will be asked. Observability pipelines may also provide monitoring and alerting functions, which allow systematic observation of data for known conditions that require specific action or attention.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing aspects of an example computing environment that includes an observability pipeline system.

FIG. 2 is a block diagram showing aspects of an example observability pipeline system deployed in a worker role.

FIG. 3 is a flow chart showing aspects of an example process for displaying search data.

FIGS. 4A-4B include schematic diagrams showing a concurrent display of results from two search queries.

FIG. 5 is a block diagram showing an example computer system.

DETAILED DESCRIPTION

In some instances, a computer system can be configured to generate an interactive graphical user interface (GUI) that displays search results for an observability pipeline system. In some examples, the GUI includes a rich, interactive data visualization that can be customized by the user. For instance, the GUI may allow users to customize the data display by focusing on specific aspects of the search results; adjusting the visual representation of the search results; adjusting the level of detail displayed; etc.

In some implementations, the search results can be structured as histogram data and presented in one or more histograms along with other information in the GUI. In some implementations, the search results represent events that have been processed by the observability pipeline system, and the histograms in the GUI represent distinct subsets of the events. For instance, the histograms can represent event sources and event destinations. The histogram data can be formatted and presented in a way that allows a user to explore the search results and potentially discover meaningful insights. For instance, the same bin parameters (e.g., the same time bins) can be used to generate each of the histograms, so that corresponding bins in each histogram represent events associated with the same time period.

In some implementations, the GUI is responsive to user interactions (e.g., a user's selection of a histogram bin) such that the histogram and other information in the GUI is updated in response to a user's interactions with the GUI. As an example, when a bin in a histogram is selected in the GUI, a corresponding bin (e.g., a bin corresponding to events in the same time period) in another histogram can be highlighted in the GUI, and event data for the selected histogram (and the corresponding bin, in some cases) can be displayed or highlighted in the GUI. As another example, the GUI can include data panels that show event data describing events represented in a histogram shown in the GUI, and the data panels can be updated to highlight event data corresponding to a bin in the histogram (e.g., a selected bin). In some cases, data panels can expand or otherwise update in response to a user selection.

In some implementations, the search data can be obtained by search functionality that is integrated with an observability pipeline system to query data at the source, in-motion when flowing through an observability pipeline system, or stored at a destination. In some instances, search functionality can be performed on structured, semi-structured, or unstructured data sets, or other types of data. The search functionality can be performed based on any terms, patterns, value/pairs, and any data type. In some implementations, search functionality allows searches to be performed across multiple data lake technologies, data stream environments, and existing tools, providing visibility across the entire observability pipeline system.

The systems and techniques described here can provide technical advantages and improvements over existing technologies. In some implementations, the methods and technologies presented here allow users to drill down into specific events or event clusters based on a characteristic of interest (e.g., an event property or time range, etc.). The methods and techniques presented here can provide dynamic visibility into system and application logs and metrics, system state, configuration files, and other information. In some cases, a graphical user interface can improve understanding and develop deeper insights based on complex search results, for instance, by presenting information in a more intuitive and responsive format. Accordingly, aspects of the systems and techniques described here can be used to improve the operation of computer systems, information and data management systems, observability pipeline systems, and other classes of technology. In some implementations, an interactive graphical user interface can provide other technical advantages.

FIG. 1 is a block diagram showing aspects of an example computing environment 100 that includes an observability pipeline system 110. In addition to the observability pipeline system 110, the example computing environment 100 shown in FIG. 1 includes data sources 102, data destinations 104, data storage 106, network 108, and a user device 120. The data sources 102 includes an application 116. The computing environment 100 may include additional or different features, and the elements of the computing environment 100 may be configured to operate as described with respect to FIG. 1 or in another manner.

In some implementations, the computing environment 100 contains the computing infrastructure of a business enterprise, an organization or another type of entity or group of entities. During operation, various data sources 102 in an organization's computing infrastructure produce volumes of machine data that contain valuable or useful information. These data sources can include applications 116 and other types of computer resources. The machine data may include data generated by the organization itself, data received from external entities, or a combination. By way of example, the machine data can include network packet data, sensor data, application program data, observability data, and other types of data. Observability data can include, for example, system logs, error logs, stack traces, system performance data, or any other data that provides information about computing infrastructure and applications (e.g., performance data and diagnostic information). The observability pipeline system 110 can receive and process the machine data generated by the data sources 102. For example, the machine data can be processed to diagnose performance problems, monitor user interactions, and to derive other insights about the computing environment 100. Generally, the machine data generated by the data sources 102 does not have a common format or structure, and the observability pipeline system 110 can generate structured output data having a specified form, format, or type. The output generated by the observability pipeline system can be delivered to data destinations 104, data storage 106, or both. In some cases, the data delivered to the data storage 106 includes the original machine data that was generated by the data sources 102, and the observability pipeline system 110 can later retrieve and process the machine data that was stored on the data storage 106.

In general, the observability pipeline system 110 can provide a number of services for processing and structuring machine data for an enterprise or other organization. In some instances, the observability pipeline system 110 provides schema-agnostic processing, which can include, for example, enriching, aggregating, sampling, suppressing, or dropping fields from nested structures, raw logs, and other types of machine data. The observability pipeline system 110 may also function as a universal adapter for any type of machine data destination. For example, the observability pipeline system 110 may be configured to normalize, denormalize, and adapt schemas for routing data to multiple destinations. The observability pipeline system 110 may also provide protocol support, allowing enterprises to work with existing data collectors, shippers, and agents, and providing simple protocols for new data collectors. In some cases, the observability pipeline system 110 can test and validate new configurations and reproduce how machine data was processed. The observability pipeline system 110 may also have responsive configurability, including rapid reconfiguration to selectively allow more verbosity with pushdown to data destinations or collectors. The observability pipeline system 110 may also provide reliable delivery (e.g., at least once delivery semantics) to ensure data integrity.

The data sources 102, data destinations 104, data storage 106, observability pipeline system 110, and the user device 120 are each implemented by one or more computer systems that have computational resources (e.g., hardware, software, firmware) that are used to communicate with each other and to perform other operations. For example, each computer system may be implemented as the example computer system 600 shown in FIG. 6 or components thereof. In some implementations, computer systems in the computing environment 100 can be implemented in various types of devices, such as, for example, laptops, desktops, workstations, smartphones, tablets, sensors, routers, mobile devices, Internet of Things (IoT) devices, and other types of devices. Aspects of the computing environment 100 can be deployed on private computing resources (e.g., private enterprise servers, etc.), cloud-based computing resources, or a combination thereof. Moreover, the computing environment 100 may include or utilize other types of computing resources, such as, for example, edge computing, fog computing, etc.

The data sources 102, data destinations 104, data storage 106, observability pipeline system 110, and the user device 120 and possibly other computer systems or devices communicate with each other over the network 108. The example network 108 can include all or part of a data communication network or another type of communication link. For example, the network 108 can include one or more wired or wireless connections, one or more wired or wireless networks, or other communication channels. In some examples, the network 108 includes a Local Area Network (LAN), a Wide Area Network (WAN), a private network, an enterprise network, a Virtual Private Network (VPN), a public network (such as the Internet), a peer-to-peer network, a cellular network, a Wi-Fi network, a Personal Area Network (PAN) (e.g., a Bluetooth low energy (BTLE) network, a ZigBee network, etc.) or other short-range network involving machine-to-machine (M2M) communication, or another type of data communication network.

The data sources 102 can include multiple user devices, servers, sensors, routers, firewalls, switches, virtual machines, containers, or a combination of these and other types of computer devices or computing infrastructure components. The data sources 102 detect, monitor, create, or otherwise produce machine data during their operation. The machine data is provided to the observability pipeline system 110 through the network 108. In some cases, the machine data is streamed to the observability pipeline system 110 as pipeline input data.

The data sources 102 can include data sources designated as push sources (examples include Splunk TCP, Splunk HEC, Syslog, Elasticsearch API, TCP JSON, TCP Raw, HTTP/S, Raw HTTP/S, Kinesis Firehose, SNMP Trap, Metrics, and others), pull sources (examples include Kafka, Kinesis Streams, SQS, S3, Google Cloud Pub/Sub, Azure Blob Storage, Azure Event Hubs, Office 365 Services, Office 365 Activity, Office 365 Message Trace, Prometheus, and others), and other types of data sources. The data sources 102 can also include other applications 116.

In the example shown in FIG. 1, the application 116 includes a collection of computer instructions that constitute a computer program such as the computer program shown in FIG. 6. The computer instructions reside in memory 620 and execute on a processor 610. The computer instructions can be compiled or interpreted. An application 116 can be contained in a single module or can be statically or dynamically linked with other libraries. The libraries can be provided by the operating system or the application provider.

The data destinations 104 can include multiple user devices, servers, databases, analytics systems, data storage systems, or a combination of these and other types of computer systems. The data destinations 104 can include, for example, log analytics platforms, time series databases (TSDBs), distributed tracing systems, security information and event management (SIEM) or user behavior analytics (UBA) systems, and event streaming systems or data lakes (e.g., a system or repository of data stored in its natural/raw format). The observability pipeline output data produced by the observability pipeline system 110 can be communicated to the data destinations 104 through the network 108.

The data storage 106 can include multiple user devices, servers, databases, or a combination of these and other types of data storage systems. Generally, the data storage 106 can operate as a data source or a data destination (or both) for the observability pipeline system 110. In some examples, the data storage 106 includes a local or remote filesystem location, a network file system (NFS), Amazon S3 buckets, S3-compatible stores, other cloud-based data storage systems, enterprise databases, systems that provide access to data through REST API calls or custom scripts, or a combination of these and other data storage systems. The observability pipeline output data, which may include the machine data from the data sources 102 as well as data analytics and other output from the observability pipeline system 100, can be communicated to the data storage 106 through the network 108.

The observability pipeline system 110 may be used to monitor, track, and triage events by processing the machine data from the data sources 102. The observability pipeline system 110 can receive an event data stream from each of the data sources 102 and identify the event data stream as pipeline input data to be processed by the observability pipeline system 110. The observability pipeline system 110 generates observability pipeline output data by applying observability pipeline processes to the pipeline input data and communicates the observability pipeline output data to the data destinations 104. In some implementations, the observability pipeline system 110 operates as a buffer between data sources and data destinations, such that all data sources send their data to the observability pipeline system 110, which handles filtering and routing the data to proper data destinations.

In some implementations, the observability pipeline system 110 unifies data processing and collection across many types of machine data (e.g., metrics, logs, and traces). The machine data can be processed by the observability pipeline system 110 by enriching it and reducing or eliminating noise and waste. The observability pipeline system 110 may also deliver the processed data to any tool in an enterprise designed to work with observability data. For example, the observability pipeline system 110 may analyze event data and send analytics to multiple data destinations 104, thereby enabling the systematic observation of event data for known conditions which require attention or other action. Consequently, the observability pipeline system 110 can decouple sources of machine data from data destinations and provide a buffer that makes many, diverse types of machine data easily consumable.

In some example implementations, the observability pipeline system 110 can operate on any type of machine data generated by the data sources 102 to properly observe, monitor, and secure the running of an enterprise's infrastructure and applications 116 while minimizing overlap, wasted resources, and cost. Specifically, instead of using different tools for processing different types of machine data, the observability pipeline system 110 can unify data collection and processing for all types of machine data (e.g., logs 204, metrics 206, and traces 208 shown in FIG. 2) and route the processed machine data to multiple data destinations 104. Unifying data collection can minimize or reduce redundant agents with duplicate instrumentation and duplicate collection for the multiple destinations. Unifying processing may allow routing of processed machine data to disparate data destinations 104 while adapting data shapes and controlling data volumes.

In an example, the observability pipeline system 110 obtains DogStatsd metrics, processes the DogStatsd metrics (e.g., by enriching the metrics), sends processed data having high cardinality to a first destination (e.g., Honeycomb) and processed data having low cardinality to a second, different destination (e.g., Datadog). In another example, the observability pipeline system 110 obtains windows event logs, sends full fidelity processed data to a first destination (e.g., an S3 bucket), and sends a subset (e.g., where irrelevant events are removed from the full fidelity processed data) to one or more second, different destinations (e.g., Elastic and Exabeam). In another example, machine data is obtained from a Splunk forwarder and processed (e.g., sampled). The raw processed data may be sent to a first destination (e.g., Splunk). The raw processed data may further be parsed, and structured events may be sent to a second destination (e.g., Snowflake).

The example observability pipeline system 110 shown in FIG. 1 includes a leader role 112 and multiple worker role 114. The leader role 112 leads the overall operation of the observability pipeline system 110 by configuring and monitoring the worker roles 114; the worker roles 114 receive event data streams from the data sources 102 and data storage 106, apply observability pipeline processes to the event data, and deliver observability pipeline output data to the data destinations 104 and data storage 106.

The observability pipeline system 110 may deploy the leader role 112 and a number of worker roles 114 on a single computer node or on many computer nodes. For example, the leader role 112 and one or more worker roles 114 may be deployed on the same computer node. Or in some cases, the leader role 112 and each worker role 114 may be deployed on distinct computer nodes. The distinct computer nodes can be, for example, distinct computer devices, virtual machines, containers, processors, or other types of computer nodes.

The user device 120, the observability pipeline system 110, or both, can provide a user interface for the observability pipeline system 110. Aspects of the user interface can be rendered on a display (e.g., the display 550 in FIG. 5) or otherwise presented to a user. The user interface (e.g., the example concurrent display of results shown in FIGS. 4A-4B or other types of user interfaces) may be generated by a web browser or another type of application that interacts with the observability pipeline system 110. The application can be deployed as software that includes application programming interfaces (APIs), graphical user interfaces (GUIs), and other modules. In some instances, the user interface for the observability pipeline system 110 may be implemented as the example concurrent display of results shown in FIGS. 4A-4B or in another manner. In some instances, operations in the example process 300 can be performed to display search results on the user interface.

In some implementations, an observability pipeline application can be deployed as a file, executable code, or another type of machine-readable instructions executed on the user device 120. The observability pipeline application, when executed, may render GUIs for display to a user (e.g., on a touchscreen, a monitor, or other graphical interface device), and the user can interact with the observability pipeline application through the GUIs. Certain functionality of the observability pipeline application may be performed on the user device 120 or may invoke the APIs, which can access functionality of the observability pipeline system 110. The observability pipeline application may be rendered and executed within another application (e.g., as a plugin in a web browser), as a standalone application, or otherwise. In some cases, an observability pipeline application may be deployed as an installed application on a workstation, as an “app” on a tablet or smartphone, as a cloud-based application that accesses functionality running on one or more remote servers, or otherwise.

In some implementations, the observability pipeline system 110 is a standalone computer system that includes only a single computer node. For instance, the observability pipeline system 110 can be deployed on the user device 120 or another computer device in the computing environment 100. For example, the observability pipeline system 110 can be implemented on a laptop or workstation. The standalone computer system can operate as the leader role 112 and the worker roles 114 and may execute an observability pipeline application that provides a user interface as described above. In some cases, the leader role 112 and each of the worker roles 114 are deployed on distinct hardware components (e.g., distinct processors, distinct cores, distinct virtual machines, etc.) within a single computer device. In such cases, the leader role 112 and each of the worker roles 114 can communicate with each other by exchanging signals within the computer device, through a shared memory, or otherwise.

In some implementations, the observability pipeline system 110 is deployed on a distributed computer system that includes multiple computer nodes. For instance, the observability pipeline system 110 can be deployed on a server cluster, on a cloud-based “serverless” computer system, or another type of distributed computer system. The computer nodes in the distributed computer system may include a leader node operating as the leader role 112 and multiple worker nodes operating as the respective worker roles 114. One or more computer nodes of the distributed computer system (e.g., the leader node) may communicate with the user device 120, for example, through an observability pipeline application that provides a user interface as described above. In some cases, the leader node and each of the worker nodes are distinct computer devices in the computing environment 100. In some cases, the leader node and each of the worker nodes can communicate with each other using TCP/IP protocols or other types of network communication protocols transmitted over a network (e.g., the network 108 shown in FIG. 1) or another type of data connection.

In some implementations, the observability pipeline system 110 is implemented by software installed on private enterprise servers, a private enterprise computing device, or other types of enterprise computing infrastructure (e.g., one or more computer systems owned and operated by corporate entities, government agencies, other types of enterprises). In such implementations, some or all of the data sources 102, data destinations 104, data storage 106, and the user device 120 can be or include the enterprise's own computer resources, and the network 108 can be or include a private data connection (e.g., an enterprise network or VPN). In some cases, the observability pipeline system 110 and the user device 120 (and potentially other elements of the computer environment 100) operate behind a common firewall or other network security system.

In some implementations, the observability pipeline system 110 is implemented by software running on a cloud-based computing system that provides a cloud hosting service. For example, the observability pipeline system 110 may be deployed as a SaaS system running on the cloud-based computing system. For example, the cloud-based computing system may operate through Amazon® Web Service (AWS) Cloud, Microsoft Azure Cloud, Google Cloud, DNA Nexus, or another third-party cloud. In such implementations, some or all of the data sources 102, data destinations 104, data storage 106, and the user device 120 can interact with the cloud-based computing system through APIs, and the network 108 can be or include a public data connection (e.g., the Internet). In some cases, the observability pipeline system 110 and the user device 120 (and potentially other elements of the computer environment 100) operate behind different firewalls, and communication between them can be encrypted or otherwise secured by appropriate protocols (e.g., using public key infrastructure or otherwise).

In some implementations, the search functionality is available through the cloud-based computing system and is provided by the observability pipeline system 110. In some instances, no additional search agent is required for performing search actions. For search-at-rest (e.g., searching data stored in AWS S3 buckets), a search process automatically can launch “executor” processes to perform the query locally. The search functionality of the observability pipeline system 110 may be performed according to a leader-to-worker node/edge node control protocol, or another type of control protocol.

In some implementations, search functionality is bounded by groups to support role-based access control, packs versioning, application of compute resources, and other functions. A search functionality can be specified in a query. A search target can be defined by one or more datasets, or one or more search strings in the query. In certain instances, the number of search targets can be defined in the query by the number of datasets or search strings.

In some implementations, operators that are supported by search functionality of the observability pipeline system 110 may include: Cribl—(Default) Custom Cribl operator—Simplifies locating specific events; Search—Locates specific events with specific text strings; Where—Filters events based on a Boolean expressions; Project—Define columns used to display results; Extend—Calculates one or more expressions and assigns the results to fields; Find—Locates specific events; Timestats—Aggregates events by time periods or bins; Extract—Extracts information from a field either via parser or regular expression; Summarize—Produces a table that aggregates the content of the input table; Limit (alias Take)—Defines the number of results to return; and other operators that enable other query capabilities.

In some implementations, search functionality supports multiple functions, including Cribl, Content, Scalar, Statistical and other function types. In some instances, different functions are available in a search language help tab of the user interface of the search functionality to define syntax, rules and provide examples for all Operators and Functions. In some instances, search recommendations may be included in the search functionality, e.g., default search settings, sample search queries, etc. The user interface of the search functionality may also include a history tab for displaying previous search queries. In some implementations, the search functionality supports complex search queries that includes multiple datasets, terms, Boolean logic, etc. These search terms or expressions can be grouped as a single search string. Wildcards may be supported for query bar terms and datasets.

In some cases, during operation, personnel (e.g., system administrators) can connect their user interface to the cloud-based computing system. A search window may appear on the user interface of the search functionality as a peer to the observability pipeline system 110. Data to query can be identified, which can be accomplished via datasets in a query or in another manner. In some contexts, a dataset is an addressable set of data defined in the query at various locations including endpoint nodes, S3 buckets, etc. Predefined datasets can be included in the search functionality, providing the ability to query state information of the observability pipeline system 110 as well as the filesystem of endpoint nodes. These include dataset definitions for leader nodes, endpoint nodes, filesystems, and S3. In some cases, administrators can define and configure their own datasets. In some implementations, the dataset model includes Name the Dataset—any unique identifier; Apply Dataset Provider—Identify external system (e.g., endpoint node, S3 Bucket, etc.); and Apply Dataset Provider Type—this identifies the schema (e.g., Cribl, Filesystem, S3, etc.).

In some instances, a search bar at the user interface of the search functionality can be configured to identify query values. Search functionality may support all personas, as a result the query expression can be simple terms or more complex literals, regexes, JavaScript expressions, etc. In some implementations, data to be queried is identified; and one or more datasets are defined. In some implementations, the search bar at the user interface of the search functionality includes “type-ahead” capability for syntax completion and query history. For example, by just typing “Dat..” the look ahead capability can provide a list of available datasets. In some implementations, the query operators are defined. Functions, terms, strings, and other query operators can be defined in a query and separated by a “|” (pipe).

In certain instances, one or more time ranges for queries can be defined. The one or more time ranges may include real-time window—seconds, minutes, hours, days; specific time range, e.g., Mar. 20, 2022: 06:00-06:30; or others. A search process can be performed according to the query. Discovery data can be returned as part of the search results as line items in table format, charts, or in another manner. The search results can be shaped and discovered data can be aggregated as part of the query (e.g., Project, Extend, Summarize operators) or afterwards with charting options. In some implemented, different chart types, color palettes, axis settings, legends to manipulate how results are displayed can be selected or defined/configured by the user. In some examples, the number of search results are limited by the query language, including time range. In certain examples, a number of results returned can also be constrained via the “Limit” operator (e.g., Limit 100 or another number).

The search functionality may allow users to tune the scope of the query as wide or narrow by specifying constraints within the search itself. For example, a “wide” query can specify a search for instances of ‘error’ on any workgroup or fleet (which may include a group of devices, equipment, computers, or nodes within a small network); a “narrow” query can specify a search for instances of ‘error’ on host: xyx, in: Var/log directory; and a query can be anywhere in between the wide and narrow queries based on rules.

In some instances, the search functionality can query data from specific third-party vendor platforms. Third-party search functions and the search functionality of the observability pipeline system 110 work independently. Administrators may use search results from the search functionality of the observability pipeline system 110 to apply additional configurations to their existing systems and/or configure. The observability pipeline system 110 can forward discovered data or other search results to the third-party systems or platforms. When accessing external data stores (e.g., AWS S3), the search functionality can define authentication rights when the specific dataset is defined.

In some implementations, a search query is generated at the user device 120 based on user input. For instance, the search query may be generated based on search terms entered by a user through a user interface provided by a web browser or other application running on the user device 120. The search query represents a request to search for data that meet specified criteria; for instance, the search query may include search operators that specify target values of parameters. In some examples, a search operator may specify a target value for event type, event time, event origin, event source, system state context, or other parameters. When the user device 120 receives or otherwise obtains search results for the search query, the search results can be displayed to the user. For instance, the search results may be displayed in a user interface provided by a web browser or other application running on the user device 120.

In some implementations, the search query is received by an agent of the observability pipeline system 110 (e.g., an agent running at the user device 120, on a server, in the cloud or elsewhere), and the agent can dispatch the search query to an appropriate resource in the observability pipeline system 110. The agent may dispatch the search query to one or more computer resources, computer systems, or locations associated with the data to be searched. For instance, a search query may be dispatched to a resource, system or location associated with a data source 102, a data destination 104, a data storage 106. Accordingly, the observability pipeline system 110 can perform the search at an endpoint node, on a server, on a cloud-based storage facility, or elsewhere.

In some implementations, a search is performed by configuring and executing an observability pipeline process. For example, an observability pipeline process (e.g., the observability pipeline process 200 shown in FIG. 2) can be configured to perform a search according to a search query. Configuring an observability pipeline process can include selecting, defining or configuring any aspect or feature of the observability pipeline process. For example, configuring the observability pipeline process may include selecting a source that will provide input data for the observability pipeline process, selecting a destination where the output data from the observability pipeline process will be sent, configuring a pipeline engine (e.g., by selecting and applying configuration settings to routes and pipelines) that will process the data. In some examples, the pipelines or aspects of a pipeline engine can include filters that are configured based on the search query. For instance, a pipeline can be configured to select events according to a search operator, for example, events that match a target value for event type, event time, event origin, event source, etc. In some examples, the data source for the observability pipeline process is defined based on the search query. For instance, if a search query specifies a device or application to be searched, the data source for the observability pipeline process can be defined as the specified device or application. In some examples, the data destination for the observability pipeline process is defined based on the search query. For instance, the agent that dispatched the search query can be defined as the data destination for the observability pipeline process.

FIG. 2 is a block diagram showing aspects of an example observability pipeline process 200 that can be applied in an observability pipeline system. For example, the observability pipeline process 200 may be performed by one or more of data sources 102, data destinations 104, data storage 106, and the worker roles 114 shown in FIG. 1 or in another observability pipeline system. In some cases, the observability pipeline process 200 can be configured to perform a search, for example, based on a search query.

The example observability pipeline process 200 shown in FIG. 2 includes data collection 230, schema normalization 220, routing 222, streaming analytics and processing 224A, 224B, 224C, and output schematization 226A, 226B, 226C, 226D, 226E. The observability pipeline process 200 may include additional or different operations, and the operations of the observability pipeline process 200 may be performed as described with respect to FIG. 2 or in another manner. In some cases, one or more of the operations can be combined, or an operation can be divided into multiple sub-processes. Certain operations may be iterated or repeated, for example, until a terminating condition is reached.

As shown in FIG. 2, the observability pipeline process 200 is applied to pipeline input data 201 from data sources, and the observability pipeline process 200 delivers observability pipeline output data 203 to data destinations. The data sources can include any of the example data sources 102 or data storage 106 described with respect to FIG. 1, and the data destinations can include any of the example data destinations 104 or data storage 106 described with respect to FIG. 1.

The example pipeline input data 201 shown in FIG. 2 includes logs 204, metrics 206, traces 208, stored data payloads 210, and possibly other types of machine data. In some cases, some or all of the machine data can be generated by agents (e.g., Fluentd, Collectd, OpenTelemetry) that are deployed at the data sources, for example, on various types of computing devices in a computing environment (e.g., in the computing environment 100 shown in FIG. 1, or another type of computing environment). The logs 204, metrics 206, and traces 208 can be decomposed into event data 202 that are consumed by the observability pipeline process 200. In some instances, logs 204 can be converted to metrics 206, metrics 206 can be converted to logs 204, or other types of data conversion may be applied.

In the example shown, the stored data payloads 210 represent event data retrieved from external data storage systems. For instance, the stored data payloads 210 can include event data that an observability pipeline process previously provided as output to the external data storage system.

The event data 202 are streamed to the observability pipeline process 200 for processing. Here, streaming refers to a flow of data, which is distinct from batching or batch processing. With streaming, data are processed as they flow through the system, e.g., continuously (as opposed to batching, where individual batches are collected and processed as discrete units). As shown in FIG. 2, the event data from the logs 204, metrics 206, and traces 208 are streamed directly to the schema normalization process (at 220) without use of the collection process (at 230), whereas the event data from the stored data payloads 210 are streamed to the collection process (at 230) and then streamed to the schema normalization process (at 220), the routing process (at 222) or the streaming analytics and processing (at 224).

In some instances, event data 202 represents events as structured or typed key value pairs that describe something that occurred at a given point in time. For example, the event data 202 can contain information in a data format that stores key-value pairs for an arbitrary number of fields or dimensions, e.g., in JSON format or another format. A structured event can have a timestamp and a “name” field. Instrumentation libraries can automatically add other relevant data like the request endpoint, the user-agent, or the database query. In some implementations, components of the events data 202 are provided in the smallest unit of observability (e.g., for a given event type or computing environment). For instance, the event data 202 can include data elements that provide insight into the performance of the computing environment 100 to monitor, track, and triage incidents (e.g., to diagnose issues, reduce downtime, or achieve other system objectives in a computing environment).

In some instances, logs 204 represent events serialized to disk, possibly in several different formats. For example, logs 204 can be strings of text having an associated timestamp and written to a file (often referred to as a flat log file). The logs 204 can include unstructured logs or structured logs (e.g., in JSON format). For instance, log analysis platforms store logs as time series events, and the logs 204 can be decomposed into a stream of event data 202.

In some instances, metrics 206 represent summary information about events, e.g., timers or counters. For example, a metric can have a metric name, a metric value, and a low cardinality set of dimensions. In some implementations, metrics 206 can be aggregated sets of events grouped or collected at regular intervals and stored for low cost and fast retrieval. The metrics 206 are not necessarily discrete and instead represent aggregates of data over a given time span. Types of metric aggregation are diverse (e.g., average, total, minimum, maximum, sum-of-squares), but metrics typically have a timestamp (representing a timespan, not a specific time); a name; one or more numeric values representing some specific aggregated value; and a count of how many events are represented in the aggregate.

In some instances, traces 208 represent a series of events with a parent/child relationship. A trace may provide information of an entire user interaction and may be displayed in a Gantt-chart-like view. For instance, a trace can be a visualization of events in a computing environment, showing the calling relationship between parent and child events, as well as timing data for each event. In some implementations, individual events that form a trace are called spans. Each span stores a start time, duration, and an identification of a parent event (e.g., indicated in a parent-id field). Spans without an identification of a parent event are rendered as root spans.

The example observability pipeline output data 203 shown in FIG. 2 include data formatted for log analytics platforms (250), data formatted for time series databases (TSDBs) (252), data formatted for distributed tracing systems (254), data formatted for security information and event management (SIEM) or user behavior analytics (UBA) systems 256, and data formatted for event streaming systems or data lakes 258 (e.g., a system or repository of data stored in its natural/raw format). Log analytics platforms are configured to operate on logs to generate statistics (e.g., web, streaming, and mail server statistics) graphically. TSDBs operate on metrics; example TSDBs include Round Robin Database (RRD), Graphite's Whisper, and OpenTSDB. Tracing systems operate on traces to monitor complex interactions, e.g., interactions in a microservice architecture. SIEMs provide real-time analysis of security alerts generated by applications and network hardware. UBA systems detect insider threats, targeted attacks, and financial fraud. Observability pipeline output data 203 may be formatted for, and delivered to, other types of data destinations in some cases.

In the example shown in FIG. 2, the observability pipeline process 200 includes a schema normalization module that (at 220) converts the various types of event data 202 to a common schema or representation to execute shared logic across different agents and data types. For example, machine data from various agents such as Splunk, Elastic, Influx, and OpenTelemetry have different, opinionated schemas, and the schema normalization module can convert the event data to normalized event data. Machine data intended for different destinations may need to be processed differently. Accordingly, the observability pipeline process 200 includes a routing module that (at 222) routes the normalized event data (e.g., from the schema normalization module 220) to different processing paths depending on the type or content of the event data. The routing module can be implemented by having different streams or topics. The routing module routes the normalized data to respective streaming analytics and processing modules. FIG. 2 shows three streaming analytics and processing modules, each applied to normalized data (at 224A, 224B, 224C); however, any number of streaming analytics and processing modules may be applied. Each of the streaming analytics and processing modules can aggregate, suppress, mask, drop, or reshape the normalized data provided to it by the routing module. The streaming analytics and processing modules can generate structured data from the normalized data provided to it by the routing module. The observability pipeline process 200 includes output schema conversion modules that (at 226A, 226B, 226C, 226D, 226E) schematize the structured data provided by the streaming analytics and processing modules. The structured data may be schematized for one or more of the respective data destinations to produce the observability pipeline output data 203. For instance, the output schema conversion modules may convert the structured data to a schema or representation that is compatible with a data destination. In some implementations, the observability pipeline process 200 includes an at-least-once delivery module that (at 228) applies delivery semantics that guarantee that a particular message can be delivered one or more times and will not be lost. In some implementations, the observability pipeline process 200 includes an alerting or centralized state module, a management module, or other types of sub-processes.

In the example shown in FIG. 2, the observability pipeline process 200 includes a collection module that (at 230) collects filtered event data from stored data payloads 210. For example, the stored data payloads 210 may represent event data that were previously processed and stored on the event streaming/data lake 258 or event data that were otherwise stored in an external data storage system. For example, some organizations have a high volume of data that is kept in storage systems (e.g., S3, Azure Blob Store, etc.) for warehousing purposes, or they may have event data that can be scraped from a REST endpoint (e.g., Prometheus). The collection module may allow organizations to apply the observability pipeline process 200 to data from storage, REST endpoints, and other systems regardless of whether the data has been processed by an observability pipeline system in the past. The data collection module can retrieve the data from the stored data payload 210 on the external data storage system, stream the data to the observability pipeline process 200 (e.g., via the schema normalization module, the routing module, or a streaming analytics and processing module), and send the output to any of the data destinations 230.

FIG. 3 is a flow chart showing aspects of an example process 300 for displaying search data. In some implementations, the operations of the example process 300 are performed by operation of a computer node associated with an observability pipeline system (e.g., a user device 120 in FIG. 1) to generate and update a graphical user interface that is configured to present search results. The example process 300 may include additional or different operations, including operations performed by additional or different components, and the operations may be performed in the order shown or in another order.

At 302, search results are obtained. In some implementations, the search results are obtained based on searching data (e.g., based on searching the example observability pipeline input data 201 or the example observability pipeline output data 203 shown in FIG. 2) in an observability pipeline system. In some instances, the data to be searched includes data stored at the computer node, a remote storage location (e.g., data destinations 104 or data storage 106 shown in FIG. 1), or at another location.

In some instances, the search results are generated based on a search query. For instance, the search query may be received by the computer node through a user interface or from a remote source. In some cases, the computer node is part of a cloud-based observability pipeline system, and the search query is received from a user device via the Internet or another network. In some implementations, the search query specifies one or more search criteria (e.g., filters, search terms, etc.); the search query may specify location information of data (e.g., bucket name, object-store prefix, access permissions, etc.) and other search parameters. The search query requests information about the data, which may include data stored at one location or multiple locations (e.g., the data storage 106, the data destinations 104, the data sources 102, etc.).

In certain instances, a search query may specify a time period. For example, a time period may be a period when the source data was created, a time when the event occurred, or another time period. The time period may be defined by number of seconds, number of minutes, number of hours, number of days, all events, a user defined time period (during a specific data and time range), or in another manner. In some implementations, the search query includes one or more search operators. Data flows from one operator to the next and events are filtered or shaped at each search operator, and then fed into the subsequent search operator. Each time the data passes through an operator, it is filtered, rearranged, summarized, or otherwise processed. Because the piping of information from one search operator to another can be sequential, the order of the search operators in the search query can be important and can affect both results and performance. In some implementations, the order of the search operators in the search query can be adjusted. In some implementations, search operators in a search query includes one or more of the following: cribl—(Implicit) locate specific events, search—locate specific events (less powerful than ‘cribl’), extend—Build custom columns in query results, limit (take)—Sets a specified number of events to return, project—Defines the data columns to display, summarize—Aggregates the results in a single table, where—Filters event based on Boolean logic, find—Locates specific events, timestats—Generate data for displaying statistical trends over time, and other search operators.

In some cases, the search targets data stored at the computer node, and the computer node executes the search query on data stored locally at the computer node and generates search results. In some cases, the search targets data stored remotely (e.g., at a distinct storage node), and the computer node communicates with one or more remote nodes to execute the search query. For example, the computer node may transmit the search query to remote computer nodes for scanning and processing respective portions of the data requested in the search query.

In some instances, one or more computing resources may be configured and initiated, for example, by the computer node remotely. In some instances, a search node, distinct from the computer node and one or more storage nodes, may be identified and configured to perform the search on data stored at the one or more storage nodes. In this case, one or more dynamic computing resources may be initiated at the search node by the computer node. In some implementations, the one or more dynamic computing resources are configured to operate as a search engine to remotely read, scan, and process the data stored at the storage node according to the search query. In some instances, an endpoint node, distinct from the computer node, is configured to perform the search on data stored locally at the endpoint node. For example, the endpoint node may search event data generated by applications and processes running locally on the endpoint node. In some implementations, the computing resources are configured to generate the search results by scanning and processing the stored data, e.g., filtering, aggregating, enhancing, and other processing operations. In some instances, the computing resource configures an observability pipeline process (e.g., the observability pipeline process 200 as shown in FIG. 2) based on the received search query, and the computing resource produces the search results by executing the observability pipeline process.

At 304, time bins are identified; the time bins can be identified based on the search results, the search query, or other information. In some implementations, events in the search results are timestamped. In other words, each event in the search results includes a timestamp, which represents information identifying when the event occurred. For example, a timestamp of an event includes a date, time of day in hour, minute, second, or a fraction of second. In some instances, a timestamp of an event in the search results may not be based on an absolute notion of time (e.g., Coordinated Universal Time, UTC); and can be relative to any arbitrary time defined by the observability pipeline system or any arbitrary time in the past. In some implementations, the search results include a series of events that span in a time period; and a size of time bins (e.g., 10 microseconds, 1 minutes, 20 minutes, 1 hour, 4 hours, 1 day, etc.) and a starting time for grouping events can be determined according to the duration of the time period and their timestamp values. In some instances, the starting time or the size of the time bins may be adjusted, changed, or updated by operation of a graphical user interface according to user's input. In some instances, the time bins may be identified in another manner. In some examples, each time bin has a duration of 10 seconds, 1 second, 100 milliseconds, or another duration. In some cases, the time bins are contiguous in time, such that they collectively span a time period. For instance, a 100-second time period can be divided into 100 non-overlapping time bins of equal duration (each 1 second in duration). Generally, the time bins can be defined in any manner that may be useful for analyzing or gaining insight into the search results.

At 306, first histogram data are generated. In some implementations, the first histogram data includes a first set of event clusters for the respective time bins. The first histogram data can be generated based on the time bins and the events in the search results. In some implementations, the first set of event clusters represent a first subset of the events in the search results. The first subset of events can be identified based on a characteristic of interest (e.g., event location, event source, event destination, etc.). For example, the first subset of events can include all events in which the characteristic of interest has a first value. For example, the first subset of the events in the first histogram data may include events that occurred at a first node, events resulting from a first type of operation, or events that share another first value for a characteristic of interest. In some implementations, events in the first subset of the events are split and grouped together into the first set of event clusters for the respective time bins based on their respective timestamp values. Accordingly, each event cluster is associated with one of the time bins and contains a group of events that occurred within (or is otherwise associated with) the timeframe represented by the time bin. In some instances, an event cluster in the first set of event clusters may include no event, one event or multiple events.

At 308, second histogram data are generated. In some implementations, the second histogram data includes a second set of event clusters for the respective time bins. The second histogram data can be generated based on the time bins and the events in the search results. In some implementations, the second set of event clusters represents a second subset of the events in the search results. The second subset of events can be identified based on the same characteristic of interest. For example, the second subset of events can include all events in which the characteristic of interest has a second value. For example, the second subset of the events in the second histogram data may include events that occurred at a second node, events resulting from a second type of operation, or events that share another second value for the characteristic of interest. In some implementations, events in the second subset of the events are split and grouped together to form the second set of event clusters for the respective time bins based on their respective timestamp values. Accordingly, each event cluster is associated with one of the time bins and contains a group of events that occurred within (or is otherwise associated with) the timeframe represented by the time bin. In some instances, an event cluster in the second set of event clusters may include no event, one event or multiple events.

In some implementations, prior to generating the first or second histogram data at operations 306, 308 of the example process 300, a characteristic of interest is identified. In some instances, a characteristic of interest may be specified based on user input obtained at the computer node or selected from a pre-configured list of characteristics of interest. For example, a characteristic of interest may be or include an event source, an event location, an event destination, an event type, or other characteristics of interest. In some implementations, the first and second subsets of the events are identified based on the value of the characteristic of interest. In particular, the first subset of the events may be identified based on a characteristic of interest having a first value; and the second subset of the events may be identified based on a characteristic of interest having a second, distinct value. As an example, a first subset of the events may be identified based on events that occurred at a first node; and a second subset of the events may be identified based on events that occurred on a second, distinct node. In some instances, the first and second subsets of the events may be identified in another manner.

At 310, a graphical user interface is generated. In some implementations, the graphical user interface is configured to present the first histogram data and the second histogram data. In some implementations, the graphical user interface is an interactive interface for data visualization, which allows users to customize data display by adjusting the size of the time bins, the time period, the values of the characteristic of interest, and other parameters. For example, users can focus on specific aspects of the data; change the visual representation of the data; and adjust the level of detail displayed. By doing this, the graphical user interface can improve understanding and develop deeper insights of complex search results. In some implementations, the graphical user interface can provide other technical advantages.

In some instances, the graphical user interface may be divided into multiple sections. In a first section (e.g., a top or bottom section in the graphical user interface), the graphical user interface includes a first histogram representing the first histogram data and a second histogram representing the second histogram data. The first histogram includes a first set of bins graphically representing the first set of event clusters of the first histogram data; and the second histogram includes a second set of bins graphically representing the second set of event clusters of the second histogram data. In a second section (e.g., the opposite section of the graphical user interface), the graphical user interface includes a first data panel for presenting event data of the first set of event clusters, and a second data panel for presenting event data of the second set of event clusters. In some instances, the graphical user interface may include other sections, and sections may be configured or arranged in a different manner. In some implementations, the first and second histogram data are displayed simultaneously side-by-side on the graphical user interface (e.g., as shown in FIGS. 4A-4B or otherwise). In some implementations, the first and second histogram data is presented using separate histograms, e.g., the first histogram and the second histogram.

FIG. 4A is a schematic diagram 400 showing a concurrent display of results from two search queries. The concurrent display of results shown in FIG. 4A can be displayed, for example, on a display device of a computer node associated with an observability pipeline system. In some instances, the computer node may be the computer node that receives and stores the search results (e.g., events, etc.). In some instances, the search results may be received by the computer node from one or more distinct nodes (e.g., search nodes or storage nodes) by execution of the same search query or different search queries. In some instances, the graphical user interface may be part of a user interface on a user device that communicates with the computer node of the observability pipeline system. In some instances, the computer node may be the computer node that generates the search results. Multiple sets of the events in the search results may be displayed. The multiple sets of the events can be automatically clustered, matched and aligned using best-effort, or best-fit temporal binning.

In the example shown in FIG. 4A, the concurrent display of results shown in FIG. 4A includes a first histogram 402A representing the first histogram data and a second histogram 402B representing the second histogram data. The first and second histograms 402A, 402B are displayed side-by-side. In particular, the first histogram 402A is displayed on the left; and the second histogram 402B is displayed on the right. Each of the first and second histograms 402A, 402B includes three bins 412A, 412B. In the first and second histograms 402A, 402B, each bin 412A, 412B graphically represents an event cluster in a corresponding time bin; and the height of each bin 412A, 412B represents the number (the quantity) of the events in the corresponding time bin. In particular, the bins 412A in the first histogram 402A graphically represent a first set of event clusters; and the bins 412B in the second histogram 402B graphically represent a second set of event clusters. In the example shown, each of the bins 412A, 412B is a vertical bar in a bar graph, and the height of each vertical bar indicates the quantity of events in a respective event cluster. A histogram may include another type of graphical representation of event clusters; for instance, other types of glyphs, graphic elements or visual indicators may be used. Generally, the number of bins displayed in a histogram corresponds to the number of event clusters, and each bin has a graphical property (e.g., height, length, diameter, color, shading, etc.) that corresponds to the number of events in the event cluster represented by the bin. In the example shown, the bins 412A, 412B correspond to time bins within the timeframe; and the first and second histograms 402A, 402B present the bins 412A, 412B in chronological order. comprising a first set of bins graphically representing the first set of event clusters

As shown in FIG. 4A, the schematic diagram 400 further includes a first data panel 404A and a second data panel 404B. The first data panel 404A corresponds to the first histogram 402A and is presented below the first histogram 402A. In some implementations, the first data panel 404A includes multiple segments 414A corresponding to the first set of bins 412A in the first histogram 402A; the first data panel 404A is configured to present detailed event data of events in each event cluster of the first set of event clusters (e.g., in each bin 412A of the first histogram 402A). For example, each segment 414A includes a text representation of events in the respective one of the first set of bins 412A and a representation 406 of the corresponding time bin (e.g., a time range of the time bin). In some instances, each segment 414A includes other information, e.g., a total number of events in a respective bin 412A.

The second data panel 404B corresponds to the second histogram 402B and is presented below the second histogram 402B. In some implementations, the second data panel 404B includes multiple segments 414B corresponding to the second set of bins 412B in the second histogram 402B; the second data panel 404B is configured to present detailed event data of events in each event cluster of the second set of event clusters (e.g., in each bin 412B of the second histogram 402B). For example, each segment 414B includes a text representation of events in the respective one of the second set of bins 412B and a representation 406 of the corresponding time bin (e.g., a time range of the time bin). In some instances, each segment 414B includes other information, e.g., a total number of events in a respective bin 412B.

In the schematic diagram 400, the segments 414A, 414B in the first and second data panels 404A, 404B are organized according to the corresponding time bins. In particular, the segments 414A, 414B containing the event data of events in the first and second sets of event clusters are organized side-by-side in a vertical array such that the event details of events in the corresponding time bin can be compared and analyzed. In some implementations, each segment 414A in the first data panel 404A is aligned with a respective segment 414B in the second data panel 404B such that each pair of aligned segments 414A/414B correspond to a respective time bin. In some implementations, the segments 414A, 414B may not be configured to display the text representation of all of the events in the respective bin 412A, 412B. In some instances, the size of the segments 414A, 414B may be adjusted by the user.

In some instances, the first and second data panels 404A, 404B do not display all the events in all the bins or event clusters; and the first and second data panels 404A, 404B may have multiple pages. When scrolling up or down pages in the first and second data panels 404A, 404B, pages in the first and second data panels 404A, 404B can be scrolled synchronously. In some instances, when scrolling up or down pages in the first and second data panels 404A, 404B, the first and second histograms 402A, 402B may be scrolled left or right synchronously. In some instances, the bins 412A, 412B in the first and second histograms 402A, 402B and the representations 406 may be color coded for improved visualization.

In some implementations, when the first subset of the events is being clustered, the number of events in the first subset of the events matching the parameters of each bin 412A is counted; and when the second subset of the events is clustered, the number of events in the second subset of the events matching the parameters of each bin 412B is also counted. At least some of the parameters of the one or more of the bins 412A, 412B are defined based on user input. In some instances, the first subset of the events shown in the first data panel 404A may include events detected at a first node during a timeframe. The second subset of the events shown in the second data panel 404B may include events detected at a second node during the same timeframe.

As an example, assume a user is troubleshooting a scenario that involves problems in communication between a sender node S and a receiver node R. Both S and R logs (e.g., logs of the sender node S and the receiver node R) are collected and stored on a computer node of an observability pipeline system. A search query can be executed based on the S and R logs and search results are obtained, for example, by performing operation 302 in the example process 300 shown in FIG. 3. Since the S and R logs are often temporally related, most of the correlation work by aligning errors/issues on one node to those of the other node based on time differences. For example, when the sender node S sent a message to the receiver node R, at a time t1, the receiver node R may log an error; and at a time t2, the sender node S may log a timeout. The S and R logs can be automatically binned according to a predetermined or default size of time bins specified by a user through the graphical user interface. In some instances, the default size of time bins can be overridden or modified according to the user input. In this case, the first histogram data on the schematic diagram 400 represents the binned R logs; and the second histogram data on the schematic diagram 400 represents the binned S logs. The binned R and S logs are displayed simultaneously to facilitate exploration and synchronous scrolling of both the organized R and S logs simultaneously.

At 312, the graphical interface is updated in response to a user selecting a bin in a histogram. In some implementations, a user can select a bin in a histogram in the concurrent display of results in FIG. 4A to explore the event data in the selected bin. For example, a user may select a bin in the first histogram, the graphical user interface can be updated to include a visual indication of the selected bin in the first histogram. The updated graphical user interface also includes a visual indication of a corresponding bin in the second histogram. The selected bin in the first histogram and the corresponding bin in the second histogram represent event clusters that are grouped in the same time bin. The update graphical user interface may further include a visual indication of first event data in a segment of the first data panel that corresponds to the selected bin the first histogram; and a visual indication of second event data in a segment of the second data panel that corresponds to the corresponding bin in the second histogram.

FIG. 4B is a schematic diagram 430 showing a concurrent display of results from two search queries. The example updated schematic diagram 430 may be displayed on a computer node of an observability pipeline system. The concurrent display of results shown in FIG. 4B may be obtained by updating the concurrent display of results shown in in FIG. 4A can be updated, for example, in response to a user selecting a bin in a histogram or in another manner. The user selection of the bin can be provided in various manners; for instance, the user selection can be made through a touchscreen, a pointing device (e.g. a mouse), a stylus, a voice command processor, or another user input device.

As shown in FIG. 4B, in response to a bin 412A in the first histogram 402A being selected by a user, the selected bin 412A is shown with a visual indication 432A; at the same time, a corresponding bin 412B in the second histogram 402B corresponding to the selected bin 412A in the first histogram 402A is also shown with a visual indication 432B. Updating the graphical user interface to include a visual indication of a bin can include changing the color, shading or other visible properties of the bin, so that the bin is visually distinct from the other bins in the histogram. Other types of visual indication may be used, including dynamic elements, geometric elements, etc. Simultaneously or subsequently, the first and second data panels 404A, 404B are updated. In particular, the time bin 406 in the corresponding segment 414A of the first data panel 404A corresponding to the selected bin 412A in the first histogram 402A is highlighted by a visual indication 434A; and the time bin 406 in the corresponding segment 414B of the second data panel 404B corresponding to the corresponding bin 412B in the second histogram 402B is also highlighted by a visual indication 434B. In some instances, visual indications to the first event data that correspond to the select bin 412A in the first histogram 402A and to the second event data that corresponds to the corresponding bin 412B in the second histogram 402B may be included in the updated concurrent display of results shown in FIG. 4B. For example, all the event data in the selected bin 412A of the first histogram 402A (e.g., a first segment 436A in the first data panel 404A) and the corresponding bin 412B of the second histogram 402B (e.g., a second segment 436B in the second data panel 404B) may be expanded to show additional events such that all the events in the first and second segments 414A, 414B are presented on the updated concurrent display of results shown in FIG. 4B. In some instances, depending on the number of events in the selected bin 412A or the corresponding bin 412B, alignment between the selected pair of segments 436A, 436B corresponding to the time bins that are highlighted by the visual indications 434A, 434B may be expanded to show the entire text representation of the event data. In certain instances, the visual indications to the text representations in the first and second segments 414A, 414B of the first and second data panels 404A, 404B may be presented in another manner.

In some instances, a bin that is previously selected can be unselected by the user. In this case, the expanded segments 436A, 436B may be returned to a default size (e.g., as shown in FIG. 4A) which allows re-alignment of the two segments. In some instances, the visual indications on the corresponding time bins and the bins 412A, 412B in the corresponding histograms 402A, 402B may be removed. In some instances, when a different bin is selected, the concurrent display of results shown in FIG. 4B may be further updated. For example, new visual indications may be added to the newly selected bins and the corresponding time bin; and the size of the newly selected segments may be changed such that the entire text representation of the event data in the respective bins are shown. In some instances, the concurrent display of results shown in FIG. 4B may include other types of visual indications or be updated in another manner.

FIG. 5 is a block diagram showing an example computer system 500. In some implementations, the example computer system 500 is configured to perform operations in the example process 300 or in another manner. The example computer system 500 includes a data processing apparatus and one or more computer-readable storage devices. The term “data-processing apparatus” encompasses all kinds of apparatus, devices, nodes, and machines for processing data, including by way of example, a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing, e.g., processor 510. The apparatus can include special-purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them.

A computer program (also known as a program, software, software application, script, or code), e.g., computer program 524, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

Some of the processes and logic flows described in this specification can be performed by one or more programmable processors, e.g., processor 510, executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both, e.g., memory 520. Elements of a computer can include a processor that performs actions in accordance with instructions, and one or more memory devices that store the instructions and data. A computer may also include or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic disks, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a phone, an electronic appliance, a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, flash memory devices, and others), magnetic disks (e.g., internal hard disks, removable disks, and others), magneto optical disks, and CD ROM and DVD-ROM disks. In some cases, the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

The example power unit 540 provides power to the other components of the computer system 500. For example, the other components may operate based on electrical power provided by the power unit 540 through a voltage bus or other connection. In some implementations, the power unit 540 includes a battery or a battery system, for example, a rechargeable battery. In some implementations, the power unit 540 includes an adapter (e.g., an AC adapter) that receives an external power signal (from an external source) and converts the external power signal to an internal power signal conditioned for a component of the computer system 500. The power unit 540 may include other components or operate in another manner.

To provide for interaction with a user, operations can be implemented on a computer having a display device, e.g., display 550, (e.g., a monitor, a touchscreen, or another type of display device) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse, a trackball, a tablet, a touch sensitive screen, or another type of pointing device) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to, and receiving documents from, a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser, or by sending data to an application on a user's client device in response to requests received from the application.

The computer system 500 may include a single computing device or multiple computers that operate in proximity or generally remote from each other and typically interact through a communication network, e.g., via interface 530. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), a network comprising a satellite link, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks). A relationship between client and server may arise by virtue of computer programs running on the respective computers and having a client-server relationship with each other.

The example interface 530 may provide communication with other systems or devices. In some cases, the interface 530 includes a wireless communication interface that provides wireless communication under various wireless protocols, such as, for example, Bluetooth, Wi-Fi, Near Field Communication (NFC), GSM voice calls, SMS, EMS, or MMS messaging, wireless standards (e.g., CDMA, TDMA, PDC, WCDMA, CDMA2000, GPRS) among others. Such communication may occur, for example, through a radio-frequency transceiver or another type of component. In some cases, the interface 530 includes a wired communication interface (e.g., USB, Ethernet) that can be connected to one or more input/output devices, such as, for example, a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, through a network adapter.

In a general aspect of what is described, a method for generating and updating a graphical user interface to display search results obtained from the execution of a search query in an observability pipeline system is presented.

In a first example, a method includes obtaining search results including events identified based on searching observability pipeline output data generated by an observability pipeline system; identifying time bins based on the search results; generating first histogram data based on the time bins and the events; and generating second histogram data based on the time bins and the events. The first histogram data includes a first set of event clusters for the respective time bins; and the first set of event clusters corresponds to a first subset of the events. The second histogram data includes a second set of event clusters for the respective time bins; and the second set of event clusters corresponds to a second, distinct subset of the events. The method further includes generating a graphical user interface. The graphical user interface includes a first histogram representing the first histogram data and including a first set of bins graphically representing the first set of event clusters, a first data panel for first event data describing one or more of the first subset of the events, a second histogram representing the second histogram data and including a second set of bins graphically representing the second set of event clusters, and a second data panel for second event data describing one or more of the second subset of the events. The method further includes in response to a user selection of one of the first set of bins, updating the graphical user interface to include: in the first histogram, a visual indication of the selected bin; in the second histogram, a visual indication of a corresponding bin; in the first data panel, a visual indication of first event data that corresponds to the selected bin in the first histogram; and in the second data panel, a visual indication of second event data that corresponds to the corresponding bin in the second histogram.

In a second example, a computing system includes one or more processors and a computer-readable medium storing instructions that are operable when executed by the one or more processors to perform one or more operations of the first example.

In a third example, a non-transitory computer-readable medium stores instructions that are operable when executed by a data-processing apparatus to perform one or more operations of the first example.

Implementations of the first, second, or third example may include one or more of the following features. The method further includes identifying a characteristic of interest; identifying the first subset of the events based on the characteristic of interest; and identifying the second subset of the events based on the characteristic of interest. The first subset of events has a first value for the characteristic of interest; and the second subset of events has a second, distinct value for the characteristic of interest. The characteristic of interest includes an event source, an event location, or an event destination.

Implementations of the first, second, or third example may include one or more of the following features. The first histogram and the second histogram are presented side-by-side in a first section of the graphical user interface. The first data panel and the second data panel are presented side-by-side in a second section of the graphical user interface, wherein the first section is presented above or below the second section. The first data panel includes first segments corresponding to the first set of bins. Each first segment includes a text representation of one or more events in a respective one of the first set of bins and a numerical representation of the time bin. The second data panel includes second segments corresponding to the second set of bins. Each second segment includes a text representation of one or more events in a respective one of the second set of bins and a numerical representation of the time bin. Each of the first segments is aligned with a respective one of the second segments such that each pair of aligned first and second segments correspond to a respective one of the time bins. The method further includes in response to a user selection of one of the first segments, expanding the selected segment to display a text representation of additional events.

While this specification contains many details, these should not be understood as limitations on the scope of what may be claimed, but rather as descriptions of features specific to particular examples. Certain features that are described in this specification or shown in the drawings in the context of separate implementations can also be combined. Conversely, various features that are described or shown in the context of a single implementation can also be implemented in multiple embodiments separately or in any suitable sub-combination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single product or packaged into multiple products.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications can be made. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A method of displaying search result data for an observability pipeline system, comprising:

obtaining search results, the search results identified based on searching data generated by an observability pipeline system, the search results comprising events processed by the observability pipeline system;

identifying time bins based on the search results;

generating first histogram data based on the time bins and the events, the first histogram data comprising a first set of event clusters for the respective time bins, wherein the first set of event clusters correspond to a first subset of the events;

generating second histogram data based on the time bins and the events, the second histogram data comprising a second set of event clusters for the respective time bins, wherein the second set of event clusters correspond to a second, distinct subset of the events;

generating a graphical user interface comprising: a first histogram representing the first histogram data and comprising a first set of bins graphically representing the first set of event clusters; a first data panel for first event data describing one or more of the first subset of the events; a second histogram representing the second histogram data and comprising a second set of bins graphically representing the second set of event clusters; and a second data panel for second event data describing one or more of the second subset of the events;

in response to a user selection of one of the first set of bins, updating the graphical user interface to include: in the first histogram, a visual indication of the selected bin; in the second histogram, a visual indication of a corresponding bin, wherein the selected bin and the corresponding bin represent event clusters for the same time bin; in the first data panel, a visual indication of first event data that corresponds to the selected bin in the first histogram; and in the second data panel, a visual indication of second event data that corresponds to the corresponding bin in the second histogram.

2. The method of claim 1, comprising:

identifying a characteristic of interest;

identifying the first subset of the events based on the characteristic of interest, wherein the first subset of events has a first value for the characteristic of interest; and

identifying the second subset of the events based on the characteristic of interest, wherein the second subset of events has a second, distinct value for the characteristic of interest.

3. The method of claim 2, wherein the characteristic of interest comprises an event source, an event location, or an event destination.

4. The method of claim 1, wherein:

the first histogram and the second histogram are presented side-by-side in a first section of the graphical user interface;

the first data panel and the second data panel are presented side-by-side in a second section of the graphical user interface, wherein the first section is presented above or below the second section.

5. The method of claim 4, wherein:

the first data panel comprises first segments corresponding to the first set of bins, each first segment comprising a text representation of one or more events in a respective one of the first set of bins and a numerical representation of the time bin; and

the second data panel comprises second segments corresponding to the second set of bins, each second segment comprising a text representation of one or more events in a respective one of the second set of bins and a numerical representation of the time bin.

6. The method of claim 5, wherein the each of the first segments is aligned with a respective one of the second segments, such that each pair of aligned first and second segments correspond to a respective one of the time bins.

7. The method of claim 5, comprising, in response to a user selection of one of the first segments, expanding the selected segment to display a text representation of additional events.

8. A computer system comprising:

one or more processors; and

a computer-readable medium storing instructions that are operable when executed by the one or more processors to perform operations for displaying search result data for an observability pipeline system, comprising: obtaining search results, the search results identified based on searching data generated by an observability pipeline system, the search results comprising events processed by the observability pipeline system; identifying time bins based on the search results; generating first histogram data based on the time bins and the events, the first histogram data comprising a first set of event clusters for the respective time bins, wherein the first set of event clusters correspond to a first subset of the events; generating second histogram data based on the time bins and the events, the second histogram data comprising a second set of event clusters for the respective time bins, wherein the second set of event clusters correspond to a second, distinct subset of the events; generating a graphical user interface comprising: a first histogram representing the first histogram data and comprising a first set of bins graphically representing the first set of event clusters; a first data panel for first event data describing one or more of the first subset of the events; a second histogram representing the second histogram data and comprising a second set of bins graphically representing the second set of event clusters; and a second data panel for second event data describing one or more of the second subset of the events; in response to a user selection of one of the first set of bins, updating the graphical user interface to include: in the first histogram, a visual indication of the selected bin; in the second histogram, a visual indication of a corresponding bin, wherein the selected bin and the corresponding bin represent event clusters for the same time bin; in the first data panel, a visual indication of first event data that corresponds to the selected bin in the first histogram; and in the second data panel, a visual indication of second event data that corresponds to the corresponding bin in the second histogram.

9. The computer system of claim 8, wherein the operations comprise:

identifying a characteristic of interest;

identifying the first subset of the events based on the characteristic of interest, wherein the first subset of events have a first value for the characteristic of interest; and

identifying the second subset of the events based on the characteristic of interest, wherein the second subset of events have a second, distinct value for the characteristic of interest.

10. The computer system of claim 9, wherein the characteristic of interest comprises an event source, an event location, or an event destination.

11. The computer system of claim 8, wherein:

the first histogram and the second histogram are presented side-by-side in a first section of the graphical user interface;

the first data panel and the second data panel are presented side-by-side in a second section of the graphical user interface, wherein the first section is presented above or below the second section.

12. The computer system of claim 11, wherein:

the first data panel comprises first segments corresponding to the first set of bins, each first segment comprising a text representation of one or more events in a respective one of the first set of bins and a numerical representation of the time bin; and

the second data panel comprises second segments corresponding to the second set of bins, each second segment comprising a text representation of one or more events in a respective one of the second set of bins and a numerical representation of the time bin.

13. The computer system of claim 12, wherein the each of the first segments is aligned with a respective one of the second segments, such that each pair of aligned first and second segments correspond to a respective one of the time bins.

14. The computer system of claim 12, wherein the operations comprise:

in response to a user selection of one of the first segments, expanding the selected segment to display a text representation of additional events.

15. A non-transitory computer-readable medium storing instructions that are operable when executed by a data-processing apparatus to perform operations comprising:

obtaining search results, the search results identified based on searching data generated by an observability pipeline system, the search results comprising events processed by the observability pipeline system;

identifying time bins based on the search results;

generating first histogram data based on the time bins and the events, the first histogram data comprising a first set of event clusters for the respective time bins, wherein the first set of event clusters correspond to a first subset of the events;

generating second histogram data based on the time bins and the events, the second histogram data comprising a second set of event clusters for the respective time bins, wherein the second set of event clusters correspond to a second, distinct subset of the events;

generating a graphical user interface comprising: a first histogram representing the first histogram data and comprising a first set of bins graphically representing the first set of event clusters; a first data panel for first event data describing one or more of the first subset of the events; a second histogram representing the second histogram data and comprising a second set of bins graphically representing the second set of event clusters; and a second data panel for second event data describing one or more of the second subset of the events;

in response to a user selection of one of the first set of bins, updating the graphical user interface to include: in the first histogram, a visual indication of the selected bin; in the second histogram, a visual indication of a corresponding bin, wherein the selected bin and the corresponding bin represent event clusters for the same time bin; in the first data panel, a visual indication of first event data that corresponds to the selected bin in the first histogram; and in the second data panel, a visual indication of second event data that corresponds to the corresponding bin in the second histogram.

16. The non-transitory computer-readable medium of claim 15, wherein the operations comprise:

identifying a characteristic of interest;

identifying the first subset of the events based on the characteristic of interest, wherein the first subset of events have a first value for the characteristic of interest; and

identifying the second subset of the events based on the characteristic of interest, wherein the second subset of events have a second, distinct value for the characteristic of interest.

17. The non-transitory computer-readable medium of claim 16, wherein the characteristic of interest comprises an event source, an event location, or an event destination.

18. The non-transitory computer-readable medium of claim 15, wherein:

the first histogram and the second histogram are presented side-by-side in a first section of the graphical user interface;

the first data panel and the second data panel are presented side-by-side in a second section of the graphical user interface, wherein the first section is presented above or below the second section.

19. The non-transitory computer-readable medium of claim 18, wherein:

the first data panel comprises first segments corresponding to the first set of bins, each first segment comprising a text representation of one or more events in a respective one of the first set of bins and a numerical representation of the time bin; and

the second data panel comprises second segments corresponding to the second set of bins, each second segment comprising a text representation of one or more events in a respective one of the second set of bins and a numerical representation of the time bin.

20. The non-transitory computer-readable medium of claim 18, wherein the each of the first segments is aligned with a respective one of the second segments, such that each pair of aligned first and second segments correspond to a respective one of the time bins.