Visualization of Unique Field Values for a Field in a Set of Events
Systems and methods are provided for visualizing the number of events having different values for a field of interest over a selected time range. The events may be derived from machine data obtained from one or more data sources. User input received via a graphical user interface may specify the field of interest, a time range, and a time granularity for displaying counts of the number of events having various values during different time slots within the selected time range. Events including the specified field during the user-selected time range are identified and values for the field are extracted from the identified events. A visualization indicating a relation between a number of the events occurring within each of a plurality of time slots over the selected time range and each of the unique extracted values of the field is provided to the user via the graphical user interface.
This application claims benefit under 35 U.S.C. §120 as a Continuation of application Ser. No. 14/165,232, filed Jan. 27, 2014. The entire contents of which is hereby incorporated by reference as if fully set forth herein. Applicant hereby rescinds any disclaimer of claim scope in the parent application(s) or the prosecution history thereof and advise the USPTO that the claims in this application may be broader than any claim in the parent application(s).
TECHNICAL FIELDThe present disclosure relates generally to techniques for processing and visualizing data field values over a period of time.
BACKGROUNDEnterprise organizations and the data analysts they employ face the challenge of finding useful information in the increasing amounts of data generated and collected by these organizations over time. Such “big data” may provide, for example, valuable insights into the organization's operational performance and business patterns associated with various parts of the organization. For example, accessing computer networks of a business enterprise and transmitting electronic communications across these networks generates massive amounts of data. Such data generated by machines may include, for example, Web logs of activity occurring at various web servers distributed across an enterprise's network.
Analysis of this data can indicate patterns in consumer behavior with respect to the particular products or brands in which consumers may be interested during a given period of time. Such pattern analysis also may be helpful in differentiating normal operational performance from anomalies. For example, the detection of unusual patterns can allow a system analyst to investigate the circumstances under which these unusual patterns emerged and determine whether any issues exist that may pose a threat to the system's operational performance or security. Moreover, analysis of such data allows business enterprises to understand how their employees, potential consumers, and/or Web visitors use the company's online resources. Such analysis can therefore provide businesses with operational intelligence, business intelligence, and an ability to better manage their information technology (IT) resources. For instance, such analysis may enable a business to better retain customers, meet customer needs, and improve the efficiency and security of the company's IT resources.
However, data analysts or systems administrators of an enterprise may encounter significant challenges when attempting to identify, collect, and analyze such large quantities of data, which may be distributed across multiple data sources within the enterprise's network environment or IT infrastructure. Such challenges may prevent these enterprise users from realizing the potential value that this data may provide. In particular, patterns in the enterprise's data as a whole, which may provide valuable insight into the operations of the enterprise, may be difficult to find due in part to the size of this data and the fact that the underlying data produced by each data source within the enterprise is usually analyzed in isolation, if at all.
SUMMARY OF A FEW EMBODIMENTSEmbodiments of the present disclosure relate to, among other things, visualizing values over time in a field defined for a set of events, which may be derived from machine data, log data, and/or other data. Each of the embodiments disclosed herein may include one or more of the features described in connection with any of the other disclosed embodiments.
In one embodiment, a method is disclosed for visualizing, over time, values of a field in events that may be derived wholly or partially from machine data. An input may be received from a user via a graphical user interface. The input specifies a field and a time range. A set of events may be identified based on the input received from the user. Each event in the identified set may occur during the time range and
may include a value for the specified field. A set of unique values for the field may be determined from the identified set of events. For each unique value in the set of unique values, a subset of events including that unique value for the field may be identified.
Each event in the identified subset may include a time-stamp coinciding with one of a plurality of time slots within the time range. A visualization of counts of events from each of the subset of events identified for each unique value of the field within the time range may be provided. The visualization may display a set of rows intersecting with a set of columns, where each row corresponds to one unique value in the set of unique values, each column corresponds to one of the plurality of time slots, and each intersection of a row and a column provides an indication of a number of events including the unique value corresponding to the row and having a time-stamp coinciding with the time slot corresponding to the column.
Various embodiments of the method may include one or more of the following features: the events may be derived at least in part from machine data; the events are derived at least in part from log files generated by one or more servers; the indication of the number of events may be an absolute or relative indication of the number of events that is provided using a color or shade; the color or shade may be applied to each intersection according to a linear scale; the color or shade may be applied to each intersection according to a logarithmic scale; the color or shade may be applied to each intersection according to an exponential scale; the color or shade may be applied to each intersection according to a rank assigned to that intersection based on the corresponding number of events; the color or shade may be applied to each intersection across each individual row, each individual column, a subset of rows and
columns selected by the user, or all displayed rows and columns of the visualization; the method may further include steps of receiving input from the user specifying a time granularity via the graphical user interface, the graphical user interface including a control element for enabling the user to vary the time granularity, and adjusting a duration of time covered by each of the plurality of time slots based on the received time granularity; the method may further include steps of receiving user input selecting a
header portion of a column in the set of columns of the visualization and sorting the set of rows in ascending or descending order according to the number of events including the value corresponding to each row in the set of rows, based on the received user input; the visualization may include a statistics table displaying a set of statistics calculated for each unique value in the set of unique values for the field, and the set of statistics is calculated based on the identified subset of events for each unique value; the visualization provided to the user may be a heat map indicating variations in an event count representing the one or more events coinciding with each of the plurality of time slots over the selected time range for each of the unique values of the specified field; and the graphical user interface may enable the user to reorder each of the set of rows by using a drag and drop gesture with a user input device.
In another embodiment, a system may include a memory having processor-readable instructions stored therein and a processor configured to access the memory and execute the processor-readable instructions, which, when executed by the processor, configures the processor to perform a plurality of functions, including functions to: receive an input from a user via a graphical user interface, where the input may specify a field and a time range; identify events within the machine data based on the input received from the user, where each event in the identified set occurring within the time range and including a value for the specified field; determine a set of unique values for the field from the identified set of events; for each unique value in the set of
unique values, identify a subset of events including that unique value for the field, each event in the identified subset having a time-stamp coinciding with one of a plurality of time slots within the time range; and provide a visualization of events from each of the subset of events identified for each unique value of the field within the time range, where the visualization displays a set of rows intersecting with a set of columns, each row corresponds to one unique value in the set of unique values, each column corresponds to one of the plurality of time slots, and each intersection of a row and a column provides an indication of a number of events including the unique value corresponding to the row and having time-stamps coinciding with the time slot corresponding to the column.
Various embodiments of the system may include one or more of the following features: the events may be derived at least in part from machine data; the events may be derived at least in part from log files generated by one or more servers; the indication of the number of events may be an absolute or relative indication of the number of events that is provided using a color or shade; the color or shade may be applied to each intersection according to a linear scale; the color or shade may be applied to each intersection according to a logarithmic scale; the color or shade may be applied to each intersection according to an exponential scale; the color or shade is applied to each intersection according to a rank assigned to that intersection based on the corresponding number of events; the color or shade may be applied to each
intersection across each individual row, each individual column, a subset of rows and columns selected by the user, or all displayed rows and columns of the visualization; the processor may be further configured to receive input from the user specifying a time granularity via the graphical user interface, the graphical user interface including a control element for enabling the user to vary the time granularity, and adjust a duration of time covered by each of the plurality of time slots based on the received time granularity; the processor may be further configured to receive user input selecting a header portion of a column in the set of columns of the visualization, and sort the set of rows in ascending or descending order according to the number of events including the value corresponding to each row in the set of rows, based on the received user input; the visualization may include a statistics table displaying a set of statistics calculated for each unique value in the set of unique values for the field, and the set of statistics is calculated based on the identified subset of events for each unique value; the visualization provided to the user may be a heat map indicating variations in an event count representing the one or more events coinciding with each of the plurality of time slots over the selected time range for each of the unique values of the specified field; and the graphical user interface may enable the user to reorder each of the set of rows by using a drag and drop gesture with a user input device.
In a further embodiment, a computer readable medium includes stored instructions that, when executed by a computer, cause the computer to perform functions to: receive an input from a user via a graphical user interface, where the input may specify a field and a time range; identify events within the machine data based on the input received from the user, where each event in the identified set occurring within the time range and including a value for the specified field; determine a set of unique values for the field from the identified set of events; for each unique value in the set of unique values, identify a subset of events including that unique value for the field, each event in the identified subset having a time-stamp coinciding with one of a plurality of time slots within the time range; and provide a visualization of events from each of the subset of events identified for each unique value of the field within the time range, where the visualization displays a set of rows intersecting with a set of columns, each row corresponds to one unique value in the set of unique values, each column corresponds to one of the plurality of time slots, and each intersection of a row and a column provides an indication of a number of events including the unique value corresponding to the row and having time-stamps coinciding with the time slot corresponding to the column.
In yet a further embodiment, a computer readable medium includes stored instructions that, when executed by a computer, cause the computer to perform functions to: display a graphical user interface enabling a user to specify a field and a time range; receive through the graphical user interface a selection of the field and the time range; identify a set of events for which the field has been defined and that are
stored in a time series data store, and that have associated time-stamps falling within the time range; determine a set of unique values for the field in the events; for each unique value in the set of unique values, determine a number of events having that unique value for the field and having a time-stamp falling within each of a set of time slots within the time range; display a set of rows, each corresponding to one of the
unique values, wherein each row contains a set of columns, each column corresponding to one of the time slots; and for a set of heat map boxes at intersections between a row and a column, provide an absolute or relative indication of the number of events having a value corresponding to the row and a time-stamp falling within the time slot corresponding to the column.
It may be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and together with the description, serve to explain the principles of the disclosure.
The present disclosure relates to systems and methods for visualizing values over time of a field identified in events that may be derived from data including, e.g., machine data. In an example, data generated by various data sources is collected and segmented into discrete events, each event corresponding to data from a particular point in time. Examples of such data sources include, but are not limited to, web servers, application servers, databases, firewalls, routers, operating systems, software applications executable at one or more computing devices within the enterprise data system, mobile devices, and sensors. The types of data generated by such data
sources may be in various forms including, for example and without limitation, server log files, activity log files, configuration files, messages, network packet data, performance measurements or metrics, and sensor measurements. The data sources may be associated with, for example, an enterprise data system distributed across a network environment. The events, which may be derived from indexing or segmenting the machine data or other data generated by these data sources, may be used to provide search and data analysis functionality to a user of the enterprise data system, e.g., a
data analyst or systems engineer interested in gaining a better understanding of the performance and/or security of an enterprise organization's information technology (IT) infrastructure. As will be described in further detail below, such functionality may include the visualization of events and values for a specified field that may be extracted from the events occurring during a given time period. In some embodiments, the visualization may be of a count or other statistic for visualizing the occurrence over time of events, by a plurality of unique values for the specified field. For example, the visualization may represent how many times events having each of the unique values for the specified field occurred during each of a plurality of time slots extending over the given time period.
While the present disclosure is described herein with reference to illustrative embodiments for particular applications, it should be understood that embodiments are not limited thereto. Other embodiments are possible, and modifications can be made to the embodiments within the spirit and scope of the teachings herein and additional fields in which the embodiments would be of significant utility.
It would also be apparent to one of skill in the relevant art that the present disclosure, as described herein, can be implemented in many different embodiments of software, hardware, firmware, and/or the entities illustrated in the figures. Any actual software code with the specialized control of hardware to implement embodiments is not limiting of the detailed description. Thus, the operational behavior of embodiments will be described with the understanding that modifications and variations of the embodiments are possible, given the level of detail presented herein.
In the detailed description herein, references to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
In an embodiment, “time-series data” and “time-series machine data” may include, among other things, a series or sequence of data points generated by one or more data sources or computing devices. Each data point may be associated with a time-stamp or be associated with a particular point in time that provides the basis for a time-stamp for the data point, and the series of data points may be plotted over a time range or time axis representing at least a portion of the time range. The data can be structured, unstructured, or semi-structured and can come from files, directories, and/or network events. Unstructured data may refer to data that is not organized according to,
for example, a predefined schema to facilitate the extraction of values or fields from the data. Machine data generated by, for example, data sources within an enterprise network environment is generally unstructured data. As will be described in further detail below, the visualization of such time-series data may be used to display statistical trends over time. The time-series machine data collected from a data source may be segmented or otherwise transformed into discrete events, where each event can be associated with a time-stamp.
In an embodiment, an “event” may include, among other things, a single piece of data corresponding to a time-stamped record of activity associated with a particular data source. Such an event may correspond to, for example, a record in a log file or other data input. In some instances, a single event may correspond to a single line in a log file or other data input. However, some inputs may have multiline events, for example, XML logs, and some inputs may have multiple events corresponding to a single line or record within the log file. Further, “events” may include, among other things, all of the events that may be derived from processing or indexing machine data, as will be described in further detail below. Events can also correspond to any time-series data, such as performance measurements of an IT component (e.g., a computer cluster, node, host, or virtual machine), or a sensor measurement including, but not limited to, sensor data from an accelerometer,
gyroscope, digital compass, barometer, location data from a Global Positioning System (GPS) or other type of sensor or device used for location determination (e.g., Wi-Fi, cell-10, and data from a Radio-Frequency Identification (RFID) reader, Near Field Communication (NFC) reader, or the like. The execution of a query or search for a name or keyword within the various stored events, or for events whose values for various fields meet various criteria, or for events occurring at particular times, may produce one or more events responsive to the particular query.
In an embodiment, a “field” may include, among other things, any searchable name/value pair that may appear within the events derived from data, such as machine data. In an example, a data intake and query system within an enterprise network environment may be configured to automatically extract certain fields from the events upon being segmented, indexed, or stored. A field may be defined by a user at any time to enable the representation of the occurrence of events containing values for that user-defined field. A field also may correspond to metadata about the events, such as a time-stamp, host, source, and source type for an event. Such metadata fields may, in some cases, be referred to as “default fields,” based on the fields being derived for all events at the time of segmenting, indexing, and/or storing of the events within one or more data stores, as will be described in further detail below. Values for these and other fields, such as user-defined fields, may be extracted from the events themselves or determined for a particular event from other sources related to the event, e.g., interpolated or extrapolated based on values for the same field included within other
events occurring within a series of events including the particular event in question. Also, user-specified fields may be extracted from the events at either index time, storage time, or search time, e.g., upon the execution of a search or query for events matching certain user-specified criteria. In some implementations, tags or aliases may be assigned to any field/value combination, for example, in order to identify fields with different names that contain equivalent pieces of information.
Exemplary EmbodimentsIn the example shown in
a smartphone or other type of mobile computing device. However, client device 110 and/or user device 120 may be implemented using a system that includes multiple devices and/or components.
Similarly, data intake query system 145 and visualization system 150 may be implemented using one or more computing devices. In an example, data intake and query system 145 and visualization system 150 may be implemented using one or more servers. Such a server may include, but is not limited to, a web server, a data server, a proxy server, a network server, or other type of server configured to provide data services or exchange electronic information with other servers and other types of computing devices (e.g., client device 110 and user device 120) via network 140. Such a server may be implemented using any type of general purpose computer that includes, for example and without limitation, at least one processor and a memory for executing and storing processor-readable instructions. The memory may include any type of random access memory (RAM) or read-only memory (ROM) embodied in a physical storage medium, such as magnetic storage including floppy disk, hard disk, or magnetic tape; semiconductor storage such as solid state disk (SSD) or flash memory; optical disc storage; or magneto-optical disc storage. Software may include one or
more applications and an operating system. Hardware can include, but is not limited to, a processor, memory, and a display (e.g., for displaying a graphical user interface).
Such a server may also be implemented using multiple processors and multiple shared or separate memory devices within, for example, a clustered computing environment or server farm.
In some implementations, data intake and query system 145 and visualization system 150 may be communicatively coupled to each other via a direct or indirect connection within, for example, a private network that may be accessible through a firewall via network 140. Further, data intake and query system 145 and visualization system 150 may be implemented as components of a single, integrated data management system, e.g., at a server (not shown) within enterprise network environment 100. Alternatively, data intake and query system 145 or visualization system 150 may be implemented as separate components of a distributed system including various computing devices communicatively coupled to one another via network 140. Alternatively, the functionality of some or all of the visualization system 150 could be included in software running on the client device 110 or user device 120.
Network 140 may be any type of electronic network or combination of networks used for communicating digital content and data between various computing devices. Network 140 may include, for example, a local area network, a medium area network, or a wide area network, such as the Internet. In addition, network 140 can include, but is not limited to, a wired (e.g., Ethernet) or a wireless (e.g., Wi-Fi, 3G, or 4G) network. Network 140 can support any of various protocols and technology
including, but not limited to, Internet protocols and/or data services. While not shown in
While not shown in
the data sources may be a web server, and the time-stamped events may correspond to entries within a log file generated by the web server over a given time period. The events in this example may correspond to, for example and without limitation, requests and responses related to web content to and from the web server and one or more computing devices associated with end-users of a service provided by the web server, e.g., via one or more web pages that may be loaded into a web browser executable at each end-user's device. The requests and responses may be in the form of, for
example and without limitation, standard Hypertext Transfer Protocol (HTTP) requests and responses between the web server and each end-user's device. Other examples of events that may be derived from log files or other types of machine data generated by a
data source include, but are not limited to, the execution of a program or script, a fault exception, an unhandled interrupt, and any security threats or other potential issues detected during the period of time. Further, an individual event may be derived from machine data corresponding to, for example, a single line of machine data within an activity log, multiple lines of machine data, a whole document, an entire configuration file, or a complete stack trace generated by the data source. Additional characteristics and features of data intake and query system 145 will be described in further detail below with respect to
In an example, client 105 and user 115 may be users of a client application executable at client device 110 and user device 120, respectively. Client 105 may be, for example, a data analyst or systems engineer within an IT department of an enterprise organization, while user 115 may be, for example, a non-technical user within a business operations or marketing department of the enterprise organization.
The client application executable at each device may enable client 105 and user 115 to interact with data intake and query system 145 and/or visualization system 150 for obtaining and analyzing different values of a machine data field over a selected time range, as will be described in further detail below. The client application may provide client 105 and user 115 with an interface for accessing the functionality provided by a data management system, e.g., including data intake and query system 145 and visualization system 150 of network environment 100. The interface may be, for
example, a GUI and/or an application programming interface (API), for enabling client 105 and user 115, or the client application executable at each of client device 110 and user device 120, respectively, to access the functionality provided by data intake and query system 145 or visualization system 150. It should be noted that in some implementations client 105 or user 115 may be an application, a service, utility, script or program written in any of various scripting languages, which may be configured to programmatically interface with the client application executable at client device 110 or user device 120, respectively.
While not shown in
In a further example, user 115 can utilize visualization system 150 or interface thereof provided via the client application executable at user device 120, as described above, in order to view the absolute and relative timings of events with respect to different values of a specified field over a selected time range. As will be described in further detail below, a visualization of events including each of a plurality of field values over time may be provided to the user via a GUI of the client application.
The GUI may, for example, allow the user to select a desired time range for the visualization of events occurring at various points during the time range. The time range may be any time period of interest including, but not limited to, minutes, hours, days, weeks, months, years, or a custom time range within any one or a combination of the preceding time periods. In one embodiment, the time range may be defined by the scope of the events represented in an entire visualization. Although the present disclosure describes a user selection of a time range for limiting the scope of events visualized, it should be appreciated that in some embodiments, the time range for the visualization may be based on the time-stamps of the events derived from the collected machine data or other data and stored within the data store(s), as described above.
The GUI may also allow the user to select a time granularity for specifying the duration of each of a plurality of time slots within the time range, to better represent the various times during which events occur during the time range. Thus, the time granularity may be used to define the unit or duration of time covered by each time
slot over the selected time range of interest. In some implementations, the duration of the time slots over the user-selected time range can be apportioned automatically without user input, e.g., based on a predefined time granularity. By way of example, if the time range is the past day (e.g., yesterday, or a preceding period of 24 hours), the time granularity may be set either automatically or by the user to 30-minute increments, thereby creating forty-eight 30-minute time slots visualized over the time range. If the time range is the past week, the time granularity may be set automatically or by the user to 12-hour increments, thereby creating fourteen 12-hour time slots visualized over the time range.
In an example, the visualization may be in the form of a heat map including a tiled or tessellated matrix of a set of rows and a set of columns, in which each of the unique values for the specified field may correspond to different rows of events, and each of the plurality of time slots for the events in each row may correspond to different columns of the heat map, as will be described in further detail below. The selected time range in this example may define a visible time range including the plurality of time slots displayed along a time axis of the visualization within a content viewing area or visualization window of the GUI. Thus, the visualization for each value of a specified field along the axis may be displayed as, for example, a row of equally-distributed time slots or “buckets” indicating the number of events occurring for each value of the specified field over the visible time range. Each time slot or bucket in this example may be used to indicate to the user that one or more events including a particular field value occurred at a particular point in time coinciding with the individual unit or duration of time represented by the time slot or bucket. In one embodiment, each time slot or bucket may be referred to, or considered to be, an “intersection” of a row corresponding to a unique one of the field values and a column corresponding to a unique one of the time slots. Alternatively, each time slot or bucket may be referred to, or considered to be, a “cell” of a table having a temporal distribution along one of the table's column and row headers, and a field value distribution over the other of the column and row headers. Also, as will be described in further detail below, the visualization of a bucket or time slot may vary according to the number of events associated with the bucket or time slot, e.g., by varying a gradient of the color or shade used to display the individual time slots or buckets within the visible time range.
In an example, the number of events (or event count) associated with each bucket or time slot for a particular field value may be based on the results of a query for events including the field value and having a time-stamp coinciding with the particular time slot within the selected time range. In some implementations, such a query may be generated dynamically by the data management system, e.g., in response to the receipt of user input specifying the field via the GUI. The criteria for the query may be based on, for example, the type of field or field values, as will be described in further detail below. Further, each query may include one search command or a series of search commands, e.g., in the form of a pipelined query or search pipeline, to be executed by a search head (e.g., search head 225 of
example, the various commands of such a search pipeline may be separated by a special operator or “pipe character” (e.g., “I”) and the output or result of one command (e.g., to the left of the pipe character) may serve as the input to the next command (e.g.,
to the right of the pipe character). The search query in this example may be, for example, a single query (e.g., a pipelined query), which may be used to retrieve all events that occur within a given time range and that include a value for a given field of interest. Accordingly, the results obtained from executing this query may be used to identify all the different unique values for a given field of interest, all the events within the time range that include these values, and the particular points in time when these events occurred during the time range.
sources 205a, 205b, and 205c and forward the data to indexers 215a, 215b, and 215c. However, it should be noted that system 145 could include a greater or fewer number of forwarders and/or indexers, as desired for a particular implementation. Further, system
145 may collect and process machine data or other data from any number of data sources, including additional data sources (not shown) that may be communicatively coupled to forwarder 210a or forwarder 210b or to one or more additional forwarders (not shown) that may be included in system 145. In an example, forwarder 210a may collect and forward streams of time-series machine data or other data generated by sources 205a and 205b to one or more of indexers 215a, 215b, and 215c. Similarly, forwarder 210b may collect and forward the time-series machine data generated by source 205c. It should be noted that the components of system 145, including forwarders 210a and 210b, indexers 215a, 215b, and 215c, and/or search head 225, may be implemented at a single server or across multiple servers or computing devices that are communicatively coupled in a distributed network environment (e.g., servers within a server farm).
Data sources 205a, 205b, and 205c may include computers, routers, databases, operating systems, and applications. Each of data sources 205a, 205b, and 205c may generate one or more different types of machine data including, but not limited to, server logs, activity logs, configuration files, messages, database records,
and the like. The machine data or other data produced by data sources 205a, 205b, and 205c may arrive at forwarder 210a or forwarder 210b as, for example, a series of time-stamped records of relevant activities or operations occurring at each data source over time. Further, such time-series machine data may be collected by forwarder 210a
or 210b in real-time, e.g., as a real-time data stream or feed to which forwarder 210a or 210b may be subscribed. Alternatively, the machine data may be collected or retrieved by forwarder 210a or 210b from each data source at periodic time intervals.
In the example shown in
Each of data stores 220a, 220b, and 220c may be implemented using any type of recording medium for storing different types of data accessible to indexers 215a, 215b, and 215c and search head 225. In some implementations, search head 225 may search events and fields in real-time, without having to access data stores 220a, 220b,
or 220c. For example, search head 225 may receive real-time streams of indexed event data directly from indexers 215a, 215b, and 215c. In an example, indexers 215a,
215b, and 215c, data stores 220a, 220b, and 220c, and search head 225 may represent different parts of a distributed network environment (e.g., enterprise network
environment 100 of
functions including, but not limited to, directing search requests to a set of search peers, e.g., indexers 215a, 215b, and 215c, and then, merging the results from each search peer for display to the user. Accordingly, search head 225 may be implemented using, for example, a centralized server communicatively coupled to indexers 215a, 215b, and 215c and data stores 220a, 220b, and 220c via a network (e.g., network 140 of
network environment 100 of
one embodiment, the assignment of parallel processing of searches to the various distributed indexers on their partial event stores may be referred to as a “map-reduce” process.
As noted above, the components of system 145, including forwarders 210a and 210b, indexers 215a, 215b, and 215c, and/or search head 225, may be implemented at a single server or across multiple servers or computing devices that are communicatively coupled in a distributed network environment. For example, each component may be implemented using a different computing device having at least one processor, a memory, and a network communications interface. Similarly, data stores 220a, 220b, and 220c may be implemented using separate data storage devices that may be accessible to the other components of system 145 via a network (e.g., network 140 of
Additional details of the features and operations of system 145, including forwarders 210a and 210b, indexers 215a, 215b, and 215c, data stores 220a, 220b, and 220c, and search head 225, will be described below with respect to
data intake and query system 145 of
Method 300 begins in step 305, which includes receiving data generated by one or more sources, e.g., sources 205a, 205b, and 205c of
A time-stamp may also be determined for each event in step 315. The time-stamp can be determined by any suitable means, including, e.g., extracting a time field from data in an event or by interpolating the time based on time-stamps extracted
from other events occurring shortly before or after the event within a particular time frame of activity associated with the same data source. In some implementations, the time-stamp for an event may correspond to the time the event data was received or generated. The time-stamp determined for each event is associated with the event in step 320. For example, the time-stamp may be stored as metadata for the event.
In step 325, the data included in a given event may be optionally transformed. Such a transformation may include, for example, removing part of an event (e.g., a portion used to define event boundaries) or removing redundant portions of an event. A user or client may specify a portion to remove using a regular expression or other type of input provided via an interface of the data intake and query system described herein.
Optionally, a keyword index can be generated to facilitate fast keyword searching of events. To build such an index, method 300 may proceed to steps 330 and 335. In step 330, a set of keywords or tokens included within the events may be identified. In step 335, each identified keyword or token may be added to a keyword index associating the keyword/token with one or more events that each include the keyword/token. In some implementations, the keyword index may include a pointer for each keyword to the corresponding event(s) including that keyword (or locations within events where the particular keyword may be found). Alternatively, the keyword index may include some other type of reference or indicator specifying how the events
including each keyword may be retrieved. When a keyword-based query is received by an indexer, the indexer may then consult this index to relatively quickly find those events containing the keyword without having to examine again each individual event, thereby greatly accelerating keyword searches.
In step 340, the events are stored in one or more data stores (e.g., data stores 220a, 220b, and 220c of
by time range. In some implementations, the time-stamps may be used to distribute the stored data across a plurality of individual time slots, each of which may correspond to a storage location for events having time-stamps that coincide with the time slot. An
event then can be associated with a storage location corresponding to a time slot representing a time period that is inclusive of the event's time-stamp. This may help optimize time-based searches by allowing events with recent time-stamps that may have a higher likelihood of being accessed to be stored at preferable memory locations (such as flash memory instead of hard-drive memory) that lend to quicker subsequent retrieval.
Referring back to the example shown in
events generated by system 145. In some implementations, the stored event data may be distributed among indexers 215a, 215b, and 215c so as to enable parallel searching of events in response to a query initiated by a user or client (e.g., user 115 or client 105 of
Also, as will be described in further detail below, a visualization system (e.g., visualization system 150 of
In an example, a set of default or predefined fields may be extracted from the event data at index time or storage time, e.g., by indexers 215a, 215b, and 215c. Other fields may be defined and included in the schema for the events at any time, up to and including search time. Examples of default fields or metadata that may be determined for each event include, but are not limited to, host, source, source-type, and time (e.g., based on the time-stamp for the event), as described above. In another example, a value for a field may be extracted from an event at search time, and the schema in this example may be referred to as a late-binding schema, as mentioned
above and as will be described in further detail below. The extraction rule for a field may include a regular expression (or “regex” or any other suitable expression) or any other rule for how to extract a value from an event. In some implementations, the visualization system may provide the user with an interactive field extraction
functionality via the GUI, which enables the user to create new custom fields. Additional details of the features and operations of the visualization system will be described below with respect to
and query system 145 of
be appreciated that method 400A may include additional steps than those shown in
As shown in
application executable at the user's device (e.g., client device 110 or user device 120 of
GUI from that provided in step 410 for presenting the obtained event information to a user, e.g., for data analysis purposes.
In step 410, method 400A may further include providing a GUI for presenting the obtained events to a user. The GUI in this example may be provided to the user via, for example, a client application executable at the user's computing device (e.g., user device 120 of
application may be a standalone software application executable at the user's device for providing the visualization functionality described herein.
In step 415, input may be received from the user via the GUI. The received input may specify a field and a time range for displaying occurrences of one or more events including the field during the selected time range. As described above, the field and the time range may be selected by the user via the same or different GUI provided by the client application executable at the user's device. In an example, the field selected by the user may be extracted from the events at search time, e.g., at the time a query including one or more search commands (e.g., in a search pipeline) is executed for a late-binding schema, as described above and as will be described in further detail below. Such a search-time field extraction may be based on, for example,
a field definition or configuration specified by the user via an interactive field extraction functionality accessible through the GUI, through regular expressions included within a configuration file accessible to the data intake and query system, or through a search command provided as part of the query itself. In some implementations, the user may specify the field via an input control element provided by the GUI, e.g., by selecting a desired field from a list of fields extracted from the events and prepopulated within a menu, dropdown window, or other type of control element for field selection, as provided by the GUI for a particular implementation. The list of fields may also include, for example, any default fields and/or user-defined fields that were extracted from the
events at index and/or storage time.
Method 400A then proceeds to step 420, which may include identifying events occurring during the selected time range, where each event includes a value for the field and has a time-stamp that falls within the time range. In step 425, unique values for the specified field may be determined from the identified events. In an example, the determination in step 425 may include extracting values for the field based on a schema or definition of the field, which may be used to execute queries for events including the field and occurring within the time range. Each field in a schema may be defined for a subset of the events in a data store and may specify how to extract a value from each of the subset of events for which the field has been defined. Extraction rules for a field may be defined using, for example, a regular expression, which may be associated with a logical type of information that is included within an event for which each rule is defined.
In some implementations, the data management system of the enterprise network environment in this example may employ the specialized type of schema, referred to herein as a “late-binding schema,” as noted previously. As alluded to above, such a late-binding schema may not be defined or applied by the data intake and query system at the time of indexing the collected data, as typically occurs with conventional database technology. Rather, in a system using late-binding schema, the schema can be developed on an ongoing basis up until the time it needs to be applied, e.g., at query time. In an example of a data intake and query system (e.g., data intake and query system 145 of
user to continue developing the schema until, for example, it is needed again for executing a subsequent query to locate data within events.
In step 430, a visualization of events occurring during the time range may be provided for each unique value of the field. The visualization provided in step 430 may indicate, for example, the number of events occurring at each of a plurality of time slots that are equally distributed over the selected time range. As will be described in further detail below, the size or duration of each time slot may be based on, for example, a time granularity specified by the user via the GUI. As described above, the specified time granularity may be used to distribute the events identified in step 420 across a plurality of buckets or time slots over the selected time range, where each time slot may correspond to the same unit, increment, or period of time within the time range, as displayed along a time axis for the visualization. Thus, for each unique value of the specified field, each identified event including that value for the specified field and occurring within the specified time range based on its time-stamp may be assigned to an appropriate time slot within the time range. The unit or period of time for each time slot may be, for example, a predetermined number of seconds, hours, days, weeks, etc. An example of such a visualization is shown in
order depicted in
Method 400B may begin in step 435, which may include identifying a set of events including values for a specified field and occurring within a selected time range. As described above, the field and the time range may be based on input received from a user (e.g., at step 415 of method 400A of
In one exemplary embodiment, the events may be identified in step 435 by executing a query for events including the particular field. As described above, a set of events may be derived from data collected from one or more data sources within
an enterprise network environment (e.g., enterprise network environment 100 of
indexers. Each indexer may be responsible for indexing and storing at least a portion of the set of events derived from the data collected from the data sources, as described above. The indexers may also be utilized by a search head (e.g., search head 225 of
amongst the various indexers, e.g., which may be able to search for events responsive to a query in parallel, the search head can utilize the indexers to execute the query and obtain query results in a shorter amount of time. It should be noted that the indexers might use any conventional or proprietary search technique for executing the query.
Also, as each indexer may store only a portion of the entire set of events and thus, produce only a partial set of search results in response to the query, the search head (e.g., search head 225 of
performed in the order depicted in
Furthermore, it should be appreciated that this method may include additional steps than those shown in
As shown in step 436 of
or more distributed indexers (e.g., indexers 215a, 215b, and 215c of
field or fields. As a late-binding schema may be used, as described above, extraction of values from events based on the specified criteria may occur at the time that the query
is processed, e.g., as opposed to the time that the machine data is indexed, segmented, or stored in the form of events. It should be appreciated that, to achieve high availability and to provide for disaster recovery, events may be replicated in multiple data stores, in which case indexers with access to the redundant events would not respond to the query by processing the redundant events. The indexers may either stream the
relevant events back to the search head or use the events to calculate a partial result responsive to the query and send the partial result back to the search head. In step 439, the search head combines all the partial results or events received from the parallel processing together to determine a final result responsive to the query.
Referring back to method 400B of
In step 445, for each unique value, a subset of the events having a value matching the unique value may be identified, where each event in the identified subset has a time-stamp coinciding with one of a plurality of time slots within the time range, as described above. Also, as described above, the number of time slots and duration of each of the time slots may be based on a predetermined time granularity or may be determined based on a time granularity set by the user via the GUI for the visualization. Once the subset of events that include the particular unique value for the field is identified in step 445, the appropriate time slot for each event in the identified subset may be identified in step 450, and in step 455, the identified time slot may be associated with the corresponding event in the identified subset. In some implementations, an association between each event and the corresponding time slot may be created programmatically using, for example, a memory pointer or other type of reference object linking the event to the appropriate time slot. Such a pointer or linking reference may be associated with an instance of the event, e.g., as it is represented and stored within one or more data stores, e.g., data stores 220a, 220b, and 220c of data intake and query system 145 of
Step 460 includes counting the number of events associated with each time slot, and calculating statistics based on the event count for each of the time slots distributed across the selected time range. The event count may be used to determine a gradient for a color (or shade) in step 465, which may be used for visualizing the time slots for each field value according to the corresponding event counts. A visualization
of the calculated statistics and event count for each time slot over the selected time range is generated in step 470 based on the gradient, as will be described in further detail below with respect to
However, it should be noted that the techniques disclosed herein are not intended to be limited thereto, and that any type of visualization of the values for a field may be provided to the user via GUI 500.
As shown in
The heat map visualization may be presented for a single field, e.g., as specified by the user via a different GUI or control window (not shown) provided to the user via GUI 500, where each row of the heat map is used to display the timing of events having the value corresponding to that row as their value for the field. The heat map visualization provided within window 510 may be used to graphically represent, for example, the event count distribution for each value (i.e., shown in each row) in time slots across a selected time range, as described previously.
In the example shown in visualization window 510, the heat map displayed for each row, which corresponds to a unique value for the selected field, may be divided into a plurality of individually colored or shaded boxes or regions, each representing a time slot or “bucket” and whose color or shade indicates the number of events having the value for the field represented by that row and that have a time-stamp falling within the time slot, as described previously. The amount of time represented by each of the time slots or buckets in the heat map may be based on, for example, a time
granularity specified by the user, e.g., based on user input received via a time granularity control element of the GUI, as shown in
may be used to indicate the event count represented by a time slot or bucket in the heat map. In some implementations, the visualization of each heat map box, e.g., each square-shaped region, corresponding to a particular time slot or heat map bucket may be adjusted according to its relative event count density compared to other heat map boxes in the entire heat map, the same row of the heat map, or the same column of the heat map. The size and shape of each box or region of the heat map may also indicate this relative event count density compared to other heat map boxes in the same row or column. In an example, the size of visualized heat map boxes or regions corresponding to relatively greater event counts densities (and thus, representing a larger number of event counts) may be relatively larger than those heat map boxes/regions having relatively lower event count densities. In an example, an outline or boundary may be visualized, e.g., as an overlay, surrounding particular heat map regions so as to
highlight or otherwise visually accentuate each region to the user as it is displayed within visualization window 510. As will be described in further detail below, it should be noted that the relevant heat map boxes or regions for the visualization may include, for example, those selected by the user via GUI 500, e.g., by manipulating one or more control elements using a mouse or other type of user input device.
Various time-related controls may be provided to the user, as shown in
rows (including both the rows of the value table 530 and the rows of window 510 of
Further, while not shown in
925. Control element 921 may be implemented as, for example, an informational tooltip within a window or dialog box (e.g., a “hover box”) displayed when the user selects or hovers a selection pointer over an appropriate portion of control 925 using a mouse or other user input device.
Referring back to
events having the value for the field represented by a given row that are covered in the visible time range as a percentage of such events in the selected time range; “Avg” indicating the average number of events per time slot or bucket within the row; “Min” for indicating the minimum number of events in any of the time slots or buckets in the selected time range; and “Max” for the maximum number of events in any of the time slots or buckets in the selected time range. Those skilled in the relevant art would appreciate that any number of other statistical calculations may be performed and
displayed within value table 530, as appropriate or desired for a particular implementation. In some implementations, the user may select a “Min” or “Max” statistic displayed in table 530 for a particular field value, e.g., by using a mouse or
other user input device to click or hover a mouse/selection pointer over the statistic as it is displayed in table 530, and the corresponding time slot or heat map square representing the minimum or maximum event count for the row may be highlighted or otherwise visually indicated to the user via GUI 500.
By enabling the user to view a visualization of the number of events having various values for a field over time, GUI 500 may enable the user to notice patterns in the occurrence of values for a given field in events. Such a visualization provided via GUI 500 may also allow the user to find potential anomalies or useful patterns (e.g. periodicity) within a field's values, e.g., simply by viewing the visualization presented in visualization window 510. In an example, the user might choose to view the values for a “server status” field, which may include categorical values of server responses (e.g., various HTTP status codes, such as 200, 301, 404, etc.). The visual representation of these values over time may enable the user to determine how the server's statuses relate to each other, and possibly, detect correlations or anomalies.
Thus, the capability to visualize a field's values over time may provide the user with a better understanding of the state of the particular server.
In another example, the user might choose a field including values representing the usage percentage of processor or central processing unit (CPU) of a server or other computing device within the enterprise network environment. Such a CPU usage field may be a percentage (e.g., 56, 75, 90, 99, etc.). By visualizing values for the field over time in the events that have that field, the user may easily determine how the CPU usage may change over time and, as before, detect any correlations or anomalies in the field's values. Because the field in this example includes numerical values, the relationship between the field's values can be meaningfully represented using two or more linear axes (e.g., value and time). Examples of such a numerical field include, but are not limited to, a CPU usage field, a network throughput field (e.g., including values representing bytes transferred), or a network latency field (e.g., including response times for requests sent over the network). However, it should be noted that a numerical field may represent any type of data that can be represented by numeric values, including integers or real number values.
In the example shown in
row representing a particular string value for the field, as shown in GUI 500. Examples of categorical fields that may be extracted from indexed event data may include, but are not limited to, server responses, universal resource identifiers (URIs), or the names of an operating system or host associated with a particular data source.
In some implementations, the particular data type of the specified field may affect the particular visualization that may be used to represent the field's values over time. The values of a categorical field may be represented using, for example, a heat map, as shown in
machine data field values over time. Examples of such other types of visualizations include, but are not limited to, line graphs, bar graphs, pie charts, fractal maps, tree maps, waterfall charts, or stream-graphs including a stacked, linear, or curvilinear area graph displaced around a central axis. Furthermore, such visualizations may be represented in two-dimensional (2-D) and/or three-dimensional (3-D) forms, e.g., using shapes visualized within 2-D and/or 3-D spaces, respectively.
In an example, the user may select a particular time slot or heat map bucket in order to view additional information related to the selected time slot and the particular field value to which it corresponds. The user may be able to select the time slot by interacting directly with a corresponding box or region of the heat map displayed within visualization window 510, e.g., by selecting the region using a mouse, touchpad, keyboard, or any other user input device. The selected time slot may be within a portion 515 of the displayed heat map, as shown in
Such information may include, for example and without limitation, the corresponding field value, a count value indicating the number of events associated with the selected time slot, and a time period represented by the time slot.
As noted previously, the user may be able to select multiple heat map boxes, squares, or regions via GUI 500, e.g., by using a mouse or other user input device to “scan” or select and drag a virtual bounding box across or around one or more rows and/or columns of the heat map displayed in visualization window 510. As the
user selects additional squares representing different time slots within the heat map, the information displayed within dialog window 545 may update automatically and in real-time as each new heat map square is selected. In this way, the user may be able to select certain heat map squares corresponding to particular values and time slots of interest, while filtering or excluding other values and/or time slots from the visualization being displayed within visualization window 510. In some embodiments, after selecting a plurality of heat map squares within a virtual bounded box, a user may de-select desired squares so that information relating to the de-selected squares is excluded from the information displayed in dialog window 545.
In some implementations, the information displayed within dialog window 545 may include, for example, hyperlinks that the user may select in order to change the view, such as drilling down to a view of the underlying events falling within the selected time slot. In the example shown in
In an exemplary embodiment, this value may be displayed within window 545 as a hyperlink. For example, if the user in this example were to select the hyperlink, GUI 500 may display additional information about all of the events corresponding to heat map box 540. Along those same lines, the count (e.g., as reflected by “COUNT: 3690”) and the time range (e.g., as reflected by “3/7/2008, 12:45 AM-12:50 AM”) may each be generated as a hyperlink that a user may click or otherwise select to view the events associated with the time slot for which the tooltip containing the hyperlink was generated.
As shown in
500. Also, as shown in the example of
provided to the user may be dependent upon, for example, the particular events and user requirements for the types of control that the user may need for visualizing the data effectively within the heat map. Further, the formatting controls 550 provided may be dependent upon on the types of data underlying the heat map visualization. In an example, GUI 500 may be provided to the user via a client application executable at a mobile computing device (e.g., a smartphone or tablet) having a touchscreen display.
The user in this example may be able to use any of various single or multi-touch gestures to manipulate different control elements, e.g., any of formatting controls 550, in order to control or customize any of various aspects of the visualization displayed within visualization window 510 and/or the values and statistics displayed within value table 530.
In the example shown in
limitation: an “ALL” option for selecting the minimum and maximum for a range of event counts to be mapped from counts of all heat map boxes in the entire visualization, i.e., across all rows and columns of the heat map; a “ROW” option for selecting the minimum and maximum for a range of event counts on a row-by-row basis; and a “COLUMN” option for selecting the minimum and maximum for a range of event counts on a column-by-column basis.
Also, as shown in
for a graduated color or shading transition may change depending on whether the values are based on a single row, a single column, or all of the rows and columns in the visualization. Thus, the selection of the linear scale option may cause the color or shade to be applied to regions of the heat map (e.g., heat map boxes along a row or column) in an even, graduated transition (e.g., from lighter to darker) in constant increments from a minimum (e.g., a heat map box representing a minimum event count)
to a maximum (e.g., a heat map box representing a maximum event count). In other words, each incremental increase in the count or change in the number of events between successive heat map boxes in a row or column may result in a proportional increase in color or shading, regardless of whether the count or change is closer to a minimum count/change or a maximum count/change. For example, the visualization according to a linear scale may cause the density or intensity of the color or shade applied to a heat map box representing an event count of 100 (or “a 100-count bucket”) to be twice as much as the density or intensity of the color or shade used for a 50-count bucket.
As another embodiment, a logarithmic scale option may be selected in order to change the applied color gradient according to a logarithmic scale based on the event count for one or more values across a row or column. In this example, the gradient of the color or shade applied to rows or columns of the heat map may change gradually along a logarithmic scale from a minimum event count (or corresponding heat map box) to a maximum event count (or corresponding heat map box), in which the color or shade is applied to successive heat map boxes from a minimum event count to a maximum event count in a graduated transition using increasingly greater increments of color or shade. In this embodiment, the change in color or shade depicted by such a logarithmic scale may be used to indicate a greater degree of difference between adjacent heat map boxes representing relatively lower event counts that are closer to the minimum within the range of event counts. For example, the visualization according to a logarithmic scale may indicate a relatively greater degree of difference between the colors or shades applied to a 50-count bucket and a 60-count bucket than the difference indicated between the shades of a 150-count bucket and a 160-count bucket. An example of such a logarithmic scale is shown by line graph 600A in
The rank option may be used to assign a color gradient or level of shading to each heat map square or time slot in a linear fashion based on the rank of the particular event count. For example, for the following set of event counts {1, 76, 77, 78}, each count or numerical value within the set may be ranked, e.g., from the lowest count to the highest. Thus, the count “1” may be ranked first or lowest, “76” may be second, “77” may be third, and “78” may be fourth or highest. The heat map square corresponding to each of the event counts in this example data set may be colored according to its assigned rank. In an example, the color or shading of the heat map square having the lowest ranked count (e.g., “1”) may be only 25% of the full color or shading, the second lowest (“76”) may have 50% color, the third lowest may have 75% color, and the highest ranked square (“78”) may have 100% of the full color. However, it should be noted that any type of ranking scheme may be used to rank the event counts. Thus, in the preceding example, the ranking order may be reversed, and the count “78” may be ranked first or lowest, “77” may be second, “76” may be third, and “1” may be fourth or highest ranked count. The rank option may be useful for differentiating tightly packed data sets having counts that are relatively close in value to one another.
While not shown in
Referring back to formatting controls 550 of
Screen control 558 may be used to enable or disable a “Fit to Screen” option that affects the display of values within value table 530 and the heat map within visualization window 510. For example, when this option is disabled (e.g., set to “NO” via control 558), value table 530 is displayed such that each heat map row has a predetermined height and each heat map column has a predetermined width, and the predetermined height and width may be set to ensure that, among other things, the values displayed within table 530 are legible for the user. An example of a heat map visualization with this option selected is illustrated by a GUI 700 in
screen control 558 is not necessarily limited to vertical scrolling and that when the fit-to-screen option is disabled using control 558; as described above, additional controls (not shown) for enabling the user to scroll side-to-side may be provided as well, thereby allowing the user to modify the visible time range displayed within visualization window 510.
When the fit-to-screen option is enabled (e.g., set to “YES” via screen control 558), value table 530 may be hidden and all rows and columns of the heat map are displayed within the visible viewing area of the GUI, as shown by the exemplary GUI 800 of
In some implementations, additional controls may be provided for changing the color gradient of the heat map across a spectrum from a light color or shade to a dark color or shade, according to the corresponding event counts of the heat map squares or associated time slots. In an example, a “High” option for such a control may cause the gradient to be adjusted from a light color or shade at low event counts to progressively darker color/shade for relatively higher event counts. Conversely, the control may include a “Low” option for adjusting the gradient from a dark color at low values to progressively lighter colors or shades for relatively higher event counts.
While the exemplary GUIs described above with respect to
below with respect to
may be supported in a particular implementation. In a further example, the user may be able to specify different colors for visualizing different values (and rows). In some implementations, the user may be able to designate a particular color to be used for a category of values, for example, as may be defined based on one or more predetermined or user-specified thresholds for arranging field values according to
different time periods (e.g., including one or more time slots) within the time range and/or different ranges of event counts.
be used for displaying a visualization of the numerical field's values over the time range, or at least a visible portion thereof. Also, as shown in
Value table 1030A may include rows of the extracted values of the specified field. The values in table 1030A may correspond to the same event data that is graphed using the bubble chart. However, table 1030A may include any suitable values or statistics, as desired for a particular implementation. In one example, value table 1030A may include the date and the field value. In some implementations, an option to hide value table 1030A may be provided in order to increase the size of the visualization as it is displayed in visualization window 1040A of GUI 1000A. When a
user selects a bubble corresponding to an event, the event may be identified in the visualization.
While the exemplary visualization shown in
may be displayed using the same or a default level of opacity, it may become difficult for the user to distinguish between overlapping points or bubbles that occur around the same time within the chart and that represent values that are the same or close to each other. This may be true even when, for example, the disparity between different event counts is relatively high (e.g., 1 and 200). Thus, in some embodiments, the overlapping bubbles occurring around the same time, e.g., within a predetermined time period, may be shown using varying levels of opacity in order to make these overlapping points or bubbles easier to distinguish from one another within the chart. For example, an area having a relatively greater number of overlapping points or events, e.g., above a predetermined threshold, may be displayed using a darker shade. Thus, in some embodiments, the density measure for different event counts may be visualized such that when an opacity of overlapping points surpasses a predetermined threshold or maximum shade, the color or shading may be changed to a distinctly unique color or shade designated to represent relatively higher density event counts. For example, a
different color and/or shading (e.g., a light red color instead of a darker red or blue) may be used to visualize relatively lower or higher density event counts. In some embodiments, multiple overlapping points or bubbles, e.g., above a predetermined threshold, within an area of the bubble chart (e.g., corresponding to the same predetermined time period) may be combined into a single bubble of a relatively larger size, thereby providing a relatively easier way to visually distinguish a plurality of event counts occurring around the same time.
In some implementations, the different colors or shading applied to various event count densities may be represented using, for example, a graphical overlay visualized with respect to the bubble chart (or one or more bubbles thereof), as displayed within visualization windows 1040A or 1040B of
different implementations for the bubble chart displayed in visualization windows 1040A or 1040B of
to split or break up the bubble chart into separate categories based on another specified field (e.g., a source type field). For example, splitting CPU usage by source type may cause the color of the bubbles in the chart to change based on the particular source
type with which each bubble and corresponding event is associated.
In some implementations, the user's selection of a field value within the table or the visualization may cause a new GUI window to appear, which displays information related to only the selected field value.
URI and the user clicked the value “/download,” a GUI 1100A may be displayed, in which only data related to the “/download” value is visualized, such as a count of events at different times that have the selected value for the field.
Further, the user may be presented with a set of controls including, for example and without limitation, controls 1110, 1112, 1114, and 1116, as shown in
11A. The visualization shown in
Also, as shown in
Further, any number of additional controls may be provided to the user via each of GUIs 1100A, 1100B, 1100C, and 1100D of
The sortable event counts in this example may correspond to, for example, the numbers shown along the y-axis of the visualized charts and line graphs shown in each of
sorted automatically based on input received from the user, and the plotted data and corresponding line graph may also be updated accordingly.
The examples described above with respect to
visualization system 150 of
If programmable logic is used, such logic may execute on a commercially available processing platform or a special purpose device. One of ordinary skill in the art may appreciate that embodiments of the disclosed subject matter can be practiced with various computer system configurations, including multi-core multiprocessor systems, minicomputers, mainframe computers, computer linked or clustered with distributed functions, as well as pervasive or miniature computers that may be embedded into virtually any device.
For instance, at least one processor device and a memory may be used to implement the above described embodiments. A processor device may be a single processor, a plurality of processors, or combinations thereof. Processor devices may have one or more processor “cores.”
Various embodiments of the present disclosure, as described above in the examples of
how to implement embodiments of the present disclosure using other computer systems and/or computer architectures. Although operations may be described as a sequential process, some of the operations may in fact be performed in parallel, concurrently, and/or in a distributed environment, and with program code stored locally or remotely for access by single or multiprocessor machines. In addition, in some embodiments the order of operations may be rearranged without departing from the spirit of the disclosed subject matter.
As shown in
Computer system 1200 also includes a main memory 1240, for example, random access memory (RAM), and may also include a secondary memory 1230. Secondary memory 1230, e.g., a read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may comprise, for example, a floppy disk drive, a magnetic tape drive, an optical
disk drive, a flash memory, or the like. The removable storage drive in this example reads from and/or writes to a removable storage unit in a well-known manner. The removable storage unit may comprise a floppy disk, magnetic tape, optical disk, etc. which is read by and written to by the removable storage drive. As will be appreciated by persons skilled in the relevant art, such a removable storage unit generally includes a computer usable storage medium having stored therein computer software and/or data.
In alternative implementations, secondary memory 1230 may include other similar means for allowing computer programs or other instructions to be loaded into computer system 1200. Examples of such means may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM, or PROM) and associated socket, and other removable storage units and interfaces, which allow software and data to be transferred from a removable storage unit to computer system 1200.
Computer system 1200 may also include a communications interface (“COM”) 1260. Communications interface 1260 allows software and data to be transferred between computer system 1200 and external devices. Communications interface 1260 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. Software and data transferred via communications interface 1260 may be in the form of signals, which may
be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 1260. These signals may be provided to communications interface 1260 via a communications path of computer system 1200, which may be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link or other communications channels.
The hardware elements, operating systems and programming languages of such equipment are conventional in nature, and it is presumed that those skilled in the art are adequately familiar therewith. Computer system 1200 also may include input and output ports 1250 to connect with input and output devices such as keyboards, mice, touchscreens, monitors, displays, etc. Of course, the various server functions may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load. Alternatively, the servers may be implemented by appropriate programming of one computer hardware platform.
Program aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of executable code and/or associated data that is carried on or embodied in a type of machine readable medium. “Storage” type media include any or all of the tangible memory of the computers, processors or the
like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide non-transitory storage at any time for the software programming. All or portions of the software may at times be
communicated through the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer of the mobile communication network into the computer platform of a server and/or from a server to the mobile device. Thus, another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links, optical links or the like, also may be considered as media bearing the software. As used herein, unless restricted to non-transitory, tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.
While principles of the present disclosure are described herein with reference to illustrative embodiments for particular applications, it should be understood that the disclosure is not limited thereto. Those having ordinary skill in the art and access to the teachings provided herein will recognize additional modifications, applications, embodiments, and substitution of equivalents all fall within the scope of the embodiments described herein. Accordingly, this disclosure is not to be considered as limited by the foregoing description.
The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.
Claims
1. A method comprising:
- creating a set of time stamped, searchable events from a set of raw data, each event in the set of time stamped, searchable events includes a portion of the set of raw data from which the time stamped, searchable event was derived, the set of raw data related to security or performance aspects of one or more information technology systems;
- identifying a set of unique values included in a particular field that is present in one or more time stamped, searchable events in the set of time stamped, searchable events;
- causing display of a plurality of rows, each row corresponding to one unique value among the set of unique values, each row having one or more indicators displayed along a timeline, each indicator among the one or more indicators indicating a number of time stamped, searchable events in the set of time stamped, searchable events within a certain time period that includes the unique value in the particular field, each indicator of the one or more indicators is positioned along the timeline according to the certain time period;
- wherein the method is performed by one or more computing devices.
2. The method of claim 1, wherein the raw data is machine data.
3. The method of claim 1, wherein the time stamped, searchable events are derived at least in part from log files generated by one or more servers.
4. The method of claim 1, wherein each indicator among the one or more indicators is an absolute or relative indication of the number of time stamped, searchable events and is displayed using a color or shade.
5. The method of claim 1, wherein each indicator among the one or more indicators is an absolute or relative indication of the number of time stamped, searchable events and is displayed using a color or shade, the color or shade is applied to each intersection according to a linear scale.
6. The method of claim 1, wherein each indicator among the one or more indicators is an absolute or relative indication of the number of time stamped, searchable events and is displayed using a color or shade, the color or shade is applied to each intersection according to a logarithmic scale.
7. The method of claim 1, wherein each indicator among the one or more indicators is an absolute or relative indication of the number of time stamped, searchable events and is displayed using a color or shade, the color or shade is applied to each intersection according to an exponential scale.
8. The method of claim 1, wherein each indicator among the one or more indicators is an absolute or relative indication of the number of time stamped, searchable events and is displayed using a color or shade, the color or shade is applied to each intersection according to a linear scale, the color or shade is applied to each intersection according to a rank assigned to that intersection based on a corresponding number of events.
9. The method of claim 1, wherein each indicator among the one or more indicators is an absolute or relative indication of the number of time stamped, searchable events and is displayed using a color or shade, the color or shade is applied to each intersection according to a linear scale, the color or shade is applied to each intersection using a scale based on a maximum event count and a minimum event count determined from (i) intersections within a row including the intersection for which the color or shade is being applied, (ii) intersections within a column including the intersection for which the color or shade is being applied, or (iii) all displayed intersections.
10. The method of claim 1, further compromising;
- receiving user input that specifies a time granularity; and
- determining a duration of time covered by each of the plurality of time periods based on the time granularity
11. The method of claim 1, further compromising;
- predicting what a plot of a number of events having a specified value for the particular field would look like for future time periods based on extrapolating from an actual number of events for previous time periods;
- causing display of a graphical representation of a plot based on the predicting.
12. The method of claim 1, further comprising:
- receiving user input indicating a particular time period to be used for sorting the plurality of rows; and
- sorting the plurality of rows, wherein each row is positioned in ascending or descending order based on a number of events corresponding to the intersection of that row with the particular time period.
13. The method of claim 1, further comprising:
- receiving user input selecting an intersection of a row and a time period; and
- causing display of information pertaining to the intersection that includes any of: a corresponding field value, a count value indicating a number of events associated with the intersection, or a time period associated with the intersection.
14. The method of claim 1, further comprising displaying a statistic for each unique value in the set of unique values for the particular field, wherein the statistic for a given unique value includes any combination of: a minimum event count corresponding to intersections in the row corresponding to the given unique value with time periods, a maximum event count corresponding to the intersections, an average of event counts corresponding to the intersections, a total count of events in multiple intersections, or a percentage of the set of time stamped, searchable events that correspond to multiple intersections.
15. The method of claim 1, further comprising:
- reordering the plurality of rows based on a drag and drop gesture received from a user input device.
16. A non-transitory computer readable storage medium, storing instructions that, when executed by one or more processors, cause performance of:
- creating a set of time stamped, searchable events from a set of raw data, each event in the set of time stamped, searchable events includes a portion of the set of raw data from which the time stamped, searchable event was derived, the set of raw data related to security or performance aspects of one or more information technology systems;
- identifying a set of unique values included in a particular field that is present in one or more time stamped, searchable events in the set of time stamped, searchable events;
- causing display of a plurality of rows, each row corresponding to one unique value among the set of unique values, each row having one or more indicators displayed along a timeline, each indicator among the one or more indicators indicating a number of time stamped, searchable events in the set of time stamped, searchable events within a certain time period that includes the unique value in the particular field, each indicator of the one or more indicators is positioned along the timeline according to the certain time period.
17. The non-transitory computer readable storage medium of claim 16, further compromising;
- predicting what a plot of a number of events having a specified value for the particular field would look like for future time periods based on extrapolating from an actual number of events for previous time periods;
- causing display of a graphical representation of a plot based on the predicting.
18. A system comprising:
- a memory having processor-readable instructions stored therein; and
- a processor configured to access the memory and execute the processor-readable instructions, which when executed by the processor, configures the processor to perform a plurality of functions, including functions to: creating a set of time stamped, searchable events from a set of raw data, each event in the set of time stamped, searchable events includes a portion of the set of raw data from which the time stamped, searchable event was derived, the set of raw data related to security or performance aspects of one or more information technology systems; identifying a set of unique values included in a particular field that is present in one or more time stamped, searchable events in the set of time stamped, searchable events; causing display of a plurality of rows, each row corresponding to one unique value among the set of unique values, each row having one or more indicators displayed along a timeline, each indicator among the one or more indicators indicating a number of time stamped, searchable events in the set of time stamped, searchable events within a certain time period that includes the unique value in the particular field, each indicator of the one or more indicators is positioned along the timeline according to the certain time period.
19. The system of claim 18, wherein the processor is further configured to perform functions to:
- predicting what a plot of a number of events having a specified value for the particular field would look like for future time periods based on extrapolating from an actual number of events for previous time periods;
20. The system of claim 18, further comprising displaying a statistic for each unique value in the set of unique values for the particular field, wherein the statistic for a given unique value includes any combination of: a minimum event count corresponding to intersections in the row corresponding to the given unique value with time periods, a maximum event count corresponding to the intersections, an average of event counts corresponding to the intersections, a total count of events in multiple intersections, or a percentage of the set of time stamped, searchable events that correspond to multiple intersections.
Type: Application
Filed: Jul 31, 2016
Publication Date: Feb 2, 2017
Inventor: Philip John Vander Broek (San Francisco, CA)
Application Number: 15/224,651
