Systems and methods for tracking processing unit usage
A method and system for monitoring the CPU time consumed by a software agent operating in a computer system is disclosed. A resource tracking process is executed on the system. When an operating agent is detected, an agent lifetime timer is initialized. Then, CPU resources for the agent are identified and stored. Checks are made at predetermined intervals to determine if the agent is still alive. When the agent terminates, a measurement is made of the CPU time utilized by the agent. The measurement is then stored in memory.
Latest IBM Patents:
- EFFICIENT RANDOM MASKING OF VALUES WHILE MAINTAINING THEIR SIGN UNDER FULLY HOMOMORPHIC ENCRYPTION (FHE)
- MONITORING TRANSFORMER CONDITIONS IN A POWER DISTRIBUTION SYSTEM
- FUSED MULTIPLY-ADD LOGIC TO PROCESS INPUT OPERANDS INCLUDING FLOATING-POINT VALUES AND INTEGER VALUES
- Thermally activated retractable EMC protection
- Natural language to structured query generation via paraphrasing
Management and workflow collaboration software systems are used by organizations to maintain the efficiency of workforces. Among other things, these systems, herein referred to as enterprise systems, allow employees to communicate, obtain information, requisition products and services, generate documents, and perform online training. Management systems may also connect portions of an organization that are separated geographically. As a result, management systems are often spread across multiple servers coupled to the network. The distributed nature of management systems along with the requirement that they perform numerous operations simultaneously makes them very large and complex.
Adding to the complexity of management systems is the fact that many of these systems have evolved over time. This evolution has largely been driven by customer needs and has resulted in rather basic management systems evolving into the large, complex ones predominant today. In addition, users must be able to write their own applications to extract maximum utility from management systems. The combination of old/new software-executable instructions and customer developed software instructions may produce operating environments that are difficult to troubleshoot. For example, an enterprise system may be executing hundreds or thousands of software agents as a matter of normal operation. A software agent as used herein refers to any self-contained potentially adaptive software application and as such, is an executable sequence of instructions. Software agents can be short lived, i.e. having lifetimes of less than a second, or can be long-lived having lifetimes measured in hours or days. Furthermore, software agents may need to operate according to a schedule. If a particular software agent is not operating at the appropriate time, it may be due to either a problem with the scheduled agent, a problem with an agent currently running, or a problem with software processes that schedule agents. Identifying the exact cause of late agents using the prior art is thus problematic. Software agents may also consume system resources while operating. Examples of resources consumed by agents are, but not limited to, system memory, CPU bandwidth, disk input/output operations, database opens, document creation, and network traffic.
Prior art systems and methods for monitoring agent activity may not easily identify problematic agents because there is no apriori mechanism for identifying when agents may become problematic. Furthermore, prior art tools may not isolate problematic agents.
There exists a need for systems and methods for identifying agents that may become problematic before they actually cause a problem. Furthermore, there exists a need for monitoring system resources consumed by software agents. In addition, there is a need for identifying and rank ordering agents causing problems so that system resources can be adaptively managed.
SUMMARY OF THE INVENTIONThe preferred embodiments of the present invention are directed at measuring and monitoring processing unit resources within computing environments. These environments include, without limitation, Agent Manager and hypertext transfer protocols (HTTP). In accordance with an aspect of the invention, a method for monitoring system processor time of a software agent operating in a computer system is provided. The method comprises identifying the agent by associating an agent identifier with it. Then, an operating interval associated with the agent is determined and monitored using an agent lifetime timer. Next, the operating interval and agent identifier are stored in a computer-readable memory.
In accordance with another aspect of the invention, a method for monitoring system processor usage, for example, as time by an agent, having an agent lifetime, which is associated with a thread, having a thread lifetime, is provided. In this method, an agent identifier is associated with the agent. An agent lifetime timer is initiated for monitoring the agent lifetime. System processor resource allocations for the agent are then determined that are used to define a footprint which can be defined as the maximum memory allocation for the agent. The footprint for the agent includes an amount of system processor resources utilized by the agent thread during its lifetime and an amount of system processor resources utilized by all the threads during the agent lifetime. In a preferred embodiment, the method then includes the step of associating the footprint with the agent identifier, storing the footprint and agent identifier in a computer-readable memory, comparing the agent's footprint to footprints associated with other software agents operating in the system, ranking the footprint of the agent against the other software agents' footprints, and displaying the footprints exceeding a predefined threshold.
In accordance with yet another aspect of the invention, a computer program product having machine-readable instructions disposed thereon for instructing a processor to perform a method for monitoring system processor time for a software agent operating in a computer system is provided. The computer program product includes instructions for initiating an agent lifetime timer for monitoring an operating interval associated with the agent, instructions for determining system processor resource allocations associated with the agent, instructions for storing the operating interval and resource allocations associated with the agent, and instructions for notifying a system operator about operating interval and resource allocations.
In accordance with still another aspect of the invention, an apparatus for tracking system processor usage time by a software agent operating in a computer system is provided. The apparatus includes a processor having executable instructions for identifying the agent by associating an agent identifier therewith, executable instructions for initiating an agent lifetime timer for monitoring an operating interval of the agent, executable instructions for determining the operating interval using the lifetime timer and executable instructions for storing the operating interval and the agent identifier in a computer-readable memory having a hash table associated therewith.
In accordance with still another aspect of the invention, a method for tracking system processor time for a target agent associated with a Hyper Text Transfer Protocol (HTTP) process running a plurality of threads in a computer is provided. In a preferred embodiment, the method includes the target agent is operating with at least one of the threads. A computer-readable hash table is created in a memory associated with the computer system. An agent tracking function is initiated in machine-executable sequence of instructions on the system. Next, members of the plurality of threads are identified by associating a thread identifier with each member of the plurality of threads. Those of the plurality of identified threads having the target agent operating therewith are identified. Then, the amount of system processor time utilized by the identified threads is determined and stored in the hash table.
In preferred embodiments, processing unit resource consumption can be tracked on two levels. First, the overall CPU consumption is tracked. When the CPU utilization for the process becomes high, an individual agent's CPU consumption is tracked. Tracking the overall usage determines when the usage is high, while the second determines which agent(s) are causing the problem.
The foregoing and other features and advantages of the system and method for monitoring CPU usage associated with software agents will be apparent from the following more particular description of preferred embodiments of the system and method as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views.
The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
Management systems such as, for example, enterprise systems handle, without limitation, electronic mail (email) receipt and distribution, access and management to intranet and Internet web sites, management of internal databases, multimedia conferencing, and online calendaring and appointment management. In addition, enterprise systems handle many other functions necessary to assist large and small organizations in operating efficiently. Management enterprise systems rely heavily on the use of software agents for managing the workflow. Agents can be generated by the enterprise system itself or by users and can typically include two types, scheduled or event driven agents. Scheduled agents are those having a defined start time. In addition, scheduled agents may have a predetermined stop time or the stop time can be dictated by an event such as, for example, but not limited to, receipt of an interrupt or the start of another agent. In contrast, event driven agents are initiated upon the occurrence of a particular event rather than at a certain time. Examples of event driven agents are, but not limited to, document creation agents, document closing agents, database opening agents, user logon processing agents, email agents activated upon receipt of a message, and web based agents.
For web based event driven agents, the most common class may be those associated with hypertext transport protocol (HTTP) processes. An HTTP process has a plurality of agent threads associated therewith, and in a preferred embodiment, the default number of multiple threads in an HTTP process, without limitation, is forty. Each agent thread has a unique identifier, for example an identification number, referred to as an HTTP thread ID. An HTTP environment can be multi-threaded with each HTTP agent thread running a single agent thus making it possible for a given agent to be running on multiple HTTP threads simultaneously. For example, in a preferred embodiment, forty copies of the same agent can be executed, or run, at the same time. An HTTP thread ID number is associated with the instance of an agent that is executing on the respective thread, thus making it possible to identify with which copy of an agent a given set of activities are associated. For example, identifying instances of agents using HTTP thread ID's makes it possible to determine when a specific instance of an agent started and/or ended. In the above example, a reference to the instance of the agent, such as an agent ID, consists of the agent's name along with the HTTP thread ID associated with the agent. In addition, a database entry or other parameter may be associated with the agent name and HTTP thread ID to uniquely identify and to further provide details about the agent. As used herein, a thread is a program path of execution, and multi-threaded means that multiple lines of a single program can be executed simultaneously. When agents are running on multiple threads, the operating system may treat each agent thread as a separate and distinct process.
When numerous agents are running simultaneously, problems can occur when one or more agents fail to operate as intended. Problematic agents can be associated with the management enterprise system itself or with user generated agents. Agents in the management system computing environment may become problematic when a new software release is implemented because of, without limitation, incompatibilities with system interfaces or user generated agents and an inability to identify a particular agent when attempting to diagnose system abnormalities. User generated agents may be problematic due to incompatibilities with operating system components, a developer's failure to understand the functionality of developed agents, security misconfigurations associated with developed agents, faulty parameters associated with developed agents, etc.
Agent failures can take many forms, however, certain failures tend to be more prevalent. Examples of common agent failures are, but are not limited to, mismanagement of system memory by agents, excessive central processing unit (CPU) usage, spawning of too many sub processes or agent threads, running longer than expected, failing to execute at the proper time, entering an infinite loop, and mis-applied/misused security settings. The preferred embodiments of the present invention address agents that consume too much CPU time.
The manager process 104 oversees management of software agent activity within a management application 100. In particular, the manager process 104 operates in connection with a plurality of executive processes 108 and a run queue 106. When an agent is scheduled to run, the manager process 104 places it in run queue 106. When an executive process 108 is free, the manager process 104 instructs the executive process 108 to begin processing the scheduled agent. When the executive process 108 begins operating on an agent, the agent becomes a running agent for the duration of time it is associated with executive process 108. When an agent has finished executing, the executive process 108 releases the agent and becomes available for running a new agent.
The management computing system 102 may also include an email task 110. The email task 110 is responsible for handling all incoming and outgoing email. Agents associated with email task 110 may be scheduled agents or event driven agents. Examples of scheduled agents operating in conjunction with email task 110 may be an email checking agent. When launched, a router operating in the software delivers mail and sends an event to an agent manager indicating that new mail has been delivered to a database. If an active agent resides in that database, it is invoked.
The HTTP task 112 may run on management computing system 102. The HTTP task 112 is responsible for processing all HTTP related traffic within the management application 100. The HTTP task 112 may start all threads upon its activation, or startup, in response to HTTP requests received from network 120. HTTP task 112 runs event driven agents which consist of threads. The HTTP task 112 may further start a new agent on any running thread. As previously discussed herein, an HTTP agent in an HTTP process can run on more than one thread. In such an embodiment, knowing only a problematic agent's name is not sufficient for identifying it. Information about the thread running the problematic agent is required to positively identify the agent.
The Web conferencing task 114 is responsible for handling all audio, video and text data associated with collaborative conferencing. The Web conferencing task 114 may interact with the email task 110 and/or the HTTP task 112 when providing web conferencing capabilities to an organization. As a result, the web conferencing task 114 may utilize scheduled and event driven agents.
The customer developed task 116 may include software developed by a user of the management application 100. User developed software may take many forms and may utilize one or more scheduled or event driven agents. An example of user developed task 116 may be an order processing agent. The order processing agent may have an event driven portion that executes when a user opens an online ordering capability. Once the order is entered, a scheduled agent may operate on a predetermined cycle seeking management approvals for the order before sending it to a vendor for fulfillment.
The system manager UI 118 may include a computer display for presenting information to a system administrator. For example, the manager process 104 may detect an improperly running agent and cause a popup window to appear on the system manager UI 118. After viewing the error report, the system administrator can take corrective action.
The network 120 connects management computing system 102 to remote sites 122A and 122B. In many large corporations, there may be a headquarters and a plurality of field offices. In
When executive process 108 begins processing an agent, the agent is referred to as an active, or running, agent. While active, an agent is deemed to be alive. When the agent has finished executing, either normally or via manual termination, it is deemed to be dead, or expired. If the agent is later placed in the holding queue it is referred to as a scheduled agent until it is once again processed by an executive process 108.
In preferred embodiments, the information about individual agents is collected in a sorted linked list for each thread/process. Information about each agent includes an agent name/database name and CPU time used by the corresponding agent. Periodically the lists generated on different threads/processes are aggregated by an embedded software probe manager. The embedded probe manager caches the collected data for each configuration, filtering or removing agents that are inappropriate. Information about CPU usage is also added to an agent log and is made available to an end user.
Threads created by the running agent are detected per step 414. Next, a thread ID is added to a group list per step 418. Then an entry for the thread is created in the hash table per step 419. After step 419, the method is shown in
After step 425, a check is made to determine if the agent is expired per step 426. If the agent is expired, a total CPU time for the agent is computed per step 428. In contrast, if the agent is not expired, the method iterates back to the input of step 414 (
After step 428, CPU time for the agent is stored in the hash table per step 430. Filtering criteria is then defined per step 432. The method of
Display device 512 may be a cathode ray tube (CRT), liquid crystal display (LCD), or the like, for displaying information to a user. Keyboard 514 and cursor control 516 allow the user to interact with computer 500. Cursor control 516 may be, for example, a mouse. In an alternative configuration, keyboard 514 and cursor control 516 can be replaced with a microphone and voice recognition means to enable the user to interact with computer 500.
Communication interface 518 enables computer 500 to communicate with other devices/systems via any communications medium. For example, communication interface 518 may be a modem, an Ethernet interface to an IP network, or a printer interface. Alternatively, communication interface 518 can be any other interface that enables communication between computer 500 and other devices or systems.
By way of example, a computer 500 consistent with the present invention provides a management computing system 102 with the ability to monitor scheduled and running agents on remote sites 122A and 122B via network 106. Computer 500 performs operations necessary to complete desired actions, such as computing a total CPU time for an agent and storing it in a hash tube as shown in
Thread presence 604 may be a flag denoting the existence of threads other than the main thread associated with an agent. A value for thread presence may reference another data structure such as a thread list data structure for storing additional information.
Start time field 606 and end time field 608 may contain the start time and completion time for a given agent, respectively. Status field 612 may contain information about the current status of an agent. For example, status field 612 may contain a flag where the value of the flag indicates whether the agent is currently alive or expired.
Alternatively, status field 612 may contain color codes such as red for an agent greatly exceeding an allocated system resource, such as a memory budget, yellow for an agent that is beginning to exceed a memory budget, and green for agents operating within a memory budget.
Severity field 614 may contain a text field providing possible reasons why an agent is exceeding a memory or CPU time budget. For example, severity field 614 may indicate to a system administrator that a particular agent is currently executing in an infinite loop and that manually terminating the agent is the only way to release CPU resources being utilized. Resolution field 616 may contain instructions for instructing a system operator on how to terminate a problematic agent.
Additional fields may be added to the data structure of
Agent identification field 622 may identify an agent having threads identified in thread ID field 624 associated therewith. Thread ID field 624 may contain information uniquely identifying each thread associated with a given agent. Threads included in thread ID list may make up a thread list or a thread group list.
Thread data structure 621 is exemplary in nature and can be modified depending on the needs of the management application 100. Peak usage field 626 may indicate the amount of memory consumed by particular threads during their respective lifetimes.
The displays shown in
While exemplary embodiments are provided, other embodiments are possible in light of the specification. Therefore, unless otherwise specified, the illustrated embodiments can be understood as providing exemplary features of varying detail of certain embodiments, and therefore, unless otherwise specified, features, components, modules, and/or aspects of the illustrations can be otherwise combined, separated, interchanged, and/or rearranged without departing from the disclosed systems and methods. Additionally, the shapes and sizes of components are also exemplary and unless otherwise specified, can be altered without affecting the disclosed systems and methods and without departing from the spirit and scope of the invention.
The foregoing description of exemplary embodiments of the present invention provides illustration and description, but is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. For example, while the above description focused on monitoring CPU time and memory utilization associated with operating agents and their associated threads, the invention can be modified to also monitor input and output data associated with agents such as that associated with disk input/output data. And, while a series of steps have been described with regard to
No element, step, or instruction used in the description of the present application should be construed as critical or essential to the invention unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items or elements. Where only one item is intended, the term “one” or similar language is used.
Many alternative embodiments of the present invention are possible in light of the teachings hereinabove. For example, in a first alternative embodiment, values displayed in agent ID 706, of display 700A, may be comprised of links that when clicked on open a new window containing detailed information about the agent. The detailed information can contain parameters associated with the agent, actual source code associated with the agent, security settings and permissions associated with the agent, etc. In addition, clicking on the link may further open a source code debugger to let the user begin debugging the source code associated with a particular problematic agent.
In a second alternative embodiment, the methods of
The scope of the invention is defined by the claims and equivalents thereof hereinbelow.
The claims should not be read as limited to the described order or elements unless stated to that effect. Therefore, all embodiments that come within the scope and spirit of the following claims and equivalents thereto are claimed as the invention.
Claims
1. A method for monitoring system processor usage time by a software agent operating in a computer system, said method comprising the steps of:
- identifying said agent by associating an agent identifier therewith;
- initiating an agent lifetime timer for monitoring an operating interval for said agent;
- determining said operating interval using said lifetime timer; and
- storing said operating interval and said agent identifier in a computer-readable memory.
2. The method of claim 1, wherein said computer-readable memory includes a hash table.
3. The method of claim 1 wherein determining said operating interval further comprises identifying a start time and a completion time for said agent.
4. The method of claim 3 wherein determining said operating interval further comprises computing an elapsed time as the difference between said starting time and said completion time for said agent.
5. The method of claim 1 further comprising:
- associating said operating interval and said agent identifier with other operating intervals and agent identifiers associated with a plurality of other software agents operating in said system.
6. The method of claim 5 further comprising:
- filtering said agent and said plurality of other agents according to predefined filtering criteria to produce a filtered set.
7. The method of claim 6 further comprising:
- rank ordering said filtered set.
8. The method of claim 7 further comprising:
- making said filtered set available to a display device.
9. The method of claim 6 further comprising:
- determining a corrective measure for at least one member of said filtered set.
10. The method of claim 9 further comprising:
- displaying said corrective measure on a display device.
11. The method of claim 9, wherein said corrective measure is implemented by said system.
12. A computer readable medium having store instructions for causing a processing unit to execute the steps of the method of claim 1.
13. A method for monitoring system processor time usage by a software agent having a thread associated therewith, said thread having a thread lifetime and said agent having an agent lifetime, said method comprising the steps of:
- associating an agent identifier with said agent;
- initiating an agent lifetime timer for monitoring said agent lifetime;
- determining system processor resource allocations associated with said agent, said resource allocations defining a footprint for said agent comprising: an amount of system processor resources utilized by said thread during said thread lifetime; and an amount of system processor resources utilized by said agent during said agent lifetime;
- associating said footprint with said agent identifier;
- storing said footprint and said agent identifier in a computer-readable memory;
- comparing said footprint of said agent to a plurality of footprints associated with a like plurality of other software agents;
- ranking said footprint of said agent against said plurality of footprints; and
- displaying those of said agent footprint and said plurality of footprints exceeding a predefined threshold.
13. The method of claim 12 further comprising:
- establishing a system processor resources configuration threshold defining a maximum amount of system processor resources to be utilized by each of said software agent and said plurality of other software agents.
14. The method of claim 13, further comprising:
- running a collection probe to determine if a total amount of consumed system processor resources exceeds said configuration threshold; and
- performing said initiating step when said total amount of consumed system processor resources exceeds said configuration threshold.
15. A computer program product having machine-readable instructions disposed
- thereon for instructing a processor to perform a method for monitoring system processor time for a software agent operating in a computer system, said computer program product comprising:
- instructions for initiating an agent lifetime timer for monitoring an operating interval associated with said agent;
- instructions for determining system processor resource allocations associated with said agent;
- instructions for storing said operating interval and said resource allocations associated with said agent; and
- instructions for notifying a system operator about said operating interval and said resource allocations.
16. The method of claim 15 further comprising:
- instructions for associating a software agent identifier with said agent, said identifier for facilitating tracking said system processor time associated with said agent.
17. The method of claim 16 further comprising:
- instructions for associating said software agent identifier with said operating interval and said resource allocations prior to storing said operating interval and said resource allocations associated with said agent.
18. An apparatus for tracking system processor time of a software agent operating in a computer system comprising:
- means for identifying said agent by associating an agent identifier therewith;
- means for initiating an agent lifetime timer for monitoring an operating interval of said agent;
- means for determining said operating interval using said lifetime timer; and
- means for storing said operating interval and said agent identifier in a computer-readable memory having a hash table associated therewith.
19. A method for tracking system processor time for a target agent operatively associated with a hypertext transport protocol process operating on a computer system and running a plurality of threads, said target agent further operating with at least one of said plurality of threads, said method comprising:
- creating a computer-readable hash table in a memory operatively associated with said computer system;
- initiating an agent tracking function in machine-executable code in said computer system;
- identifying members of said plurality of threads by associating a thread identifier with each member of said plurality of threads producing a like plurality of identified threads;
- identifying those of said plurality of identified threads having said target agent operating therewith producing an identified thread set;
- determining an amount of said system processor time utilized by said identified thread set; and
- storing said system processor time for said identified thread set in said hash table, thereby tracking said system processor time for said target agent.
20. The method of claim 19 further comprising:
- computing statistics for said identified thread set.
21. The method of claim 19 further comprising:
- rank ordering those of said plurality of identified threads having said target agent operating therewith.
22. The method of claim 21 further comprising:
- providing said identified set to a display device.
Type: Application
Filed: Apr 14, 2004
Publication Date: Oct 20, 2005
Applicant: International Business Machines Corporation (Armonk, NY)
Inventor: A. Kadashevich (Tyngsboro, MA)
Application Number: 10/824,065