Performance evaluation of applications that access external resources
Evaluating application performance includes: obtaining a request, processing the request, including invoking a function that accesses an external resource, determining whether a response time associated with invoking the function exceeds a threshold, and in the event that the response time exceeds the threshold, capturing a runtime attribute associated with the function.
Latest New Relic, Inc. Patents:
- Automation solutions for event logging and debugging on KUBERNETES
- Automation solutions for event logging and debugging on kubernetes
- Automation solutions for event logging and debugging on KUBERNETES
- Automation solutions for event logging and debugging on Kubernetes
- Automation solutions for event logging and debugging on KUBERNETES
As computer software becomes more sophisticated, performance evaluation has also become a more complex task. In some client-server applications, for example, many different calls are made to the server application and it can be difficult and time-consuming to pinpoint performance bottlenecks. One technique often employed by software developers is to manually insert logging code to facilitate performance evaluation. The addition of logging code, however, can negatively impact the overall performance of the system and may only provide limited information on how the bottlenecks occurred.
Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
To process a request, the application will access an external resource (i.e., a resource that is not an integral part of the application), such as database 106, web service 108, network service 110, or other appropriate resource. An external resource is typically installed and executed on a device other than the application server, but may sometimes be installed and executed on the same application server in a different process. The application is programmed to accesses the external resource using a suitable protocol. For example, to access a database such as ORACLE™ or Microsoft SQL SERVER™, Structured Query Language (SQL) calls are made; to access web services such as credit card processing service or message queuing service, HTTP calls are made; and to access network services such as socket services, networking calls such as open_socket are made. These external resource access calls sometimes cause performance bottlenecks, and can be specifically identified and diagnosed using techniques described below.
A performance monitor 110 is configured to monitor the performance of application 104. In some embodiments, the performance monitor is implemented as special instrumentation code (also referred to as probes) that cooperates with the application to detect any slowdown in the application's access to external resources. For example, in some embodiments where the application is implemented using RUBY ON RAILS™ (Rails), the source code of certain external resource access classes is replaced with performance monitoring code that carries out the same function and additionally performs monitoring functions. In a more specific example, SQL database access calls all flow through a class named ActiveRecord::Base, which is replaced with instrumented code that invokes the original code, and additionally collects performance information such as the amount of time spent on accessing the external resource (referred to as the response time) and other runtime attributes. In some embodiments where the application is implemented using JAVA™, the byte code of certain external resource access functions and libraries is replaced with byte code with additional monitoring functions. Other implementations such as binary code replacement may be used as appropriate.
At 206, it is determined whether the response time exceeds a predefined threshold. Depending on whether the response time is a total response time or a remote function call response time, different thresholds may be set. In some embodiments, both types of response time are compared with their respective thresholds. If the response time does not exceed the threshold, the performance of the function's invocation is considered normal and no further process is required. If, however, the response time exceeds the threshold, at 208, further investigation is needed and one or more runtime attributes associated with the invocation of the function are captured. By examining these runtime attributes, the software developer can identify certain causes for the slowdown. For example, in some embodiments, the runtime attributes include a stack trace of the function's stack frame(s), which can be used to pinpoint certain code that invokes the external resource access and causes the slowdown. In some embodiments, the runtime attributes include parameter(s) associated with the invocation of the function. For example, an incorrect parameter can slow down the function call. In some embodiments, the runtime attributes include the return value/payload information of the function, which may indicate that too much data is being passed and therefore slowing down the access. In some embodiments, the runtime attributes include performance statistics, which may indicate that resource constraints or a heavy load on the external resource is causing the slowdown.
At 312, post processing is performed to get more in-depth diagnostic information. Since post processing operations can be expensive to perform, they are sometimes performed asynchronously, in a thread or a process that is separate from the function invocation to more efficiently batch process multiple calls and reduce performance impact on the main application.
The following is a pseudocode example of an instrumented database access function:
Function Instrumented_SQLCall
record start time
execute Original_SQLCall
record finish time
if (finish time−start time)>threshold
-
- save configuration
- get runtime attributes such as stack trace, etc.
At 406, a new database connection is opened using the same database connection information as the original call. At 408, an explain plan is generated for the original database call, over the new database connection. The explain plan is a database feature that describes how the database would execute the database access function. For example, whether indexes are used, which indexes are used, what type of database operation is performed, if temporary tables are used, and how many rows would be accessed. Analysis of the explain plan can be helpful for determining the cause for the slowdown. For example, queries on tables that don't use an index run quickly when the number of rows is small but progressively get slower as the number of rows grows. Knowing that a query is slow, looking at an explain plan and seeing that a large number of rows is accessed and that an index is not being used is enough information for a software developer to quickly see that adding an index will speed up queries.
In some embodiments, post processing further includes invoking a debug version of the function. The debug function is invoked under substantially the same condition, using the same parameters as the invocation of the original function. The debug version of the function will provide extra diagnostic information such as log information, break points, or any other appropriate debugging facilities.
The following is a pseudo code example for a post processing function that operates in a separate thread and diagnoses slow SQL calls:
Function Post_Process_SQL
for each slow SQLCall
-
- get database connection information
- open new database connection
- generate explain plan (SQLCall(new database connection))
The software developer can select a call from the table to view further performance monitoring details.
Performance evaluation of applications that invoke function calls that access external resource has been described. Although database access and using explain plan to identify bottlenecks in the access calls have been discussed extensively for purposes of example, the technique can also be used to diagnose slowdowns in other types of external resource access.
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.
Claims
1. A system for evaluating application performance, comprising: a processor configured to:
- receive a plurality of external resource access calls that access a plurality of external resources, the plurality of external resources comprising at least a database resource and a web service resource;
- process the plurality of external resource access calls, including for each external resource access call: invoking an instrumented function that includes an original function that accesses an external resource and monitoring code to monitor the performance of invoking the original function; determine a response time associated with said each external resource access call; determine whether the response time associated with said each external resource access call exceeds a response time threshold; use a result of the determination as a basis for whether to further process the external resource access call, including: in the event that the response time exceeds the response time threshold: capture a runtime attribute associated with the external resource access call that exceeds the response time threshold; and mark the external resource access call that exceeds the response time threshold for post processing to obtain diagnostic information; in the event that the response time does not exceed the response time threshold: omit capturing the runtime attribute and omit marking with respect to the external resource access call that does not exceed the response time threshold; and
- perform post processing on the each marked external resource access call exceeding the response time threshold using at least in part the captured runtime attribute to obtain additional performance information, wherein performing post processing includes invoking a post processing function corresponding to the marked external resource access call using the corresponding captured runtime attribute, wherein the post processing function includes additional diagnostic code used to analyze the original function that accesses an external resource, and wherein the post processing function is different from the instrumented function; and
- a memory coupled to the processor, configured to provide the processor with instructions.
2. The system of claim 1, wherein the post processing is performed asynchronously.
3. The system of claim 1, wherein the original function is a database access function.
4. The system of claim 3, and the post processing comprises generating an explain plan of the database access function.
5. The system of claim 4, wherein the runtime attribute includes database connection information associated with the invocation of the database access function, and generating the explain plan includes opening another database connection using the database connection information.
6. The system of claim 5, wherein the database connection information includes user credential information, database host information and port information.
7. The system of claim 1, wherein the runtime attribute includes a stack trace associated with the external resource access call that exceeds the response time threshold.
8. The system of claim 1, wherein the runtime attribute includes a parameter that is used to invoke the external resource access call that exceeds the response time threshold.
9. The system of claim 1, wherein the runtime attribute includes a return value of the external resource access call that exceeds the response time threshold.
10. The system of claim 1, wherein the runtime attribute includes performance statistics.
11. The system of claim 1, wherein the external resource includes a networking resource.
12. A method for evaluating application performance, comprising:
- receiving a plurality of external resource access calls that access a plurality of external resources, the plurality of external resources comprising at least a database resource and a web service resource;
- processing the plurality of external resource access calls, including for each external resource access call: invoking an instrumented function, that includes an original function that accesses an external resource and monitoring code to monitor the performance of invoking the original function; determine a response time associated with said each external resource access call; determining whether the response time associated with said each external resource access call exceeds a response time threshold; using a result of the determination as a basis for whether to further process the external resource access call, including: in the event that the response time exceeds the response time threshold, further comprising: capturing a runtime attribute associated with the external resource access call that exceeds the response time threshold; and marking the external resource access call that exceeds the response time threshold for post processing to obtain diagnostic information; and in the event that the response time does not exceed the response time threshold: omitting capturing the runtime attribute and omitting marking with respect to the external resource access call that does not exceed the response time threshold; and
- performing post processing on the each marked external resource access call exceeding the response time threshold using at least in part the captured runtime attribute to obtain additional performance information, wherein performing post processing includes invoking a post processing function corresponding to the marked external resource access call using the corresponding captured runtime attribute, wherein the post processing function includes additional diagnostic code used to analyze the original function that accesses an external resource, and wherein the post processing function is different from the instrumented function.
13. The method of claim 12, wherein the original function is a database access function.
14. A computer program product for evaluating application performance, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
- receiving a plurality of external resource access calls that access a plurality of external resources, the plurality of external resource types comprising at least a database resource and a web service resource;
- processing plurality of external resource access calls, including for each external resource access call: invoking an instrumented function that includes an original function that accesses an external resource and monitoring code to monitor the performance of invoking the original function; determine a response time associated with said each external resource access call; determining whether the response time associated with said each external resource access call exceeds a response time threshold; using a result of the determination as a basis for whether to further process the external resource access call, including: in the event that the response time exceeds the response time threshold, further comprising: capturing a runtime attribute associated with the external resource access call that exceeds the response time threshold; and marking the external resource access call that exceeds the response time threshold for post processing to obtain diagnostic information; and in the event that the response time does not exceed the response time threshold: omitting capturing the runtime attribute and omitting marking with respect to the external resource access call that does not exceed the response time threshold; and
- performing post processing on the each marked external resource access call exceeding the response time threshold using at least in part the captured runtime attribute to obtain additional performance information, wherein performing post processing includes invoking a post processing function corresponding to the marked external resource access call using the corresponding captured runtime attribute, wherein the post processing function includes additional diagnostic code used to analyze the original function that accesses an external resource, and wherein the post processing function is different from the instrumented function.
15. The computer program product of claim 14, wherein the original function is a database access function.
16. The system of claim 1, wherein different response time thresholds are set in response to the response time corresponding to the total amount of time spent processing the external resource access call or the time for receiving the response to the external resource access call.
17. The system of claim 1, wherein in the event that the response time exceeds the response time threshold, connection information relating to the external resource access call is recorded, the connection information including a user credential associated with the external resource access call, a database host associated with the external resource access call, a port associated with the external resource access call, or any combination thereof.
6249783 | June 19, 2001 | Crone et al. |
6513154 | January 28, 2003 | Porterfield |
6816874 | November 9, 2004 | Cotner et al. |
7702642 | April 20, 2010 | Wolfman et al. |
20030066053 | April 3, 2003 | Al-Azzawe |
20030182276 | September 25, 2003 | Bossman et al. |
20040143795 | July 22, 2004 | Matsuishi |
20050138111 | June 23, 2005 | Aton et al. |
20050283457 | December 22, 2005 | Sonkin et al. |
20060085375 | April 20, 2006 | Egan et al. |
20060212417 | September 21, 2006 | Barsness et al. |
20070174335 | July 26, 2007 | Konig et al. |
20070266045 | November 15, 2007 | Bansal et al. |
20080215536 | September 4, 2008 | Day et al. |
Type: Grant
Filed: Sep 10, 2009
Date of Patent: Apr 3, 2018
Assignee: New Relic, Inc. (San Francisco, CA)
Inventors: Lewis Karl Cirne (Palo Alto, CA), James Robert Gochee (Portland, OR), Saxon Michael D'Aubin (Portland, OR)
Primary Examiner: Vivek Srivastava
Assistant Examiner: Hien Doan
Application Number: 12/584,833
International Classification: G06F 15/173 (20060101); H04L 29/08 (20060101); H04L 12/26 (20060101); H04L 29/06 (20060101);