PERFORMANCE REGRESSION FRAMEWORK
A performance regression test that monitors a resource based on a metric measured for each of a plurality of builds of a test instance is run. Measurement data of the metric triggered by an operation is logged in an operation log for each of a plurality of builds. The operation is logged in response to an incoming request for processing associated with the performance regression test for monitoring the resource. Based on the operation log and for each of the plurality of builds, the measurement data of the metric triggered by the operation associated with the performance regression test for monitoring the resource is collected. The collected measurement data of the metric for each of the plurality of builds is presented.
The present application claims priority to U.S. patent application Ser. No. 16/123,776, filed on Sep. 6, 2018, the contents of which are entirely incorporated herein by reference, as if fully set forth in this application.
TECHNICAL FIELDEmbodiments described herein generally relate to software performance regression testing, and more particularly, to performing regression testing based on available operation-level metrics on incremental software builds before conducting full performance software testing at the end of the cycle.
BACKGROUNDA variety of enterprise and/or information technology (IT) related software applications may be utilized to support various functions of an enterprise such as Finance, Human Resource (HR), IT, Legal, Marketing, Sales, and the like. The software applications may be deployed on an instance platform on a server and accessed as needed over a network such as a Local Area Network (LAN) or the Internet. The server may be a local enterprise server as part of a self-hosted system or a remote server located in the Cloud as part of a cloud computing system.
Cloud computing relates to sharing of computing resources that are generally accessed via the Internet. In particular, cloud computing infrastructure allows users to access a shared pool of computing resources, such as servers, storage devices, networks, applications, and/or other computing-based services. By doing so, users, such as individuals and/or enterprises, are able to access computing resources on demand that are located at remote locations in order to perform a variety of computing functions that include storing and/or processing computing data. For enterprise and other organization users, cloud computing provides flexibility in accessing cloud computing resources without accruing up-front costs, such as purchasing network equipment and investing time in establishing a private network infrastructure. Instead, by utilizing cloud computing resources, users are able redirect their resources to focus on core enterprise functions.
In today's communication networks, examples of cloud computing services a user may utilize include software as a service (SaaS) and platform as a service (PaaS) technologies. SaaS is a delivery model that provides software as a service rather than an end product. Instead of utilizing local network or individual software installations, software is typically licensed on a subscription basis, hosted on a remote machine, and accessed as needed. For example, users are generally able to access a variety of enterprise and/or information technology (IT) related software via a web browser. PaaS acts as an extension of SaaS that goes beyond providing software services by offering customizability and expandability features to meet a user's needs. For example, PaaS can provide a cloud-based developmental platform for users to develop, modify, and/or customize applications and/or automate enterprise operations without maintaining network infrastructure and/or allocating computing resources normally associated with these functions.
SUMMARYThe following presents a simplified summary of the disclosed subject matter in order to provide a basic understanding of some aspects of the subject matter disclosed herein. This summary is not an exhaustive overview of the technology disclosed herein. It is not intended to identify key or critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is discussed later.
In one embodiment a method includes: running a performance regression test that monitors a resource based on a metric measured for each of a plurality of builds of a test instance; logging measurement data of the metric triggered by an operation in an operation log for each of a plurality of builds, wherein the operation is logged in response to an incoming request for processing associated with the performance regression test for monitoring the resource; collecting, based on the operation log and for each of the plurality of builds, the measurement data of the metric triggered by the operation associated with the performance regression test for monitoring the resource; and presenting the collected measurement data of the metric for each of the plurality of builds.
In another embodiment, the method may be embodied in computer executable program code and stored in a non-transitory storage device. In yet another embodiment, the method may be implemented on a (cloud-based or self-hosted) computer system.
For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments disclosed herein. It will be apparent, however, to one skilled in the art that the disclosed embodiments may be practiced without these specific details. In other embodiments, structure and devices are shown in block diagram form in order to avoid obscuring the disclosed embodiments. Moreover, the language used in this disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resorting to the claims being necessary to determine such inventive subject matter. Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment.
The terms “a,” “an,” and “the” are not intended to refer to a singular entity unless explicitly so defined, but include the general class of which a specific example may be used for illustration. The use of the terms “a” or “an” may therefore mean any number that is at least one, including “one,” “one or more,” “at least one,” and “one or more than one.” The term “or” means any of the alternatives and any combination of the alternatives, including all of the alternatives, unless the alternatives are explicitly indicated as mutually exclusive. The phrase “at least one of” when combined with a list of items, means a single item from the list or any combination of items in the list. The phrase does not require all of the listed items unless explicitly so defined.
The term “computing system” is generally taken to refer to at least one electronic computing device that includes, but is not limited to a single computer, virtual machine, virtual container, host, server, laptop, and/or mobile device or to a plurality of electronic computing devices working together to perform the function described as being performed on or by the computing system.
As used herein, the term “medium” or “memory” refers to one or more non-transitory physical media that together store the contents described as being stored thereon. Embodiments may include non-volatile secondary storage, read-only memory (ROM), and/or random-access memory (RAM).
As used herein, the term “application” refers to one or more computing modules, programs, processes, workloads, threads and/or a set of computing instructions executed by a computing system or one or more hardware processors. Example embodiments of an application include software modules, software objects, software instances and/or other types of executable code.
This disclosure pertains to implementing automated tests on a system that helps find any performance related regressions on recurring builds of an instance (e.g., production-ready platform on which one or more enterprise-level software applications are deployed) during an ongoing development process, without waiting for “full-blown” performance regression tests at the end of the software development lifecycle. Techniques disclosed herein look to utilize granular operation-level logs to monitor a resource (e.g., application programming interface (APIs), uniform resource locator (URLs), and the like) based on a minimal set of metrics already captured, so as to detect deviations between recurring builds early in the software development lifecycle.
During ongoing software development of the instance, a version control system may provide a centralized source code repository for collaborative software development so that multiple check-outs and check-ins (e.g., commits) of the source code of the instance may be made simultaneously on a same updated version of the source code. A continuous integration (CI) server may build an installable distribution of the source code as an artifact or bundled artifact (e.g., test instance) on a recurring basis (e.g., every night, every predetermined number of hours, every week, every month, every predetermined number of source code commits, based on user operation, and the like) on a dedicated machine. One or more performance regression tests (e.g., one or more test projects, or test suites) may be run on top of each recurring build of the test instance to detect any performance regressions (e.g., performance degradations, performance deviations) over time. In one embodiment, multiple bundled artifacts may be created for respective source code commits to identify the specific source code check-in or line of code that caused the performance deviation.
Each performance regression test may be configured to detect the performance regression or deviation of specific one or more monitored resources (e.g., application programming interfaces (APIs), uniform resource locators (URLs), and the like) deployed on or associated with the test instance based on one or more existing metrics already measured and captured by the test instance and stored in an operation log of a plurality of operations. A request (e.g., hypertext transfer protocol (HTTP) request) to the test instance for accessing the monitored resource (e.g., a call to the API, request to open a form corresponding to a URL, and the like) may result in creation of one or more operations (e.g., transactions, activities, and the like) in the operation log. The existing metrics already captured and measured by test instance may include processing time, central processing unit (CPU) percentage, response time, a network time, structured query language (SQL) time, SQL count, enterprise rule time, enterprise rule count, client response time, client network time, total page load time, number of requests, browser time, client script time, user interface (UI) policy time. A test module may collect, from the operation log for each build of the test instance, measurement data (e.g., values) for a particular metric (e.g., SQL time) from the plurality of metrics associated with the monitored resource (e.g., “create new incident” page of the test instance) of the performance regression test. The test module may then output, for example, a graph or trend line, indicating the value of the metric of the monitored resource over time to appropriate recipients. Based on the measurement data, the test module may also automatically detect a performance degradation event (e.g., deviation in value of measurement data for a specific build or a specific source code commit) for the monitored resource. The system may thus efficiently narrow down to a specific build or a specific source code commit that caused the deviation. Corrective action (e.g., debugging) may thus be immediately taken on source code corresponding to recurring builds to maintain or improve performance during the software development lifecycle, without waiting for the dedicated full-blown performance testing after “code complete”, at the end of the software development life cycle.
Client computers 115 (i.e., 115A, 115B, and 115C), which may take the form of any smartphone, gaming system, tablet, computer, set top box, entertainment device/system, television, telephone, communications device, or intelligent machine, including embedded systems, may also be coupled to networks 105, and/or data server computers 110. In some embodiments, network system 100 may also include network printers such as printer 120 and storage systems such as 125, which may be used to store user session data or other data that are referenced herein. To facilitate communication between different network devices (e.g., data servers 110, end-user computers 115, network printer 120, and storage system 125), at least one gateway or router 130 may be optionally coupled there between. Furthermore, in order to facilitate such communication, each device employing the network may comprise a network adapter circuit and related software. For example, if an Ethernet network is desired for communication, each participating device must have an Ethernet adapter or embedded Ethernet capable ICs. Further, the devices may carry network adapters for any network in which they might participate (including, but not limited to, personal area networks (PANs), LANs, WANs, and cellular networks).
Cloud computing infrastructure 200 also includes cellular network 203 for use with mobile communication devices. Mobile cellular networks support mobile phones and many other types of mobile devices such as laptops etc. Mobile devices in cloud computing infrastructure 200 are illustrated as mobile phone 204D, laptop 204E, and tablet 204C. A mobile device such as mobile phone 204D may interact with one or more mobile provider networks as the mobile device moves, typically interacting with a plurality of mobile network towers 220, 230, and 240 for connecting to the cellular network 203. Although referred to as a cellular network in
In
To utilize computing resources within cloud resources platform/network 210, network operators may choose to configure data centers 212 using a variety of computing infrastructures. In one embodiment, one or more of data centers 212 are configured using a multi-tenant cloud architecture such that a single server instance 214, which can also be referred to as an application instance, handles requests and serves more than one customer. In some cases, data centers with multi-tenant cloud architecture commingle and store data from multiple customers, where multiple client instances are assigned to a single server instance 214. In a multi-tenant cloud architecture, the single server instance 214 distinguishes between and segregates data and other information of the various customers. For example, a multi-tenant cloud architecture could assign a particular identifier for each customer in order to identify and segregate the data from each customer. In a multitenancy environment, multiple customers share the same application, running on the same operating system, on the same hardware, with the same data-storage mechanism. The distinction between the customers is achieved during application design, thus customers do not share or see each other's data. This is different than virtualization where components are transformed, enabling each customer application to appear to run on a separate virtual machine. Generally, implementing a multi-tenant cloud architecture may have a production limitation, such as the failure of a single server instance 214 causing outages for all customers allocated to the single server instance 214.
In another embodiment, one or more of the data centers 212 are configured using a multi-instance cloud architecture to provide every customer its own unique client instance. For example, a multi-instance cloud architecture could provide each client instance with its own dedicated application server and dedicated database server. In other examples, the multi-instance cloud architecture could deploy a single server instance 214 and/or other combinations of server instances 214, such as one or more dedicated web server instances, one or more dedicated application server instances, and one or more database server instances, for each client instance. In a multi-instance cloud architecture, multiple client instances could be installed on a single physical hardware server where each client instance is allocated certain portions of the physical server resources, such as computing memory, storage, and processing power. By doing so, each client instance has its own unique software stack that provides the benefit of data isolation, relatively less downtime for customers to access the cloud resources platform/network 210, and customer-driven upgrade schedules. An example of implementing a client instance within a multi-instance cloud architecture will be discussed in more detail below when describing
In one embodiment, utilizing a multi-instance cloud architecture, a first client instance may be configured with a client side application interface such as, for example, a web browser executing on a client device (e.g., one of client devices 204A-E of
To facilitate higher availability of client instance 308, application server instances 310A-310D and database server instances 312A and 312B are shown to be allocated to two different data centers 306A and 306B, where one of data centers 306 may act as a backup data center. In reference to
Although
Production instance 445 may be substantially similar to client instance 308, as described in
Production instance 445 may include storage device 450 and operation log 452, and may act as a hosted client instance platform on which various enterprise and/or IT related software applications are deployed and through which related enterprise data from a live database is accessible. The enterprise data may be stored in storage device 450 (or elsewhere) and may correspond to data of a plurality of rows (or records) of a plurality of tables of the live database. The live database may be updated in real-time as a user interacts with (e.g., insert, update, delete a record in a table) production instance 445 (via, e.g., remote client device 455). The software applications hosted on production instance 445 may provide coverage in one or more capability areas of the enterprise such as IT, IT support, security, customer service, technical support, e-mail, backup and storage, HR, finance, legal, marketing, sales, compliance, and governance. For example, the software applications may include components related to the following applications and modules of the enterprise: IT Service Management, Incident Management, Problem Management, Change and Release Management, Benchmarks, Cost Management, Request Management, Configuration Management Database, Asset Management, Service Catalog, Knowledge Management, Survey and Assessment, Service Level Management, IT Operations Management, Discovery, Cloud Management, Event Management, Orchestration, Service Mapping, Operational Intelligence, IT Business Management, Project Portfolio Management, Demand Management, Resource Management, Agile Development, Application Portfolio Management, Cost Transparency, Financial Planning, Financial Reporting, Performance Analytics, Software Asset Management, Security, Security Operations, Governance, Risk and Compliance, Customer Service, Customer Service Management, Field Service Management, Knowledge Management, HR Service Delivery, Case and Knowledge Management, Employee Service Center, Employee Onboarding and Transitions.
Production instance 445 may also include operation log 452 that records all user interactions or activity with production instance 445. For example, any request or user interaction (e.g., HTP request) coming into production instance 445 from an external source for processing (e.g., from a browser on remote client device 455, from an application programming interface (API), and the like) may be treated as an operation (e.g., transaction, activity, and the like) and production instance 445 may record a log of the operation as an entry in operation log 452. Thus, operation log 452 may record all browser activity for production instance 445 for troubleshooting and debugging the operations and events that take place within production instance 445. Although operation log 452 is shown as being separate from storage device 450, operation log 452 may he a database table that is actually stored in storage device 450.
An exemplary operation log entry 505 is described in further detail in
In addition, as shown in GUIs 500 and 600, each operation log entry 505 may also include one or more metrics 610 that indicate performance related to processing of the operation for various criteria. Metrics 610 may be automatically captured and recorded in operation log 452 when the operation to process the corresponding incoming request (e.g., HTTP request) is performed. Metrics 610 correspond to measurable tasks or sub-operations that are triggered when the corresponding operation to process the incoming request is executed. Values of the measurable tasks or sub-operations may be gathered from various sources by production instance 445 and stored as measurement data of metrics 610 in operation log 452. Exemplary metrics that may be measured and corresponding measurement data recorded for each operation log entry 505 in operation log 452 may include: “Response Time” (e.g., Round trip response time for the browser request, in milliseconds); “Network Time” (e.g., Latency time of the network response after the browser request is made, in milliseconds); “SQL time” (e.g., Elapsed time for the execution of SQL server commands for this activity or operation); “SQL Count” (e.g., Number of SQL server commands executed for this activity); “Enterprise rule time” (e.g., Elapsed time for the execution of the enterprise rules for this activity); “Enterprise rule count” (e.g., Number of enterprise rules executed for this activity); “Output length” (e.g., Size of the output string sent by production instance 445 to the browser, in bytes). As shown in GUI 600, other metrics that may also be measured and corresponding measurement data recorded for each operation log entry in operation log 452 may include: Client Response Time, Client Network Time; Total page load time; Number of (e.g., Asynchronous JavaScript and Extensible Markup Language (AJAX®)) requests; Browser time; Client script time; UI policy time; Processing Power; Processing Time (e.g., time it takes to process the incoming request); Starting Timestamp; Ending Timestamp; and the like. The number and type of metrics, or the type of corresponding measurement data that may be gathered, measured, and recorded in operation log 452 is not limited to the above examples. Additional or alternative metrics and corresponding measurement data may also be plausible. Further, a metric may be based on one or more other metrics. As an example, the Processing Time metric may be measured based on measured metrics for the Starting Timestamp and the Ending Timestamp which are already captured and recorded in operation log 452.
In the example shown in
Returning to
Production instance 445 is a live production instance (of a particular enterprise customer) that acts as a hosted client instance platform where various enterprise and/or IT related software applications are deployed, and stores in storage device 450, a live database of related enterprise data (e.g., operation log 452 data of operations, including metrics and corresponding measurement data for all request activity of the particular enterprise). While production instance 445 is a live production instance of a particular customer built from a current released version of a source code, the source code in source code repository 415 of version control system 410 may correspond to a future release version of the hosted client instance platform that is currently under development. However, source code repository 415 may also include source code for the current and/or historical versions of the hosted client instance platform and associated deployed enterprise and/or IT related software applications and other relevant data. For example, one or more of the enterprise and/or IT related software applications may be updated to add, change or remove functionality, or additional applications added to the future release version of the hosted client instance platform that is currently under development. The source code (that is undergoing development) for this new, future release may be stored in source code repository 415 of version control system 410.
Version control system 410 may manage changes to the source code over time and may include a system for centralized or distributed version control for the source code. When version control system 410 is a distributed system, version control system 410 may provide revision control based on a peer-to-peer model. For example, distributed version control system 410 may maintain source code repository 415 on each peer's (e.g., software developer, quality assurance engineer, tester, project manager, administrator and the like) machine as a working copy of the source code that is kept synchronized. Distributed version control system 410 may or may not include a central repository on which client computing devices synchronize. Alternately, version control system 410 may be implemented as a centralized system with source code repository 415 as a single authoritative data store. Version control system 410 (centralized or distributed) may allow for reverting files back to a previous state, reverting an entire project back to a previous state, comparing changes over time, determining who may have introduced an issue and when, and the like.
Check-outs and check-ins (e.g., commits 405) of source code may be done on the source code in source code repository 415 as a single authoritative data store in centralized version control system 410. Alternatively, in distributed version control system 410, no single repository is authoritative, and data can be checked out and checked into (e.g., commits 405) any repository. The changes may then be synchronized across repositories 415 by exchanging patches from peer to peer. In one embodiment, version control system 410 for software development and other version control tasks may be implemented using GIT®, Apache Subversion®, and other similar technologies.
Version control system 410 may be configured to interface with a continuous integration (CI) server 420 via network 401. The interface between version control system 410 and CI server 420 may be realized as an application programming interface (API) or a plugin. CI server 420 may enable setting up of a continuous integration or continuous delivery environment for various combinations of programming languages and source code repositories using pipelines, as well as automating other routine development tasks. The continuous integration or continuous delivery environment of CI server 420 may be implemented via available commercial or open source technologies such as Jenkins®, Appveyor®, Atlassian Bamboo®, and the like. CI server 420 may include artifact repository 425, test instance 430, measurement data module 435, and test module 434. CI server 420 may provide a robust environment for orchestrating the entire chain of building, testing, code analysis, and deployment tools for running performance regression tests on recurring builds based on a current state of the source code (that is under development) in source code repository 415 of version control system 410. A build refers to a bundled distribution that is based on a current state of the source code in source code repository 415. The bundled distribution is used to bring up an instance (e.g., test instance 430) that includes the hosted client instance platform on which various enterprise and/or IT related software applications are deployed and corresponding database schema created for storing related enterprise data (e.g., operation log data) in a database. Thus, test instance 430 brought up from the bundled distribution of the recurring build is similar to production instance 445, except that it is based on an updated (or modified) version of the source code which is under development.
Test module 434 on CI server 420 may enable operations such as create test projects, run test projects, trigger recurring builds (e.g., test instance 430) using build tool (e.g., Maven®) profiles and based on predetermined criteria, and the like. CI server 420 may thus call build tool profiles using jobs at any stage of orchestration (generally in test phase) and complete operations. Test module 434 may also be deployed as a pluggable utility on, for example, remote client device 455, for developers to create or run test projects (e.g., test projects can be plugged in test module 434 as a java archive (JAR) file created by developers).
Test module 434 may provide a library of tools (e.g., profiles, batch, command line options, or scripts) based on which developers may create the test projects for performance regression testing of any module, functionality, software application or application component (e.g., a monitored API resource, a monitored URL resource, and the like) that is deployed on or otherwise associated with a given build artifact or bundled artifact (e.g., test instance 430). For example, test module 434 may provide a developer with an out of the box (OOTB) client for load testing functional behavior and measuring performance of various application components of the given build. The OOTB (e.g., HTTP) client may enable testing by triggering operation requests (e.g., HTTP requests) to the build artifact. The OOTB client may include OOTB scripts for performing various routine test project tasks. The OOTB scripts may be written in any appropriate load testing tool like JMeter®, Selenium®, LoadRunner®, and the like. For example, the OOTB scripts may provide functionality for handling authentication while logging-in to the build artifact. The OOTB scripts may support single sign-on (SSO) authentication in addition to default authentication mechanisms. Test module 434 may also support various file types including excel, xml, html, comma separated value (csv) and the like to read test projects, test cases and/or test results.
CI server 420 may be configured via test projects in test module 434 to trigger and host builds by various means, for example, by commit (e.g., one or more commits 405) in version control system 410, by scheduling via a time based job scheduler, and by requesting a specific build URL. Test module 434 may also be configured to trigger builds based on completion of other builds in a queue. That is, test module 434 may poll version control system 410 at recurring (e.g., every predetermined (fixed or changing) number of hours, days, weeks, months, and the like) intervals, or in response to receiving a poll command from a developer console (e.g., in response to a developer committing (e.g., commit 405) source code to source code repository 415) to bundle a distribution based on a current state of the source code in source code repository 415 and create a new build as test instance 430. For example, CI server 420 may use build tools (e.g., Maven®) to compile source code obtained by polling version control system 410 and create binaries for execution. In one embodiment, test module 434 may “cut” a new build every night using the source code from source code repository 415 to which one or more commits (e.g., commit 405) may have been applied. The new nightly build artifact may be deployed as test instance 430 on CI server 420 for performance regression testing. In one embodiment, one or more of the builds (e.g., artifacts, bundled artifacts) created by test module 434 may be stored in artifact repository 425.
Test instance 430 may be deployed on CI server 420 or on a separate physical machine or server. Like production instance 445, test instance 430 may also act as a hosted diem, instance platform on which various enterprise and/or IT related software applications are deployed. However, test instance 430 may be based on source code that is undergoing development and may be considered to be a production-ready instance that acts as a test hosted client instance platform on which various enterprise and/or IT related software applications are deployed for testing. Database schema of test instance 430 may also be created to provide test instance data storage capability. Thus, test instance 430 may correspond to an OOTB “vanilla” instance whose components including the instance, database, and the like may all physically reside on a same machine dedicated for testing every (e.g., nightly) build during the software development lifecycle. Further, like production instance 445, test instance 430 may also include storage device 431 and operation log 432. Functionality of storage device 431 and operation log 432 is respectively similar to that of storage device 450 and operation log 452 (except that storage device 431 may not include any production data of any enterprise since test instance 430 is not a production instance that is released into production to an end customer). Further detailed description of storage device 431 and operation log 432 is omitted here.
Test module 434 may run one or more test projects (e.g., performance regression test) on recurring build artifacts deployed as test instance 430. A test project may be configured to test performance degradation of predetermined applications, components, or modules deployed for testing on test instance 430 by triggering corresponding operations for processing by test instance 430. In one embodiment, test module 434 may be configured to automatically build new artifacts or builds on a recurring basis, install the build on CI server 420, bundle the installable distribution to bring up test instance 430, and run one or more test projects on the up and running test instance 430. Each test project may track one or more predetermined metrics (e.g., CPU time, SQL time, SQL count, Enterprise rule count, and the like) for a predetermined resource (e.g., a specific URL, a specific API, and the like) associated with test instance 430 whose performance is to be monitored for degradation or deviation. In one embodiment, a developer may write script (e.g., a JAR file) that extends a library of tools provided by an OOTB client of test module 434 to create the test project for testing the one or more predetermined metrics for the predetermined resource. In another embodiment, the developer may simply utilize the library of tools provided by the OOTB client of test module 434 to create the test project for testing the one or more predetermined metrics for the predetermined resource. For example, the developer may simply feed in information regarding the predetermined resource (e.g., one or more URLs, one or more APIs, and the like) to be monitored in a file (e.g., csv file, excel file, text file, and the like) to test module 434, and set one or more parameters (e.g., frequency of builds, automatic processing of the test project, one or more metrics to be tracked, and the like) to create the test project. Test module 434 may then take the file and utilize the OOTB client and scripts to handle authentication login to test instance 430, and load the predetermined resource for a predetermined number of iterations to thereby generate corresponding operation log 432 data including measurement data of a plurality of metrics including the particular metric for the predetermined resource being tracked for regression in the test project.
That is, when the test project is run on test module 434 for each build of test instance 430, one or more requests (e.g., HTTP requests) for accessing the predetermined resource on test instance 430 may be triggered for processing by each build of test instance 430, and corresponding one or more operations may be recorded in corresponding operation log 432 of each build of test instance 430 as one or more operation log entries with corresponding measurement data of a plurality of metrics captured. Based on a comparison of the recorded measurement data of one or more predetermined metrics tracked by the test project between multiple builds of test instance 430, test module 434 may detect performance degradation or deviation for the predetermined resource in a simple manner. That is, by creating test projects that rely on preexisting operation-level metrics that are already captured and recorded as operation log entries in operation log 432, and by triggering corresponding operations of the monitored resource for each (e.g., nightly) build of test instance 430 by running the test project on the build, collecting corresponding operation-level metric measurement data, and capturing any deviations in the metric measurement data between builds, performance degradation for the monitored resource can be detected (and fixed or addressed by debugging) quickly during the software development lifecycle, without waiting for the “full-blown” performance testing at the end of the cycle.
For example, the test project may be created so that for every nightly build of test instance 430, a predetermined metric (e.g., SQL count) of a predetermined resource (e.g., URL resource of “/home.do” webpage whose operation log entry is shown in
By identifying the build (and corresponding one or more source code commits 405) where performance deteriorated (e.g., deviation in value), debugging operations to restore performance as measured by the test project may be dramatically simplified because the regression can be pinpointed to a specific (e.g., nightly) build instead of having to later perform root cause analysis after “code complete” at the end of the software development lifecycle. In addition to regression testing on recurring builds during ongoing software development process based on system 400 and debugging based on the identified regressions on a build-by-build basis, “full-blown” performance testing may also be performed at the end of the development process, before release of the software into production. The “full-blown” performance testing may be performed during a time dedicated for testing at the end of the cycle in a simulated load environment with a production similar hardware infrastructure where a test instance may be installed with a cluster. In “full-blown” performance testing, customer load traffic patterns may be mimicked to test instance behavior in a simulated production “real-world” environment and the behavior and performance of the test instance compared to a current release (e.g., production instance 445). However, performance “bugs” discovered during the “full-blown” performance testing may require extensive root cause analysis and may be very expensive to fix.
By performing regression testing based on system 400, many regressions may be proactively detected (and debugged right away) during ongoing development process, thereby simplifying the debugging process at the end of the cycle during the “full-blown” performance test. That is, inserting this additional layer of testing based on system 400 during development may address regression issues upfront before starting the “full-blown” performance testing. As a result of the additional layer of testing, significant time and cost savings are achieved that may otherwise be incurred during the complex and expensive “full-blown” performance testing. By utilizing measurement data of preexisting metrics that are already being captured in operation log 432 at a granular level of an operation, system 400 may be able to generate performance related output of each build of test instance 430 for regression testing in an efficient manner, without relying on expensive hardware or on expensive end of cycle “full-blown” functional performance tests. Further, by implementing system 400, a developer may immediately be able to assess performance impact of a source code commit (e.g., commit 405) based on trended operation-level metric measurement data. Thus, the need for the developer to have to wait for availability of test infrastructure of the end of cycle “full-blown” functional performance test is diminished.
Since performance regression testing on every build by system 400 is performed based on operations recorded in operation log 432 and corresponding preexisting metrics already captured by test instance 430, consistency of the measurement of the metrics captured in operation log 432 for each build must be ensured. Further, since the measurement of the simple operation-level metrics may vary from machine to machine, system 400 ensures consistency in measurement by utilizing a same set of hardware (e.g., same CPU cycle, same machine time) of a dedicated physical machine (e.g., CI server 420) to perform testing on every build. Measurement data of the metric may thus be compared between builds with high accuracy to detect performance regression.
Returning to
Performance regression framework 437 may be hosted on a server and coupled to CI server 420 via network 401. Alternately, performance regression framework 437 may be provided as a service or run as a process on one or more of CI server 420, production instance 445, and remote client device 455. Performance regression framework 437 provides a tool for consuming measurement data published by measurement data module 435 for each test project, and presenting the data to a user by visualizing the data. In addition, performance regression framework 437 may include logic for automatically detecting performance regression based on the consumed data, and predetermined setting and presenting results of the detection to the user. Performance regression framework 437 may include configuration and reporting module 440, regression detection module 442, and user interface (UI) rendering engine 443. Although not shown in
Configuration and reporting module 440 may act as a front-end for a user (e.g., developer) to set configuration and reporting requirements for each test project. Configuration and reporting module 440 may allow a user to set, for each test project whose measurement data is received from measurement data module 435, one or more metrics that are to be trended over time for each build. Based on the configuration settings, for each recurring build of test instance 430, measurement data received from measurement data module 435 for the one or more metrics to be tracked for regression of a resource being monitored by the corresponding test project may be stored in a storage device. Configuration and reporting module 440 may allow a user compare measurement data of a predetermined metric for two or more builds (e.g., previous released version corresponding to production instance 445, and current nightly build) specifically identified by the user. Configuration and reporting module 440 may also allow a user to set reporting parameters so that appropriate users may be notified (e.g., via email) when updated measurement data from measurement data module 435 is collected, trended, and visualized.
UI rendering engine 443 may include logic to visualize the measurement data at the front end. For example, UI rendering engine 443 may include logic to visualize widgets on a client device (e.g., remote client device 455). Widgets may refer to graphical visualizations of scores (e.g., measurement data) of a metric associated with a monitored resource. Widgets can be enhanced by adding targets, thresholds, trendlines, and useful comments for significant changes. An exemplary graph widget visualizing a trendline of a monitored metric based on measurement data from measurement data module 435 is illustrated in
In the example shown in
Performance regression framework 437 may also include regression detection module 442 for automatically detecting regression for a given test project based on received corresponding measurement data. Based on preset parameters set by a user, empirical or heuristics data, and the like, regression detection module 442 may set a condition (e.g., threshold value for a build, threshold difference between builds, and the like) corresponding to a metric of a monitored resource. Based on the value of the metric of the monitored resource for each build, regression detection module 442 may determine whether the set condition is satisfied. Regression detection module 442 may thus automatically detect regression corresponding to a particular build for the monitored resource. Configuration and reporting module 440 may further report the automatically detected regression to appropriate users so the users may act on the regression right away and prevent the deterioration in performance, or otherwise determine (e.g., via cost benefit analysis) whether the deterioration in performance is justified for making the corresponding change in the source code that caused the deviation.
In addition to detecting the specific build that caused the performance degradation for the monitored resource, regression detection module 442 may further include logic to narrow down to a specific commit (e.g., one of commits 405A-N), or to a specific line of code in the specific commit, corresponding to the specific build that caused the regression. For example, upon detecting the particular build that caused the regression, regression detection module 442 may trigger execution of a “child” test project by test module 434. Test module 434 may then in turn run the “child” test project for the monitored resource by replaying a plurality of commits (e.g., commits 405) that are associated with the particular detected (“parent”) build and installing a plurality of “child” builds (e.g., bundled artifacts) respectively corresponding to plurality of commits. Test module 434 may then run the test project on each “child” build (e.g., bundled artifact) to detect measurement data of the metric of the monitored resource from corresponding operation log 432. Based on the detected measurement data for each “child” build, regression detection module 442 may be able to narrow down to a specific “child” build (and corresponding source code commit) that caused the performance regression. Configuration and reporting module 440 may then report the automatically detected deviation of the “child” build to appropriate users so that targeted and efficient debugging operations may be performed with respect to the particular commit (e.g. one of commits 405A-N), or the particular line of code in the particular commit, that caused performance degradation of the metric.
At block 810, test module 434 may run a particular test project on test instance 430 built at block 805. The test project may be for a particular monitored resource (e.g., specific API or URL on test instance 430) and may be configured to track regression based on one or more preexisting metrics (e.g., SQL time, processing time, and the like) that are already captured for each operation (e.g., incoming HTTP request to test instance 430) on test instance 430. Thus, at block 810A, test module 424 may execute the test project by accessing the associated monitored resource, and at block 810B, test instance 430 may automatically record in operation log 432, operations associated with processing the request associated with accessing the monitored resource as operation log entries. Further, at block 810B, test instance 430 may automatically record for each operation log entry, measurement data of a plurality of metrics (e.g., see
At block 820, measurement data module 435 may collect measurement data of metrics associated with operation log entries in operation log 432 that correspond to or are triggered by processing the request associated with accessing the monitored resource tracked by the test project by test instance 430. The collected measurement data may then be published by measurement data module 435 to performance regression framework for consumption and trending. For example, an API of measurement data module 435 may push the collected data to performance regression framework 437. Thus, measurement data module 435 may keep publishing the collected measurement data for each build.
Further, at block 815, test module 434 may tear down the current build of test instance 430 (installed at block 805) after relevant measurement data in operation log of the current build of test instance 430 has been published. For example, test module 434 may use an API to tear down the current build of test instance 430. Alternately, test module 434 may store the current build as an installable distribution in artifact repository 425.
Processing performed at blocks 805-820 may be repeated for each build of test instance 430. Setup of each build may be triggered by test module 434 based on predetermined criteria (e.g., on a periodic or aperiodic basis, “on-demand” based on user operation, by polling version control system 410 for detecting changes to the source code, and the like).
For each build (e.g., artifact, bundled artifact) installed by CI server 420, performance regression framework 437 may process the collected measurement data at block 825 by extracting the measurement data (block 825A) published by measurement data module 435 (e.g., using an API). At block 825B, performance regression framework 437 may store the extracted measurement data for one or more test projects in a storage device. Thus, block 825B, may result in storage of measurement data for a plurality of builds.
At block 830, performance regression framework 437 may access one or more settings for a test project whose data was extracted and stored at block 825 to configure reports (block 830A), notify appropriate users, and automatically detect regression (block 830B) for the monitored resource associated with the test project. For example, the settings may specify out of the plurality of metrics whose measurement data is stored at block 825B, a particular metric that is to be trended over time for the plurality of builds. The settings may also include notification and reporting settings for outputting the measurement data for display. At block 830B, performance regression framework 437 may automatically detect regression for a particular build, commit, or line of code, based on a comparison of corresponding measurement data of a metric with corresponding measurement data of other builds, commits, or lines of code. For example, performance regression framework 437 may detect the regression when a value or a change in the value of the measurement of the metric of the monitored resource satisfies a predetermined condition or meets a predetermined threshold. Performance regression framework 437 may then automatically notify a user of the detected regression for the particular build, commit, or line of code, for enabling targeted and efficient debugging (e.g., alert the developer who checked in source code whose corresponding build caused the deviation). For example, the data may be output as a widget (e.g., graph) showing a trendline as shown in
In some examples (without abstraction) computing device 900 and its elements as shown in
As illustrated in
Persons of ordinary skill in the art are aware that software programs may be developed, encoded, and compiled in a variety of computing languages for a variety of software platforms and/or operating systems and subsequently loaded and executed by processor 905. In one embodiment, the compiling process of the software program may transform program code written in a programming language to another computer language such that the processor 905 is able to execute the programming code. For example, the compiling process of the software program may generate an executable program that provides encoded instructions (e.g., machine code instructions) for processor 905 to accomplish specific, non-generic, particular computing functions.
After the compiling process, the encoded instructions may then be loaded as computer executable instructions or process steps to processor 905 from storage 920, from memory 910, and/or embedded within processor 905 (e.g., via a cache or on-board ROM). Processor 905 may be configured to execute the stored instructions or process steps in order to perform instructions or process steps to transform the computing device into a non-generic, particular, specially programmed machine or apparatus. Stored data, e.g., data stored by a storage device 920, may be accessed by processor 905 during the execution of computer executable instructions or process steps to instruct one or more components within the computing device 900.
A user interface (e.g., output devices 915 and input devices 930) can include a display, positional input device (such as a mouse, touchpad, touchscreen, or the like), keyboard, or other forms of user input and output devices. The user interface components may be communicatively coupled to processor 905. When the output device is or includes a display, the display can be implemented in various ways, including by a liquid crystal display (LCD) or a cathode-ray tube (CRT) or light emitting diode (LED) display, such as an organic LED (OLED) display. Persons of ordinary skill in the art are aware that the computing device 900 may comprise other components well known in the art, such as sensors, powers sources, and/or analog-to-digital converters, not explicitly shown in
At least one embodiment is disclosed and variations, combinations, and/or modifications of the embodiment(s) and/or features of the embodiment(s) made by a person having ordinary skill in the art are within the scope of the disclosure. Alternative embodiments that result from combining, integrating, and/or omitting features of the embodiment(s) are also within the scope of the disclosure. Where numerical ranges or limitations are expressly stated, such express ranges or limitations may be understood to include iterative ranges or limitations of like magnitude falling within the expressly stated ranges or limitations (e.g., from about 1 to about 10 includes 2, 3, 4, etc.; greater than 0.10 includes 0.11, 0.12, 0.13, etc.). The use of the term “about” means ±10% of the subsequent number, unless otherwise stated.
Use of the term “optionally” with respect to any element of a claim means that the element is required, or alternatively, the element is not required, both alternatives being within the scope of the claim. Use of broader terms such as comprises, includes, and having may be understood to provide support for narrower terms such as consisting of, consisting essentially of, and comprised substantially of Accordingly, the scope of protection is not limited by the description set out above but is defined by the claims that follow, that scope including all equivalents of the subject matter of the claims. Each and every claim is incorporated as further disclosure into the specification and the claims are embodiment(s) of the present disclosure.
It is to be understood that the above description is intended to be illustrative and not restrictive. For example, the above-described embodiments may be used in combination with each other. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the invention therefore should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It should be noted that the discussion of any reference is not an admission that it is prior art to the present invention, especially any reference that may have a publication date after the priority date of this application.
Claims
1. A method comprising:
- installing, by one or more processors of a continuous integration (CI) server, a build of a test instance, wherein the build of the test instance is representative of a version of a portion of a source code updated by a plurality of commits;
- receiving, by the one or more processors, a file for a performance regression test that monitors a resource, the file identifying the resource and one or more metrics to monitor associated with accessing the resource;
- running, on the build of the test instance and by the one or more processors, the performance regression test based on the file, wherein the build of the test instance is configured to access the resource during the running of the performance regression test;
- recording, in an operation log, first measurement data of the one or more metrics triggered by a first operation of a plurality of operations, wherein the first measurement data is logged in response to an incoming request for processing associated with accessing the resource via the build of the test instance, and wherein the operation log comprises second measurement data from running the performance regression test on one or more other builds of the test instance;
- extracting, by the one or more processors, the measurement data from the operation log;
- detecting, by the one or more processors, a performance regression for the resource in the build of the test instance based on a comparison of the first measurement data from the build of the test instance to the second measurement data from the one or more other builds of the test instance; and
- generating, by the one or more processors, a notification for the detected performance regression, wherein the notification is configured to indicate the build of the test instance as a regressed build.
2. The method of claim 1, wherein the one or more metrics comprise a processing time, a central processing unit percentage, a response time, a network time, a structured query language (SQL) time, a SQL count, an enterprise rule time, an enterprise rule count, a client response time, a client network time, a total page load time, a number of requests, a browser time, a client script time, or a user interface policy time, or a combination thereof
3. The method of claim 1, comprising:
- installing, by the one or more processors, a plurality of bundled artifacts corresponding to each of the plurality of commits on the version of the source code corresponding to the build of the test instance in response to detecting performance regression for the resource in the build of the test instance;
- running, by the one or more processors, the performance regression test on each of the plurality of bundled artifacts to determine a specific bundled artifact responsible for the performance regression in the build of the test instance; and
- generating an additional notification indicating the specific bundled artifact or specific commit of the plurality of commits responsible for the performance regression.
4. The method of claim 1, comprising, transmitting the notification to a computing device.
5. The method of claim 1, comprising running an additional performance regression test on the build of the test instance.
6. The method of claim 1, wherein the performance regression test is run on the one or more other builds of the test instance before the performance regression test is run on the build of the test instance.
7. A system comprising:
- a continuous integration (CI) server device configured to perform actions comprising: installing a build of a test instance, wherein the build of the test instance is representative of a version of a portion of a source code updated by a plurality of commits; receiving a file for a performance regression test that monitors a resource, the file identifying the resource and one or more metrics to monitor associated with accessing the resource; running, on the build of the test instance, the performance regression test based on the file, wherein the build of the test instance is configured to access the resource during the running of the performance regression test; and recording, in an operation log, first measurement data of the one or more metrics triggered by a first operation of a plurality of operations, wherein the first measurement data is logged in response to an incoming request for processing associated with accessing the resource via the build of the test instance, wherein the operation log comprises second measurement data from running the performance regression test on one or more other builds of the test instance, and wherein the first measurement data and the second measurement data can be compared to identify a performance regression.
8. The system of claim 7, wherein the one or more metrics comprise a processing time, a central processing unit percentage, a response time, a network time, a structured query language (SQL) time, a SQL count, an enterprise rule time, an enterprise rule count, a client response time, a client network time, a total page load time, a number of requests, a browser time, a client script time, or a user interface policy time, or a combination thereof.
9. The system of claim 7, wherein the CI server device is configured to perform actions comprising:
- extracting the first measurement data and the second measurement data from the operation log;
- detecting the performance regression for the resource in the build of the test instance based on a comparison of the first measurement data from the build of the test instance to the second measurement data from the one or more other builds of the test instance; and
- generating a notification for the detected performance regression, wherein the notification is configured to indicate the build of the test instance as a regressed build.
10. The system of claim 9, wherein the CI server device is configured to perform actions comprising:
- transmitting the notification to a computing device.
11. The system of claim 10, wherein the CI server device is configured to perform actions comprising:
- installing a plurality of bundled artifacts corresponding to each of the plurality of commits on the version of the source code corresponding to the build of the test instance in response to detecting performance regression for the resource in the build of the test instance;
- running the performance regression test on each of the plurality of bundled artifacts to determine a specific bundled artifact of the plurality of bundled artifacts responsible for the performance regression in the build of the test instance; and
- generating a notification indicating the specific bundled artifact or specific commit of the plurality of commits responsible for the performance regression.
12. The system of claim 7, comprising an additional server device, wherein the additional server device is configured to perform actions comprising:
- extracting the first measurement data and the second measurement data from the operation log;
- detecting the performance regression for the resource in the build of the test instance based on a comparison of the first measurement data from the build of the test instance to the second measurement data from the one or more other builds of the test instance; and
- generating a notification for the detected performance regression, wherein the notification is configured to indicate the build of the test instance as a regressed build.
13. The system of claim 12, wherein the additional server device comprises a remote client device.
14. The system of claim 13, wherein generating the notification comprises executing logic to visualize, on the remote client device, one or more widgets, wherein the one or more widgets comprise a graphical visualization of measurement data of the resource for a plurality of builds including the regressed build and the one or more other builds of the test instance.
15. One or more non-transitory, tangible, and computer-readable media, storing instructions that, when executed by one or more processors of a continuous integration (CI) server, cause the CI server to perform operations comprising:
- installing a build of a test instance, wherein the build of the test instance is representative of a version of a portion of a source code updated by a plurality of commits;
- receiving a file for a performance regression test that monitors a resource, the file identifying the resource and one or more metrics to monitor associated with accessing the resource;
- running, on the build of the test instance, the performance regression test based on the file, wherein the build of the test instance is configured to access the resource during the running of the performance regression test;
- recording, in the operation log, first measurement data of the one or more metrics triggered by a first operation of a plurality of operations, wherein the first measurement data is logged in response to an incoming request for processing associated with accessing the resource via the build of the test instance, and wherein the operation log comprises second measurement data from running the performance regression test on one or more other builds of the test instance;
- extracting the measurement data from the operation log;
- detecting a performance regression for the resource in the build of the test instance based on a comparison of the measurement data from the build of the test instance to the second measurement data from the one or more other builds of the test instance; and
- generating, by the one or more processors, a notification for the detected performance regression, wherein the notification is configured to indicate the build of the test instance as a regressed build.
16. The one or more non-transitory, tangible, and computer-readable media of claim 15, wherein the one or more metrics comprise a processing time, a central processing unit percentage, a response time, a network time, a structured query language (SQL) time, a SQL count, an enterprise rule time, an enterprise rule count, a client response time, a client network time, a total page load time, a number of requests, a browser time, a client script time, or a user interface policy time, or a combination thereof.
17. The one or more non-transitory, tangible, and computer-readable media of claim 15, wherein the operations comprise:
- installing a plurality of bundled artifacts corresponding to each of the plurality of commits on the version of the source code corresponding to the build of the test instance in response to detecting regression for the resource in the build of the test instance;
- running the performance regression test on each of the plurality of bundled artifacts to determine a specific bundled artifact of the plurality of bundled artifacts responsible for the performance regression in the build of the test instance; and
- generating an additional notification indicating the specific bundled artifact or specific commit of the plurality of commits responsible for the performance regression.
18. The one or more non-transitory, tangible, and computer-readable media of claim 15, wherein the performance regression test is run on the one or more other builds of the test instance before the performance regression test is run on the build of the test instance.
19. The one or more non-transitory, tangible, and computer-readable media of claim 15, wherein the operations comprise transmitting the notification to a remote client device.
20. The one or more non-transitory, tangible, and computer-readable media of claim 15, wherein running the performance regression test on the build of the test instance and the one or more other builds of the test instance comprises loading the resource for a predetermined number of iterations.
Type: Application
Filed: Sep 17, 2020
Publication Date: Jan 7, 2021
Inventors: Ravishankar Srinivasan (Milpitas, CA), Sravan Srungarapu (Fremont, CA)
Application Number: 17/024,500