Abstract: Embodiments presented herein provide techniques for evaluating an asynchronous application using a test framework. The test framework may perform a load test of an asynchronous application or service composed from a collection of applications or services. To do so, the test framework may submit transactions to a distributed application at a specified transaction rate and monitor how the distributed application operates at that transaction rate. An aggregate load test component may evaluate the remaining work pending at work accumulation points of the distributed application to determine whether the distributed application can sustain the specified transaction rate. A transaction tracking component may initiate transactions to generate load at the specified transaction rate without blocking while the transactions are processed by the distributed application.

Abstract: A network-based scalable production load test service may be implemented on a provider network including a plurality of computing devices in order to provide load testing for network-based production systems. In some embodiments, the plurality of computing devices is configured to receive a request to capture to a load test data repository items of transaction data for a network-based production service. In some embodiments, the plurality of computing devices is configured to capture to the load test data repository the items of transaction data. The transaction data include input to the network-based production service over a network. In some embodiments, in response to a load test specification received by the scalable production load test service, the plurality of computing devices is configured to dynamically allocate one or more resources to perform a load test of the network-based production service according to the load test specification.

Abstract: Systems and methods for validation of log formats are described herein. Log data is stored via a logging service in a data store or other storage system. An example log or proposed log format is received by the logging service. The proposed log format is validated against validation rules provided by log consumers.

Abstract: Methods and systems for automated tuning of a service configuration are disclosed. An optimal configuration for a test computer is selected by performing one or more load tests using the test computer for each of a plurality of test configurations. The performance of a plurality of additional test computers configured with the optimal configuration is automatically determined by performing additional load tests using the additional test computers. A plurality of production computers are automatically configured with the optimal configuration if the performance of the additional test computers is improved with the optimal configuration.

Abstract: A contributed test management system receives a first request from a consumer system, where the first request comprises a request for a contributed test to be added to a deployment pipeline of a producer system, and where the contributed test is associated with an application component in the deployment pipeline. The contributed test management system causes the contributed test to test a code update provided by the producer system for the application component in the deployment pipeline, detects whether the first test fails during execution, and, if so indicates to the consumer system that the first test has failed.

Abstract: Methods, systems, and computer-readable media for ordered test execution based on code coverage are disclosed. A suite of tests are executed on a first version of program code to generate data indicative of code coverage of respective tests with respect to the program code. A mapping of the tests to the program code is determined based at least in part on the data indicative of code coverage and is stored. The mapping comprises data indicative of one or more portions of the program code exercised by respective tests from the suite. Based at least in part on the mapping of the tests to the program code and on data indicative of one or more modified or new portions of a second version of the program code, a subset of the tests is determined that are likely to be exercised by the second version of the program code.

Abstract: Optimization preferences are defined for optimizing execution of a distributed application. Candidate sets of application parameter values may be tested in test execution environments. Measures of performance for metrics of interest are determined based upon the execution of the distributed application using the candidate sets of application parameter values. Utility curves may be utilized to compute measures of effectiveness for metrics of interest. A multi-attribute rollup operation may utilize the computed measures of effectiveness and weights to compute a grand measure of merit (MOM) for the candidate sets of application parameter values. An optimized set of application parameter values may then be selected based upon the computed grand MOMs. The optimized set of application parameter values may be deployed to a production execution environment executing the distributed application.

Abstract: Methods, systems, and computer-readable media for determining the maximum throughput of a service are disclosed. A first sequence of load tests is initiated for a service host. Individual ones of the load tests comprise determining a respective throughput at the service host for a respective number of concurrent connections to the service host. The number of concurrent connections increases nonlinearly in at least a portion of the first sequence of load tests. The first sequence of load tests is discontinued when the throughput is not increased by a threshold from one of the load tests to the next. An estimated maximum throughput for the service host is determined based at least in part on the first sequence of load tests. The estimated maximum throughput corresponds to a particular number of concurrent connections to the service host.

Abstract: Some embodiments include methods and systems for performing non-deterministic load testing of a network-based production service. Some embodiments perform, using a load testing system implemented on one or more computers, receiving a load testing specification. The load testing specification indicates one or more load testing conditions for performing non-deterministic load testing of a network-based production service executing transactions in a networked computing environment. Some embodiments perform requesting from the virtual computing service resources to perform the non-deterministic load testing of the network-based production service, perform the non-deterministic load testing of the network-based production service. The non-deterministic load testing comprises executing test transactions on the network-based production service at pseudo-randomly varying times to measure system performance of the network-based production service at a plurality of levels of transaction intensity.

Abstract: Embodiments presented herein provide techniques for dynamically generating synthetic test data used to test a web service. In one embodiment, a service description language document defining a web service may include a test interface definition. The test interface definition specifies rules for generating the synthetic data to use when testing API interfaces exposed by the web service, e.g., to generate synthetic data needed carry out load and performance testing. Including rules for generating test data in the service description language document provides a centralized and authoritative source for both building and testing the web service.

Abstract: Methods and systems for automated tuning of a service configuration are disclosed. An optimal configuration for a test computer is selected by performing one or more load tests using the test computer for each of a plurality of test configurations. The performance of a plurality of additional test computers configured with the optimal configuration is automatically determined by performing additional load tests using the additional test computers. A plurality of production computers are automatically configured with the optimal configuration if the performance of the additional test computers is improved with the optimal configuration.

Abstract: Methods and systems for automated tuning of a service configuration are disclosed. An optimal configuration for a test computer is selected by performing one or more load tests using the test computer for each of a plurality of test configurations. The performance of a plurality of additional test computers configured with the optimal configuration is automatically determined by performing additional load tests using the additional test computers. A plurality of production computers are automatically configured with the optimal configuration if the performance of the additional test computers is improved with the optimal configuration.

Abstract: Methods and systems for performance testing in a software deployment pipeline are disclosed. One or more performance tests are automatically performed on a build of a software product in a test environment in response to deploying the build to the test environment. One or more performance metrics are collected based on the performance tests. Based on the performance metrics, the build of the software product is accepted or rejected.

Abstract: Methods, systems, and computer-readable media for load testing with automated service dependency discovery are disclosed. A request is received to approve load testing for a service. One or more downstream services are identified for the service. The one or more downstream services are identified based at least in part using automated discovery. The availability of the one or more downstream services for load testing is determined. The request is approved or denied based at least in part on the availability of the one or more downstream services for load testing.

Abstract: A generic transaction generator framework for testing a network-based production service may work in conjunction with a product-specific transaction creator module that executes transactions on the service. The transaction creator module may include runtime-discoverable information, such as source code annotations, to communicate product specific details to the framework. Runtime-discoverable information may identify transaction types, transaction methods, as well as dependencies between different transaction types and transaction methods. The framework may generate and execute various test transactions and may call a substituted transaction method for a transaction type on which a generated transaction depends prior to calling the generated transaction. The output from the substituted transaction may be used as input to the generated transaction when executed subsequently.

Abstract: Technologies are disclosed herein for generating comments in a source code review tool using code analysis tools. A producer module can be executed in order to obtain source code from a source code review tool. One or more source code analysis modules can then be executed in order to analyze the source code. A reporter module can then store the output of the source code analysis modules as comments in the source code review tool for use by a developer of the source code. The producer, reporter, and source code analysis modules can be executed in response to a request from the source code developer to perform a source code review, by a job scheduler, or in another manner. An application programming interface (API) exposed by the source code review tool can be utilized to obtain the source code and to store the comments associated with the source code.

Abstract: Embodiments presented herein provide techniques for evaluating an asynchronous application using a test framework. The test framework may perform a load test of an asynchronous application or service composed from a collection of applications or services. To do so, the test framework may submit transactions to a distributed application at a specified transaction rate and monitor how the distributed application operates at that transaction rate. An aggregate load test component may evaluate the remaining work pending at work accumulation points of the distributed application to determine whether the distributed application can sustain the specified transaction rate. A transaction tracking component may initiate transactions to generate load at the specified transaction rate without blocking while the transactions are processed by the distributed application.

Abstract: Techniques are described for aggregating code coverage data generated from various types of testing of software modules, and automatically determining whether to promote software upwards in a multi-level software deployment hierarchy based on the aggregated code coverage data. In embodiments, a code coverage metric is determined for a software module, and the metric is compared to a set of promotion criteria, including whether the metric meets a predetermined threshold for quality. In some cases, the threshold may be a general threshold, a threshold based on the level of possible promotion, and/or a threshold that is based on an identified category for the software module such as whether the module is a front-end module, a shared module, a legacy module, or a critical module.

Abstract: A software testing framework provides functionality for utilizing pre-existing tests to load and performance test a network service. Methods can be tagged with annotations indicating that they are tests, such as integration tests. The methods implementing the integration tests can also be tagged with other types of annotations that can be used to select individual tests for use in testing, such as annotations indicating whether a test is a positive or negative test, annotations specifying dependencies upon other tests, or annotations indicating that a test is a member of a test suite. The annotations can be utilized in conjunction with test selection criteria to select individual integration tests for use in load and performance testing of the network service. The selected integration tests can be deployed to and executed upon load-generating instances to execute the integration tests and generate requests to the network service at high throughput.

Abstract: Technologies are disclosed herein for distributed software testing. A software testing service is disclosed that can receive tests for software under test. The software testing service can determine a number of test instances to utilize to perform the tests. The determined number of test instances can then be provisioned, such as through the use of an on-demand computing service executing in a service provider network. The test instances can also be configured with the software under test and a test runner capable of performing the tests on the software. The software testing service can place the tests on a queue, such as a queue provided by a queue service. The test runners executing on the test instances can dequeue the tests and perform the tests on the software. Once the testing of the software under test has been completed, the test instances can be de-provisioned.