Abstract: A generic transaction generator framework for testing a network-based production service may work in conjunction with a product-specific transaction creator module that executes transactions on the service. The transaction creator module may include runtime-discoverable information, such as source code annotations, to communicate product specific details to the framework. Runtime-discoverable information may identify transaction types, transaction methods, data provider methods and data sources. The framework may generate and execute various test transactions and may call a data provider method to prepare data for the transaction and pass the prepared data to a transaction method. The framework may also load and parse test data from a data source and provide the test data to the data provider method for use when preparing data for the transaction.

Abstract: Technologies are disclosed herein for distributed software testing. A software testing service is disclosed that can receive tests for software under test. The software testing service can determine a number of test instances to utilize to perform the tests. The determined number of test instances can then be provisioned, such as through the use of an on-demand computing service executing in a service provider network. The test instances can also be configured with the software under test and a test runner capable of performing the tests on the software. The software testing service can place the tests on a queue, such as a queue provided by a queue service. The test runners executing on the test instances can dequeue the tests and perform the tests on the software. Once the testing of the software under test has been completed, the test instances can be de-provisioned.

Abstract: Methods, systems, and computer-readable media for determining the maximum throughput of a service are disclosed. A first sequence of load tests is initiated for a service host. Individual ones of the load tests comprise determining a respective throughput at the service host for a respective number of concurrent connections to the service host. The number of concurrent connections increases nonlinearly in at least a portion of the first sequence of load tests. The first sequence of load tests is discontinued when the throughput is not increased by a threshold from one of the load tests to the next. An estimated maximum throughput for the service host is determined based at least in part on the first sequence of load tests. The estimated maximum throughput corresponds to a particular number of concurrent connections to the service host.

Abstract: Optimization preferences are defined for optimizing execution of a distributed application. Candidate sets of application parameter values may be tested in test execution environments. Measures of performance for metrics of interest are determined based upon the execution of the distributed application using the candidate sets of application parameter values. Utility curves may be utilized to compute measures of effectiveness for metrics of interest. A multi-attribute rollup operation may utilize the computed measures of effectiveness and weights to compute a grand measure of merit (MOM) for the candidate sets of application parameter values. An optimized set of application parameter values may then be selected based upon the computed grand MOMs. The optimized set of application parameter values may be deployed to a production execution environment executing the distributed application.

Abstract: Methods, systems, and computer-readable media for determining the maximum throughput of a service are disclosed. A first sequence of load tests is initiated for a service host. Individual ones of the load tests comprise determining a respective throughput at the service host for a respective number of concurrent connections to the service host. The number of concurrent connections increases nonlinearly in at least a portion of the first sequence of load tests. The first sequence of load tests is discontinued when the throughput is not increased by a threshold from one of the load tests to the next. An estimated maximum throughput for the service host is determined based at least in part on the first sequence of load tests. The estimated maximum throughput corresponds to a particular number of concurrent connections to the service host.

Abstract: One or more computers is configured to run an end-to-end test including at least a plurality of independent tests of multiple stages of an asynchronous multi-stage data processing system. One of the set of independent tests is configured to send a request for test input data from a test data repository service for a particular stage. A converted version of the test input data is obtained. A comparison of the converted version to the output of the particular stage to verify operation of the particular stage is obtained. The output of the particular stage is transmitted to the test data repository service. One or more computers is configured to provide the test data repository service. The test data repository service is configured to store in the test data storage the output of the particular stage as test input data for a next stage of the asynchronous multi-stage data processing system.

Abstract: A generic transaction generator framework for testing a network-based production service may work in conjunction with a product-specific transaction creator module that executes transactions on the service. The transaction creator module may include runtime-discoverable information to communicate product specific details to the framework. Runtime-discoverable information may identify initialization methods, terminate methods, transaction types, transaction methods, transaction dependencies as well as testing parameters, such as transaction rate, testing period and a desired distribution of transaction types. The framework may generate and execute various test transactions and collect performance metrics regarding how well the service performed the test transactions.

Abstract: Methods, systems, and computer-readable media for ordered test execution to enable faster feedback are disclosed. A likelihood of failure is estimated for individual tests in a set of tests. Based at least in part on the likelihood of failure, an ordered sequence is determined for the set of tests, such that the tests are ordered in estimated likelihood of failure. The set of tests is initiated in the ordered sequence, such that one or more computer-executable programs are subjected to individual ones of the tests. A failure of one or more of the tests is determined prior to performing one or more remaining tests in the ordered sequence.

Abstract: A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.

Abstract: Automated methods of creating a predictive test load, using clustering to create the predictive test load from production data, for testing in a production environment. Client requests to a production system are captured and processed into production request data. The production request data undergoes a clustering analysis to determine cluster definitions. In some embodiments, the production request data is turned into vectors and the vectors undergo the clustering analysis instead of the production data. A specification may be received that specifies modifications to be made to the production data. The production request data may be processed using the cluster definitions and the specified modifications to create a predictive test load. In some embodiments, the predictive test load is played to a production system to simulate a predictive load according to a test plan. The test plan may specify the rate at which the test load is replayed.

Abstract: Methods and systems for load generation for scalable load testing are disclosed. A plurality of job descriptions are generated based on a load step description. The load step description specifies a total transaction frequency or a total number of concurrent connections for a load test of a service over a period of time. The job descriptions specify subdivisions of the total transaction frequency or the total number of concurrent connections and subdivisions of the period of time. The job descriptions are placed in a job queue. A plurality of worker hosts remove the job descriptions from the job queue and concurrently execute local jobs based on the job descriptions.

Abstract: Techniques are described for aggregating code coverage data generated from various types of testing of software modules, and automatically determining whether to promote software upwards in a multi-level software deployment hierarchy based on the aggregated code coverage data. In embodiments, a code coverage metric is determined for a software module, and the metric is compared to a set of promotion criteria, including whether the metric meets a predetermined threshold for quality. In some cases, the threshold may be a general threshold, a threshold based on the level of possible promotion, and/or a threshold that is based on an identified category for the software module such as whether the module is a front-end module, a shared module, a legacy module, or a critical module.

Abstract: A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.

Abstract: Proposed updates to systems are evaluated in a manner that is automated and horizontally scalable. Input to a first system is provided to a second system. The first system and second system process the input and each generates output. The output from the first system and second system is analyzed and differences in the output data between the two systems are identified. Analyzing the output may be performed by a fleet of data processing units and the work of analyzing the output may be performed such that differences in the output data are traceable to subsystems of the second system that caused the differences.

Abstract: Systems and associated processes for testing a reverse proxy server are disclosed. A backend proxy server test system can receive a request from a reverse proxy server under test. The request may be generated in response to a request from a test client to access a backend service. In responding to the received request, the backend proxy server test system can include a copy of the received request. Upon the test client receiving the response from the proxy server to the test client's request, the test client can extract the embedded copy of the received request that the reverse proxy server generated to determine whether it matches the request that a functioning reverse proxy server generates. Based, at least in part on the result of this comparison, the test client can determine whether the reverse proxy server is malfunctioning.

Abstract: Methods and systems for automated tuning of a service configuration are disclosed. An optimal configuration for a test computer is selected by performing one or more load tests using the test computer for each of a plurality of test configurations. The performance of a plurality of additional test computers configured with the optimal configuration is automatically determined by performing additional load tests using the additional test computers. A plurality of production computers are automatically configured with the optimal configuration if the performance of the additional test computers is improved with the optimal configuration.

Abstract: A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.

Abstract: A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.

Abstract: Methods and systems for automated tuning of a service configuration are disclosed. An optimal configuration for a test computer is selected by performing one or more load tests using the test computer for each of a plurality of test configurations. The performance of a plurality of additional test computers configured with the optimal configuration is automatically determined by performing additional load tests using the additional test computers. A plurality of production computers are automatically configured with the optimal configuration if the performance of the additional test computers is improved with the optimal configuration.

Abstract: Systems and methods for validation of log formats are described herein. Log data is stored via a logging service in a data store or other storage system. An example log or proposed log format is received by the logging service. The proposed log format is validated against validation rules provided by log consumers.