Abstract: Described are systems, methods, and techniques for collecting, analyzing, processing, and storing time series data and for evaluating and determining whether and how to include late or delayed data points for inclusion when publishing or storing the time series data. Maximum delay values can identify a duration for waiting for late or delayed data, such as prior to publication. In some examples, maximum delay values can be dynamically adjustable based on a statistical evaluation process. For late or delayed data points that are received after the maximum delay elapses, some data points can be included in the stored time series data, such as if they are received in the same order that they are generated.

Abstract: Described are systems, methods, and techniques for collecting, analyzing, processing, and storing time series data and for evaluating and determining whether and how to include late or delayed data points when publishing or storing the time series data. Maximum delay values can identify a duration for waiting for late or delayed data, such as prior to publication. In some examples, maximum delay values can be dynamically adjustable based on a statistical evaluation process. For late or delayed data points that are received after the maximum delay elapses, some data points can be included in the stored time series data, such as if they are received in the same order that they are generated.

Abstract: Described are systems, methods, and techniques for collecting, analyzing, processing, and storing time series data and for evaluating and determining whether and how to include late or delayed data points when publishing or storing the time series data. Maximum delay values can identify a duration for waiting for late or delayed data, such as prior to publication. In some examples, maximum delay values can be dynamically adjustable based on a statistical evaluation process. For late or delayed data points that are received after the maximum delay elapses, some data points can be included in the stored time series data, such as if they are received in the same order that they are generated.

Abstract: Described are systems, methods, and techniques for collecting, analyzing, processing, and storing time series data and for evaluating and determining whether and how to include late or delayed data points for inclusion when publishing or storing the time series data. Maximum delay values can identify a duration for waiting for late or delayed data, such as prior to publication. In some examples, maximum delay values can be dynamically adjustable based on a statistical evaluation process. For late or delayed data points that are received after the maximum delay elapses, some data points can be included in the stored time series data, such as if they are received in the same order that they are generated.

Abstract: A network-based scalable production load test service may be implemented on a provider network including a plurality of computing devices in order to provide load testing for network-based production systems. In some embodiments, the plurality of computing devices is configured to receive a request to capture to a load test data repository items of transaction data for a network-based production service. In some embodiments, the plurality of computing devices is configured to capture to the load test data repository the items of transaction data. The transaction data include input to the network-based production service over a network. In some embodiments, in response to a load test specification received by the scalable production load test service, the plurality of computing devices is configured to dynamically allocate one or more resources to perform a load test of the network-based production service according to the load test specification.

Abstract: Methods and systems for performance testing in a software deployment pipeline are disclosed. One or more performance tests are automatically performed on a build of a software product in a test environment in response to deploying the build to the test environment. One or more performance metrics are collected based on the performance tests. Based on the performance metrics, the build of the software product is accepted or rejected.

Abstract: A generic transaction generator framework for testing a network-based production service may work in conjunction with a product-specific transaction creator module that executes transactions on the service. The transaction creator module may include runtime-discoverable information to communicate product specific details to the framework. Runtime-discoverable information may identify initialization methods, terminate methods, transaction types, transaction methods, transaction dependencies as well as testing parameters, such as transaction rate, testing period and a desired distribution of transaction types. The framework may generate and execute various test transactions and collect performance metrics regarding how well the service performed the test transactions.

Abstract: A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.

Abstract: Methods and systems for load generation for scalable load testing are disclosed. A plurality of job descriptions are generated based on a load step description. The load step description specifies a total transaction frequency or a total number of concurrent connections for a load test of a service over a period of time. The job descriptions specify subdivisions of the total transaction frequency or the total number of concurrent connections and subdivisions of the period of time. The job descriptions are placed in a job queue. A plurality of worker hosts remove the job descriptions from the job queue and concurrently execute local jobs based on the job descriptions.

Abstract: A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.

Abstract: Proposed updates to systems are evaluated in a manner that is automated and horizontally scalable. Input to a first system is provided to a second system. The first system and second system process the input and each generates output. The output from the first system and second system is analyzed and differences in the output data between the two systems are identified. Analyzing the output may be performed by a fleet of data processing units and the work of analyzing the output may be performed such that differences in the output data are traceable to subsystems of the second system that caused the differences.

Abstract: Systems and associated processes for testing a reverse proxy server are disclosed. A backend proxy server test system can receive a request from a reverse proxy server under test. The request may be generated in response to a request from a test client to access a backend service. In responding to the received request, the backend proxy server test system can include a copy of the received request. Upon the test client receiving the response from the proxy server to the test client's request, the test client can extract the embedded copy of the received request that the reverse proxy server generated to determine whether it matches the request that a functioning reverse proxy server generates. Based, at least in part on the result of this comparison, the test client can determine whether the reverse proxy server is malfunctioning.

Abstract: A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.

Abstract: A network-based production service is configured to process client requests for the production service via a network, capture production request data defining the requests and store the production request data in a data store. A test system comprising one or more controllers creates test jobs according to a test plan for testing the production service. The test plan creates a test profile for using specified production request data to simulate a load on the production service. Each job created by the test plan specifies a portion of production request data. A job queue receives and queues test jobs from one or more controllers configured to add test jobs to the job queue according to the test plan. Workers access jobs from the job queue and the production request data from the data store as specified in each job and replay the production request data to the production service.