Abstract: In a service provider environment, software deployments can be implemented by automatically routing an incrementally larger percentage of requests to a new software stack based on a success rate for previous requests. Thus, new and old versions of the software stack can be executed simultaneously with a small portion being routed to the new version until a sufficient number of requests have been handled. By slowly bringing a new software stack online, a confidence level in the new deployment can increase until it is fully deployed. Different success rates and different ramping rates can be used for each customer or based on different components of the request, such as API name, API parameters, location, etc. The software deployments can be extended to web-page requests.

Abstract: A distributed system may implement dynamic specification auditing. A specification for a distributed system may be maintained in a machine readable format. Specification assertion agents corresponding to different components of the distributed system may request and receive the specification in order to verify whether tasks performed by the respective component satisfy the specification. The specification assertion agents may then send assertions of the success or failure of the component to be stored in order to verify compliance with specification. Various reports may be generated which identify failures or components in the distributed system that are not reporting assertions.