Abstract: Techniques for calculating and using a request cost index for throttling application programming interface (API) requests to execute operations in a provider network. The techniques encompass the step receiving a request to execute an operation at an API service in the provider network. Further steps include determining to execute the operation based on a request rate limiting algorithm, executing the operation to yield an operation result, and sending the operation result. Additional steps include calculating a request cost index that reflects an amount of computing resources utilized by executing the operation, determining an adjustment amount for a state variable of the request rate limiting algorithm based on the calculated request cost index, and adjusting (e.g., lowering) the state variable by the adjustment amount. Other (e.g., subsequent) requests to execute queries received at the API service that are metered by the state variable can be throttled by the API service.

Abstract: Fair queuing of request tasks spawned by requests to execute generative operations such as, for example, graph query language requests to execute a graph query language query, mutation, or subscription operations. Queuing techniques are used to prevent a heavy generative operation from dominating usage of computing resources of a host that executes many generative operations concurrently including a mix of heavy and normal generative operations. Generative operations are analyzed and classified as heavy or normal as the request tasks they spawn are being executed. If a generative operation is classified as heavy, then subsequent request tasks spawned by the heavy generative operation are added to an overload queue while request tasks spawned by concurrently executing normal generative operations as added to a main queue. For fairness, request tasks are polled from the main queue for execution at greater frequency than request tasks in the overload queue.