Abstract: Methods, systems, and computer-readable storage media for receiving, by a request processing engine of the ML model runtime, an inference request associated with a version model (VM), and determining, by the request processing engine that a VM-specific token and a global token are available for the inference request, the VM-specific token being available from a token pool that is specific to the VM, and in response: selecting a VM process (VMP) in a set of VMPs for execution of the inference request, the VMP being executed by a processor that is different from one or more processors executing one or more other VMPs in the set of VMPs, each VMP in the set of VMPs being specific to the VM and being designated for execution by a respective processor by a respective affinity setting, and providing the inference request to the VMP for execution.