Abstract: Systems and methods for evaluating performance of experimental code using a candidate jobs framework are provided. The techniques may include obtaining an indication of a module under evaluation; configuring a customer environment to provide an evaluation compute, separate from a customer compute executing customer-directed jobs, for executing the module under evaluation; deploying the module under evaluation in the customer environment, wherein deploying the module under evaluation causes the customer environment to execute the module under test using the evaluation compute; configuring the evaluation compute to operate under predetermined test conditions based on a script associated with the module under evaluation; and obtaining an evaluation metric from the customer environment, wherein the evaluation metric is representative of an execution of the module under evaluation based on the predetermined test conditions.

Abstract: Systems and methods for evaluating performance of experimental code using a candidate jobs framework are provided. The techniques may include generating a module in a development environment that include the experimental code. For example, the experimental code may be configured to test an alternate model to a model currently deployed in a customer environment or a module under a predetermined set of test conditions. The techniques may then deploy the module to the customer environments for execution thereat. Deploying the module may cause the customer environments to spin up new compute resources to mitigate the impact on execution of customer jobs. When executing the code, the customer environment may generate evaluation data that is transmitted to an anonymizer for aggregation and/or anonymization. The outputs of the anonymizer may be obtained at the development environment to determine whether to update current models utilized by customers.