Abstract: Techniques for phased deployment of machine learning models are described. Customers can call a training API to initiate model training, but then must wait while the training completes before the model can be used to perform inference. Depending on the type of model, machine learning algorithm being used for training, size of the training dataset, etc. this training process may take hours or days to complete. This leads to significant downtime where inference requests cannot be served. Embodiments improve upon existing systems by providing phased deployment of custom models. For example, a simple, less accurate model, can be provided synchronously in response to a request for a custom model. At the same time, one or more machine learning models can be trained asynchronously in the background. When the machine learning model is ready for use, the customers' traffic and jobs can be transferred over to the better model.

Abstract: A machine learning-based service processes requests using compressed and complete models to provide faster response times for servicing requests to process data. Initially, a host processes data using a compressed model that is stored in the host's memory and then switches to a larger, more accurate complete model after it is loaded into the host's memory. A host of the machine-learning based service may receive one or more requests to process data. In response, the host uses a compressed version of a model to begin processing the data. The host starts loading the complete version of the model into the host's memory. When the complete version of the model is loaded into memory, the host switches to process a remaining portion of the data using the complete version of the model.