Abstract: The subject matter of this disclosure relates to systems and methods for monitoring and managing machine learning models and related data. Histogram structures can be used to aggregate streams of numerical data for storage and metric calculations. Drift in such data can be identified and monitored over time. When significant drift is detected and/or when model accuracy has deteriorated, models can be automatically refreshed with updated training data and/or replaced with one or more other models. A model controller is used to automate model monitoring and management activities across multiple prediction environments where models are deployed and prediction jobs are executed.