Model monitoring and drift: catching decay before your users do
Models degrade quietly. The signals that move first, and how to alert on them without drowning in noise.
Software fails loudly. Models fail quietly — accuracy slides a few points a month while every dashboard stays green, and you find out from a complaint.
Three things drift, and they need different responses
Input drift is your data changing shape. Prediction drift is your outputs changing distribution. Concept drift is the relationship itself changing — the same input should now produce a different answer. Only the third genuinely requires retraining, and conflating them wastes a lot of effort.
Three kinds of drift
Input drift
- What changed
- Your incoming data changed shape
- What to do
- Investigate upstream — often a pipeline change
Prediction drift
- What changed
- Your outputs shifted distribution
- What to do
- Symptom — trace it to input or concept
Concept drift
- What changed
- The relationship itself changed
- What to do
- The one that genuinely needs retraining
| What changed | What to do | |
|---|---|---|
| Input drift | Your incoming data changed shape | Investigate upstream — often a pipeline change |
| Prediction drift | Your outputs shifted distribution | Symptom — trace it to input or concept |
| Concept drift | The relationship itself changed | The one that genuinely needs retraining |
Monitor inputs, because labels arrive late
Ground truth often lands weeks after the prediction, if at all. Input distributions are available immediately, which makes them your early warning system even though they are an indirect signal.
Find the proxy for outcomes
Nearly every system has a behavioural signal that correlates with quality and arrives fast:
- Override rate — how often humans change the model's answer
- Escalation and fallback rate
- Downstream rework — how often a decision is reversed later
- Abandonment part-way through an AI-assisted flow
Decide the response before the alert
- 1
Who investigates
A named owner, not "the team".
- 2
What the rollback is
The previous model, deployable in minutes, tested regularly.
- 3
What triggers retraining
A threshold agreed in advance, not argued about at the time.
- 4
How it is communicated
An alert with no agreed response is a notification.
Alert on sustained change, not noise
Distribution metrics are jumpy. Alerting on daily deltas produces alarms everyone learns to ignore. Compare rolling windows against a stable reference period and require the shift to persist before paging anyone.
Decide the response in advance
Write down now what happens when drift is detected: who investigates, what the rollback is, what the retraining trigger looks like. An alert with no agreed response is a notification, not a control.