The best approach depends on what you're trying to detect. In practice, many production ML systems combine statistical tests with ML-based monitoring because they solve different problems.
Here's a useful framework.
| Goal | Statistical tests | ML-based methods |
|---|
| Detect feature distribution shifts | Excellent | Good |
| Detect changes in feature relationships | Limited | Excellent |
| Detect gradual drift | Good | Excellent |
| Explain which feature changed | Excellent | Moderate |
| Computational cost | Low | Medium to High |
| Ease of deployment | Simple | More complex |
Statistical drift detection
This is the most common starting point because it's easy to implement and interpret.
Common methods include:
- Population Stability Index (PSI): Widely used for tabular features. Values above about 0.2–0.25 often warrant investigation, though thresholds depend on the application.
- Kolmogorov–Smirnov (KS) test: Good for continuous variables.
- Chi-square test: Useful for categorical features.
- Jensen-Shannon divergence or Wasserstein distance: Effective for comparing probability distributions.
- Mean/variance monitoring using control charts for simple operational metrics.
Example:
Training age distribution:
Mean = 42
Std = 11
Production:
Mean = 55
Std = 14
A KS test or PSI would quickly flag this as significant.
ML-based drift detection
These methods learn more complex changes that simple statistical tests may miss.
Examples include:
1. Domain classifier
Train a classifier to distinguish:
- Training samples
- Production samples
If it achieves high accuracy (for example, AUC > 0.8), production data is measurably different.
This works well because it captures interactions among many features.
2. Embedding drift
For image, text, or deep learning systems:
- Generate embeddings
- Compare embedding distributions
- Cluster changes over time
Often much more informative than comparing raw inputs.
3. Reconstruction error
Train an autoencoder on training data.
If reconstruction error increases in production:
Training error: 0.08
Production error: 0.31
The new data likely differs from what the model learned.
4. Density estimation
Estimate
[
P(x)
]
using methods like:
- Gaussian Mixtures
- Normalizing Flows
- Variational Autoencoders
Low likelihood indicates unfamiliar inputs.
Don't monitor only inputs
Input drift does not necessarily mean model performance has degraded.
Monitor at multiple levels:
- Input features
- Model outputs
- Confidence scores
- Ground-truth performance (when labels become available)
- Business metrics
For example:
Feature drift: YES
Prediction distribution: Stable
Accuracy (after labels): Stable
Revenue: Stable
This may indicate harmless seasonal variation rather than a problem.
Alerting strategy
Avoid triggering alerts on every statistically significant change. With large datasets, even tiny differences can produce very small p-values.
A practical alerting strategy combines multiple conditions:
- Minimum effect size (e.g., PSI > 0.2 or Wasserstein distance above a chosen threshold)
- Statistical significance (when applicable)
- Persistence (drift observed for several consecutive monitoring windows)
- Breadth (multiple important features drifting simultaneously)
- Business impact (changes in prediction rates, error rates, or key outcomes)
For example:
| Severity | Example trigger |
|---|
| Info | One low-importance feature exceeds threshold once |
| Warning | Three important features exceed thresholds for three consecutive windows |
| Critical | Drift plus a measurable drop in model performance or business KPIs |
This helps reduce alert fatigue.
Recommended production setup
For most tabular ML systems, a layered monitoring approach works well:
- Feature drift: PSI or KS test for each important feature.
- Multivariate drift: Domain classifier run daily or weekly.
- Prediction monitoring: Track score distributions, class proportions, and confidence.
- Performance monitoring: Compute accuracy, precision/recall, calibration, or regression error once delayed labels arrive.
- Business monitoring: Watch downstream metrics such as conversion rate, fraud rate, or customer churn.
This combination provides interpretable alerts from statistical methods while also catching complex distribution changes that simple univariate tests may miss.
