Model Drift Detection

Model drift is when an AI feature that worked well starts performing worse — and no one notices until users complain. It happens silently: the model doesn't announce when it's struggling with new input patterns, when your data distribution shifts, or when an upstream model update changes behaviour. This skill builds the detection system that catches drift before users do.

---

Context

The three types of drift (each requires a different detection approach):

Type	What it is	Example	How to detect
Data drift	Input distribution shifts — users are sending different kinds of queries than before	A product-search AI trained on short queries, now receiving long conversational ones	Statistical comparison of input features over time
Concept drift	The correct answer to the same input has changed	A news classifier whose categories are now outdated	Track accuracy against a human-labelled gold set
Model drift	The model itself changes — due to provider updates, fine-tune decay, or version changes	OpenAI silently updates GPT-4o behaviour	Monitor output distribution and quality metrics for sudden changes

Why drift is a PM problem, not just an engineering problem:

Drift has product consequences. Quality drops lead to user churn before they show up in support tickets.

---

Step 1 — Identify drift risk for the feature

Assess: model source, input type, output type, ground truth availability, provider update risk. Output a drift risk profile with primary drift type to monitor and risk level.

Step 2 — Choose the drift detection methods

Four methods:

Output distribution monitoring — track statistical properties of outputs over time

Quality metric monitoring — track acceptance rate, CSAT, accuracy, re-query rate with amber/red thresholds

Canary test set monitoring — maintain 20–50 fixed test inputs with known expected outputs, run regularly

Input distribution monitoring — track input length, language, topic distribution over time

Step 3 — Build the drift response playbook

Define amber alert (48-hour investigation) and red alert (same-day response) procedures with specific mitigation options: revert model version, add prompt-level compensating instructions, route to fallback model, or increase human review rate.

Step 4 — Define the monitoring schedule

Daily automated checks, weekly PM review, on every model/prompt change, and on provider changes.

Quality check before delivering

Drift risk profile includes the primary drift type to monitor

Canary test set is defined — not "we'll add canaries later"

Alert thresholds have numeric values

Response playbook has a defined owner and timeframe

Model provider update risk is addressed

Post-drift review process is included

Suggested next step: Build the canary test set this week — not after the first drift incident. Pull the top 20 queries from last week's production traffic and write the expected outputs now, while the model is performing well.