When the Judge Drifts: Calibration Monitoring for LLM Evaluators

Every team that runs evals at scale eventually stops grading by hand and starts using an LLM as the judge. It is faster, it is cheaper, and on the day you set it up, it lines up with your human reviewers well enough to trust. So you ship it. The dashboard turns green. You move on.

Six weeks later, your top-line "judge agrees with humans" number on the eval scorecard is still 0.82 Cohen's kappa. Everything looks fine. Except the rubric definition shifted in week three when product redefined what counts as "helpful," and your judge model got a silent provider update in week four, and your human labellers — who were on rotation — have been swapped twice.

The dashboard is still green because nobody is re-measuring. The judge has drifted.

What "calibration drift" actually means

Calibration drift is not "the model under test got worse." That is a different signal, and it shows up in the scorecard's primary metric.

Calibration drift is the judge's agreement with humans got worse. Same inputs, same outputs, same scoring criterion — but the verdicts now diverge from what a careful human reviewer would say.

Three things can cause it, often together:

1. The rubric definition shifted. Three weeks ago "PII" did not include indirect quasi-identifiers; today it does. Your judge prompt still uses the old definition.
2. The judge model changed. Anthropic, OpenAI, or your fine-tuner pushed an update. The same prompt produces subtly different verdicts.
3. The human distribution shifted. Your reviewers rotated. The new reviewers grade stricter (or looser) on the same examples.

You cannot tell which one happened without a stored baseline to compare against.

The baseline pattern

Winnow's calibration surface stores one baseline per (suite_id, dimension) pair — the Pearson r, MAE, and Cohen's kappa computed from a labelled sample at a moment you trust the judge. Future checks compute the same three metrics on a fresh labelled sample and compare to the baseline.

A drop past a configurable threshold flags the dimension as drifted.

curl -sS -X POST http://localhost:8000/api/v1/scoring/calibration-baselines \
  -H 'content-type: application/json' \
  -d '{
    "suite_id":"support-quality",
    "dimension":"helpfulness",
    "human_scores":[1.0,0.0,1.0,1.0,0.0,1.0,1.0,0.0,1.0,0.0],
    "judge_scores":[1.0,0.0,1.0,0.0,0.0,1.0,1.0,0.0,1.0,1.0],
    "window_days":7
  }'

A week later, collect a new labelled batch and call /check:

curl -sS -X POST \
  http://localhost:8000/api/v1/scoring/calibration-baselines/support-quality/helpfulness/check \
  -H 'content-type: application/json' \
  -d '{
    "human_scores":[1.0,0.0,1.0,1.0,0.0],
    "judge_scores":[0.0,1.0,0.0,0.0,1.0]
  }'

The response is one of three verdicts:

• stable — every metric is within tolerance. Keep moving.
• drifted — at least one of κ, Pearson r, or MAE breached. The reasons array names which.
• no_baseline — no baseline persisted for this pair yet. Establish one before checking.

Surfaced on the Scorecard

A baseline you cannot see is a baseline you will not maintain. Winnow surfaces the drift state inline on the Scorecard → Dimensions tab — one pill per criterion, the same row where you read the judge's mean score.

The pill is small on purpose. Green "stable" pills fade into the background; that is correct, because the absence of drift is not news. Red "drifted" pills pulse, because that is news. Grey "no baseline" pills sit between — a soft reminder that monitoring is not yet enabled for this dimension, not an alarm.

Clicking any pill expands an inline detail panel showing all three metrics (current vs baseline), the sample size, the reasons each threshold tripped, and three actions:

• Recompute now. Re-run the check with a fresh labelled batch.
• Establish new baseline. Replace the stored baseline with a current trusted sample. Use this after you have intentionally updated the rubric.
• Edit criterion prompt. Jump to the EvalBuilder for the underlying rubric.

Why the pill semantics are honest

A pill that says "stable" when no fresh check has run would be lying. Winnow's pill is honest about what it knows: if a baseline exists, it shows "stable" until a check_drift POST returns a different verdict. The "drifted" state only appears when the platform has actually measured drift, not when it is guessing.

That is also why the detail panel exposes paste boxes for human and judge scores rather than auto-joining. The platform does not own your labelled review queue. You do. The cookbook walks through wiring a small cron job that pulls labels from your review system and calls /check on a schedule.

When to re-baseline

Re-baseline when you intentionally change the rubric, the judge model, or the labelling rubric. Do not re-baseline to silence a red pill — that defeats the entire point. The signal you would lose is the one that tells you a regression in the eval pipeline is hiding a regression in the model.

If you find yourself re-baselining frequently, consider tightening the threshold or adding human-in-the-loop review on flagged dimensions. The thresholds default to kappa < baseline - 0.1, pearson < baseline - 0.1, mae > baseline + 0.1. For high-stakes safety dimensions, tighten to 0.05.

What this PR did and did not ship

This shipped the on-platform UI — the drift column on the Dimensions tab, the inline detail panel, and the three action buttons. The math (Pearson r, MAE, Cohen's kappa) and the storage layer (calibration_baselines table) shipped earlier as the A5 backend in [PR #X]. The cookbook recipe at docs/cookbook/judge-calibration-drift.md documents the API workflow end-to-end.

Two follow-ups are tracked in docs/UNBUILT-FEATURES.md:

• A GitHub PR-comment that posts when a baseline changes (extends scoring/pr_comment_*.py). The intent is to make rubric edits visible to the team in the same place they review code.
• A cross-suite Calibration Center (one screen showing every drifted dimension across every suite), which is emergent enough to be its own lane.

See also

• The cookbook: docs/cookbook/judge-calibration-drift.md
• The math: scoring/calibration_drift.py
• The wiki entry: /wiki/wolfe-judge-calibration-drift
• The design doc: docs/wolfe-effort-unified-design.md §3.1 (Flow A)