Replacing N independent LLM-judge criteria with one rubric collapses N HTTP round-trips per trial into one — same per-item κ, ~Nx cost savings, no behavioral regression.
Five Judges in One Call: Rubric-Itemized Evaluation in Winnow
Every team that scales an eval suite eventually hits the same wall. You start with one or two LLM-judge criteria — "helpful," "accurate" — and they fit on a single dashboard tile. Six months later, the same suite has eight criteria, each running its own LLM call per trial. The cost graph is shaped like an exponential and the slowest part of every run is waiting on the judges.
The fix is not to drop criteria. The fix is to stop calling the LLM eight times.
What a rubric is in Winnow
A rubric in Winnow is a list of N items scored together by a single LLM call. Each item carries its own name, definition, pass criterion, and polarity. The judge sees one prompt that bundles all N item definitions, and it returns one structured response with one verdict per item.
Winnow parses that response into N RubricItemResult rows — each with pass, fail, or unknown — and the aggregator reports per-item pass rates plus a single overall score weighted by item.
The math: same per-item fidelity, ~Nx fewer round-trips per trial.
The new card in Eval Builder
Eval Builder now carries five cards on the criterion-type picker: LLM judge, LLM judge (critical), Code assertion, Human, and Rubric (with an A1 NEW pill and a help-circle that deep-links to the wiki).
Selecting the Rubric card expands an inline editor under the same criterion card — no modal, no new tab. You name the rubric, paste a prompt template, and add up to 5 items. Each item gets a name, a passes_if condition, and a polarity (should_occur for desired behaviors, should_not_occur for failure modes like toxicity or hallucination).
On save, the editor PUTs the rubric to /scoring/rubrics/{rubric_id} where the id is a deterministic slug of the criterion's name. Per-trial scoring uses POST /scoring/rubrics/{rubric_id}/judge, which builds the prompt, calls the LLM, parses the structured response, and returns per-item verdicts.
When to use it
Use a rubric when you have ≥3 LLM-judge criteria on the same suite that all read the same trial inputs. The classic shape: a safety suite with helpful, accurate, concise, refuses-medical-advice, and cites-source running as five independent llm_judge criteria. Collapse them into one rubric and you cut the per-trial judge cost by 5x.
Use independent llm_judge criteria when the prompts are genuinely different (one criterion judges the response, another judges a tool-call trajectory) — bundling unrelated checks into a single prompt confuses the judge and degrades per-item kappa.
Honesty: what this ships and what it doesn't
- Ships now: picker card + inline editor + PUT-on-save. The runtime path uses the existing
/judgeroute. The wiki entry at/wiki/wolfe-rubric-judgehas the full data model. - Backend-side: the per-item κ row on the Scorecard → Dimensions tab follows the A5 drift indicator pattern shipped 2026-05-22. Read that post for how calibration drift is detected per item.
- Out of scope: drag-to-reorder items, JSON import, a rubric template library. The 5-item cap is intentional — past 5, you should split into two rubrics or ask whether you have one rubric or two.
If you have a suite running 5+ separate llm_judge criteria today, this is the cheapest win in the platform.