The Harness Pill: Decomposing a Scorecard Without Leaving the Page

When an agent eval regresses, the first question is "where did it come from?" The second question is "which of the dozen things I changed this week?" The honest answer is usually "I don't know yet — let me cross-reference the scorecard against the scaffold timeline against the model registry in three browser tabs and a notebook."

The Scorecard Overview tab is supposed to be the place where the first two questions get answered without the notebook. To do that, it needs to decompose itself along whichever axis is interesting today: the arm (control vs treatment), the model (gpt-4o-mini vs claude-3.5-sonnet), the harness (react-default vs planner-only). Winnow's BreakdownToggle — the small three-pill control above the metrics table — has shipped the UI for a week. Today's release makes the harness pill actually do something.

Where it lives

The toggle sits in the top-right of the Scorecard Overview tab, just below the tab strip. Three pills, in order: Arm (default), Model, Harness. Clicking any active pill regroups the table below it.

There is no new tab, no new modal, no settings page. The chart-or-table area that used to render a control-vs-treatment comparison now renders one row per (metric, bucket) when a non-arm pill is selected. Bucket = gpt-4o-mini / claude-3.5-sonnet / etc when the axis is Model; bucket = react-default@v1.2 / planner-only@v1.0 / etc when the axis is Harness.

When the axis snaps, the help-circle next to the toggle label routes to /wiki/wolfe-breakdown-toggle and explains the three-axis decomposition pattern, with a longer treatment of the harness concept on /wiki/wolfe-harness.

Why the Harness pill is the load-bearing one

The Arm pill answers "did the change in prompt or routing or selection logic move the metric?" That's the classic A/B-test framing. It's necessary but rarely sufficient for agent evals, because agents don't ship prompt changes in isolation — they ship scaffold changes alongside them. A new tool gets added. A retry policy gets tightened. A summarization step gets folded in. The scaffold (the harness) is often the bigger lever than the prompt.

The Model pill is the obvious second decomposition: a single arm often calls more than one model. The control arm might route 80% of traffic to gpt-4o-mini and 20% to gpt-4o for cost/quality tradeoff. When the scorecard regresses, "which model dragged the mean down?" is a real question — and it's a question the per-arm view literally cannot answer.

The Harness pill is the load-bearing one because most regressions in production agent suites come from harness drift, not prompt drift. A new tool that returns 2K-token verbatim into context. A summarization step that subtly loses fidelity. A retry loop that masks tool-call failures. The harness pill is what makes those regressions visible in the scorecard surface, without requiring a notebook.

Honesty point — the backend rollup is deferred

The pill ships today as discoverable but disabled on most scorecards. The reason: the /api/v1/experiments/{id}/scorecard route in api/routes/stats.py aggregates from per-arm metric_aggregates (sum / sum_sq / count) and returns a control_mean / treatment_mean block per metric. There is no per-model or per-harness aggregator on the route today, and no ?break_by=... query parameter.

The storage layer has what it needs. The trace_metadata table records harness_id and harness_version per trace (shipped 2026-05-20 with the Tier-S S3 backend), and the observation rows carry the model id in their metadata JSON. The missing piece is the join: observations → traces → trace_metadata, grouped per (metric, harness_id@version). That join is the deferred backend slice tracked in docs/UNBUILT-FEATURES.md under the per-harness rollup entry and its per-model sibling.

When the rollup lands, the pill enables itself the moment the scorecard payload carries harness_rollups or model_rollups — the frontend reads them defensively, treats missing as disabled, and snaps the toggle back to Arm if the user lands with a saved state pointing at an axis that turns out to be unavailable in the current payload. The wiring is forward-compatible: a backend-only PR ships the join, no frontend change required.

What's in the box today

Three small changes:

1. BreakdownToggle accepts disabledOptions. Per-axis disabled state with a tooltip reason — the only new chrome on the pill itself. Disabled pills carry aria-disabled="true" for screen readers.
2. OverviewTab actually swaps tables on axis change. The legacy 5-column control-vs-treatment view (OverviewArmTable) for the Arm axis; a new OverviewBreakdownTable (Metric / axis / Mean / 95% CI / n) for Model and Harness. The swap is a clean if/else inside the same tab.
3. Typed BreakdownRollup shape. ScorecardMetric grows optional model_rollups: Record<string, BreakdownRollup> and harness_rollups: Record<string, BreakdownRollup> fields. Absent today; non-breaking when the backend adds them.

The discipline (CLAUDE.md core principle: minimize surface area) is the same across this suite: 0 new routes, 0 new tabs, 0 new modals, 0 new settings toggles, 0 new sidebar entries. The pill already exists; this PR makes it effective on one axis (Arm) and discoverable on the other two until the backend catches up.

What's next

The two UNBUILT-FEATURES.md rows above are the natural follow-ups. The per-harness rollup is the higher-value of the two (most agent-eval regressions come from harness changes, and the storage join key is already in place). The per-model rollup is the simpler of the two (no traces ↔ observations join needed — the model id lives directly on the observation row).

When both ship, the harness pill goes from "discoverable but disabled" to "the first thing you click when a scorecard regresses." Until then: flip the pill, read the deferral, and know that the answer is one backend slice away.