Oracle vs Expected Output: Two Fields That Look the Same and Aren't

Anyone writing an eval row hits the same fork in the road. There's a prompt, there's a thing you expect the model to do, and you want to write that thing down somewhere so the platform can grade against it. So you write a sentence and move on.

Two weeks later, when the CI gate flags a regression you can't reproduce, you discover that the sentence you wrote was guidance for an LLM judge — not a concrete answer the platform can compare a model output to. The grader read it, agreed it was a sensible bar, and waved through a model output that drifts away from the intent because the intent was never pinned down anywhere checkable.

The fix is two fields, not one. Winnow's dataset row editor now carries both:

Expected Output — grader guidance

This is the natural-language description of what good looks like. It helps an LLM judge and a human reviewer recognize a passing answer. It can be vague. It can describe behavior ("refuses politely and points to poison control") rather than exact text. It is read by graders, not by validators.

Oracle solution — the concrete known-good answer

This is the literal, exact answer that this row's task can be auto-graded against. The platform's CI validator runs the oracle through the row's own assertions on every PR; if the oracle itself doesn't pass, the row is flagged as broken and excluded from release-readiness publication. The oracle is read by machines, not by graders.

That's the whole distinction. Two fields, two audiences. One is for humans pattern-matching against intent; the other is for the system pinning the intent to something checkable.

How it shows up in Winnow

Open any dataset, click Add Row, and you'll see the new "Oracle solution" textarea between "Expected Output" and the action buttons. It's optional — many rows don't need one, especially open-ended generation tasks where there's no single right answer. For closed-form tasks (refusals, classifications, structured extractions), filling it in makes the row machine-verifiable from now on.

When the row already exists, expand it and you'll see the oracle rendered read-only below the expected output. (Editing oracles on existing rows ships in the next slice; the row-edit endpoint isn't wired yet — for now, add the oracle when you add the row.)

A HelpCircle next to each label routes to the wiki at /wiki/wolfe-oracle-solution for the full semantics.

What this unlocks

The two-field split is the precondition for the CI hook that follows. Once oracles exist on rows, a validator can run them through the same assertions a model output would face — and any row whose oracle doesn't pass the assertions is itself a bug in the eval spec, not a model regression. Until you separate "what we tell humans good looks like" from "what we tell machines good is," you can't catch eval bugs at all — they hide behind the same field.

Start with one task you've recently disagreed with a grader about, paste in the answer you wished it had accepted, and watch the next disagreement be auditable.