Stage 3: Adversarial Probe, Not Optional

Most task-quality bugs are not "the agent is wrong." They are "the assertion the author wrote accepts a bad answer." A vacuous refusal ("I cannot help with that") passes any output_contains: "cannot" check. A passive deflection ("financial decisions depend on your individual situation") passes any output_does_not_contain: "buy" check. Neither response actually solves the task — and neither would fail your assertions.

This is what an exploit probe catches.

What it is

Stage 3 of Winnow's task QA flow runs an adversarial agent against your assertions. It's not the model under test. It's a deterministic bad actor that tries N pre-defined strategies (vacuous refusal, passive deflection, policy quote, hallucinated disclaimer, format compliance, refusal with prefix injection) and reports which ones passed your assertions despite not solving the task.

If any strategy passes, the task is exploitable. The assertion is loose. The task is held until you tighten.

Where it lives

The exploit probe results live on a new QA tab on every dataset's detail page — Winnow's "tasks" are dataset rows, and the QA tab is the per-dataset adversarial review surface. It sits as the 5th tab, after Rows / Versions / Import & Export / Capture Rules.

The tab body is three cards in a row, top-to-bottom:

1. Stage 1 — Dummy probe + oracle. Does every row have a known-good oracle answer that passes its assertions, and does an empty agent fail them? (Catches assertions that accept literally any input.)
2. Stage 2 — Cascade rubric over spec. Is the task spec unambiguous and the criteria two-sided? (Catches assertions that reward one direction without penalizing the other.)
3. Stage 3 — Adversarial exploit probe. The headline. N strategies, deterministic, run against the assertions. Each pass is a flag.

Below the cards: the exploit detail panel. Each exploited probe expands inline to show every strategy attempt with its rationale ("strategy vacuous_refusal passed your assertion output_contains: cannot"). Clean probes don't get a row — they get a single-line summary at the bottom.

Approve & publish at the bottom of the tab is disabled until stage 3 is green or an author-only override is recorded.

Why deterministic strategies

The temptation with adversarial work is to plug in an LLM and ask it to generate bypasses. Winnow's first slice ships deterministic strategies — six named, fixed bad-actor agents that the author can read in source (agent/exploit_agent.py::DEFAULT_STRATEGIES).

Three reasons:

1. Reproducibility. A deterministic probe gives the same result every CI run. An LLM-generated probe drifts with the model; one day the assertion is exploitable, the next it isn't, with no code change.
2. Auditability. The author needs to know which strategy passed, not "an LLM thought of something." Deterministic strategies have names and rationales the author can read.
3. Coverage of the common mistakes. Most assertion-design bugs are not exotic. They are the same six patterns over and over: someone wrote output_contains: "cannot" and a one-word refusal slipped through. The deterministic set catches these without needing an LLM call.

LLM-driven adversarial strategies are deferred, not denied — they're a separate lane.

Honesty about what isn't built yet

Three follow-ups are tracked in docs/UNBUILT-FEATURES.md P2:

• Per-row stage_1/stage_2 backend. Stage 3 reads real exploit_probe_results rows from the backend. Stages 1 + 2 are derived from row state today (oracle + expected output presence; graded coverage) — the in-tab footer is explicit about this. Typed columns + a task_qa_stage_results table land in a follow-up.
• In-app probe trigger. The exploit-probe POST /scoring/exploit-probe/run route shipped 2026-05-21 as a CI hook. Triggering from the UI needs a small assertion-spec form (prompt + accept_patterns + optional strategy subset) — deferred.
• "Tighten assertion" jump. The mockup's Step 7 (click "Tighten assertion" → land on the row editor scrolled to the failing assertion) requires a per-row assertion editor surface that dataset rows don't have today. Deferred until the assertion-editor lane lands.

The 90-day author-only visibility window is enforced server-side; this UI surfaces the gating but doesn't yet honor a viewer_can_see_exploit flag for non-author viewers. That's also tracked as a deferred row.

The probe is the highest-leverage check Winnow can run on a task — not because it tests the agent, but because it tests the test.