Optimization Agent
Let AI Optimize Your AI
An autonomous agent that continuously tunes your prompts, parameters, and model selection. Choose from three autonomy modes — Observer, Supervised, or Autonomous — with auto-rollback safety nets and declarative playbooks.
import winnow
# Configure the optimization agent
agent = winnow.OptimizationAgent(
project="search-ranking",
autonomy_mode="supervised", # observer | supervised | autonomous
objectives=["quality_score", "latency_p95", "cost_per_query"],
guardrails=winnow.GuardrailPreset.MODERATE,
playbooks=[
winnow.Playbook(
when="quality_score < 0.85 for 15 minutes",
then="rollback to last known good",
never="deploy during peak hours (9am-11am UTC)",
),
],
)
agent.start() # begins optimization loopAutonomous, Not Reckless
Full control over what the agent can change, when it can act, and how far it can go.
Three Autonomy Modes
Observer watches and surfaces diagnoses. Supervised generates candidates and waits for your approval. Autonomous executes approved categories of changes on its own. Every mode keeps the Experiment Journal up-to-date.
Auto-Rollback on Guardrail Breach
When any guardrail fires, the agent instantly reverts to the last known-good configuration. No human intervention required, zero downtime.
OPRO Prompt Optimization
Uses the OPRO (Optimization by PROmpting) technique to iteratively improve your prompts. The agent generates, evaluates, and selects better prompt variants automatically.
Bayesian Parameter Search
Efficiently explores temperature, top-p, frequency penalty, and other parameters using Bayesian optimization with Gaussian process surrogate models.
Playbooks (WHEN / THEN / NEVER)
Define operational rules in plain English. WHEN conditions are met, THEN take action. NEVER rules create hard boundaries the agent cannot cross.
Experiment Journal
Every autonomous change, on the record.
The optimization agent keeps a persistent Experiment Journal — what it proposed, why, the signal that triggered it, and what happened. Sorted best-first, diffable, queryable. Autonomy you can audit, not a black box.
Proposer provenance
Every journal entry names the proposer — OPRO, Bayesian, model-swap, or portfolio policy — so you know which optimization strategy made each call.
Rationale + refutation
Each proposal includes a rationale and a could_be_wrong_if clause — the exact signal that would flip the conclusion. Auditable whether a human approved it or not.
Outcome tracking
Journal entries record whether the proposal produced a win, no improvement, or a rollback — building a track record the agent uses to calibrate future proposals.
Change-risk approval queue
You stay in the loop, ranked by risk.
Every proposal gets a change-risk score. Low-risk prompt tweaks can auto-ship within your guardrails. A model swap or parameter change with high impact waits for a human. The queue is sorted so the changes that need eyes are at the top.
- Risk scored 0–1 per proposal; threshold configurable per workspace
- Approve, reject, or defer — rejections teach the agent what to avoid
- Autonomous mode auto-executes below the risk threshold; above it, always waits
One proposer portfolio, four strategies
Cost × quality, live
The number the allocator optimizes is quality per dollar.
Winnow treats the inference budget k (how many samples you draw per query) as a first-class action axis, right alongside model and prompt. The bandit optimizes cost-adjusted reward on your traffic, where several cheap samples routinely beat one expensive one.
| arm (model, k) | pass@k | $ / correct | traffic |
|---|---|---|---|
| 70B · k=1 | 0.82 ±.03 | $0.041 | 18% |
| 8B overtrained · k=5 | 0.84 ±.03 | $0.017 | 61% ▲ |
| 8B · k=1 | 0.71 ±.04 | $0.006 | 14% |
| 70B · k=3 | 0.86 ±.02 | $0.121 | 7% |
The allocator is shifting traffic to 8B @ k=5, statistically tied on quality with the frontier arm, at 2.4× lower cost per correct answer. Promotion stays human/MAB-gated; the agent never ships autonomously.
Δpass@k / Δcost falls off a cliff after k=4 on this task: the agent proposes (model, k) pairs, the MAB resolves the winner, and a hard per-query compute ceiling keeps a quality win from breaking the budget.
Research-backed
Why does your frontier model lose on cost per correct answer?
Roberts et al. (2026) introduced Train-to-Test scaling laws: once inference cost is properly accounted for, small overtrained models doing repeated sampling often outperform frontier models at lower per-query cost. That is the prior. Winnow is the layer that validates the prior against your production traffic, under real cost and reward signals, with statistical rigor.
Sample budget as an action axis
The agent proposes (model, k) pairs, for example "3B overtrained, k=5" versus "70B, k=1". The MAB resolves which wins on actual traffic.
Cost-adjusted reward by default
For agentic workloads the allocator optimizes quality per dollar, not raw quality. Several samples from a smaller model can beat one sample from a larger one on the number that matters.
Compute budget guardrails
Per-query and portfolio-level spend ceilings prevent the agent from selecting variants that improve quality but break your inference budget. Hard caps, not soft suggestions.
Reference: Roberts, N. et al. (2026). Test-Time Scaling Makes Overtraining Compute-Optimal. arXiv:2604.01411. University of Wisconsin-Madison and Stanford University.
Six workflow templates for AI optimization
The same lifecycle coverage we ship for experimentation, retargeted at cost, sampling, latency, and quality drift. Click any card to expand setup guidance.
Cost Per Correct Answer
[?]Minimize (cost_per_call × k) / pass@k at a fixed quality floor. Grid over (model, k).
metric: cost_per_correct_answerstop: mSPRTClick to expand setup guidance ↓
Cost Per Correct Answer
[?]Your model bill is growing faster than your traffic, or a new cheaper model just dropped and you want to know whether the economics work on your workload.
Define a quality floor on pass@k. Supply the (model, k) grid. Point at a verifier or reference answers. Reuse an existing eval suite where possible.
Verifier drift mid-experiment makes cost-per-correct-answer misleading. Cheap models also lose asymmetrically on the tail, so stratify quality by difficulty before declaring a winner.
Pass@k Knee Finder
[?]Sweep k ∈ {1, 2, 4, 8, 16} on one model. Find the smallest k where Δpass@k / Δcost drops below your floor.
metric: marginal pass@kstop: Knee confirmedClick to expand setup guidance ↓
Pass@k Knee Finder
[?]First rollout of multi-sample inference on a task, or after swapping the verifier, changing prompts, or moving to a new model family.
Pick the model. Pick your test set. Set τ as the quality-per-dollar floor (a natural starting value is your current config's pass@k / cost).
Parallel sampling is cheap on wall-clock; sequential sampling is not. Cap k_sequential separately if you have tight SLAs.
Latency vs Accuracy Frontier
[?]Trace the Pareto front over k_parallel and k_sequential so you pick the sampling strategy your SLA allows.
metric: pass@k at p95 ≤ SLAstop: Stable frontierClick to expand setup guidance ↓
Latency vs Accuracy Frontier
[?]Your p95 latency is creeping up as you raise k, or you need to prove an accuracy target is reachable without blowing the latency SLA.
Define p95 and p99 ceilings. Configure both k_parallel and k_sequential. Pick an accuracy target. The template maps the frontier across runs.
Rate-limit throttling from the provider inflates latency under parallel fan-out. Validate at peak load, not off-hours.
Verifier Strength Audit
[?]Compare pass@k with and without the verifier. The delta tells you whether to spend on k or on a better verifier.
metric: Δpass@k from selectionstop: Fixed horizonClick to expand setup guidance ↓
Verifier Strength Audit
[?]Sampling lift is weaker than the literature suggests, or you are planning to swap the reward model or grader.
Run identical queries twice: once with verifier-based selection, once with a baseline (random or first-sample). Report the gap on a small held-out set.
A flat delta is the verifier failing, not k failing. No amount of sampling closes the ROC gap; fix the verifier first.
Model Swap Under Budget
[?]Smaller model at higher k versus incumbent at k=1. The T² recommendation shipped as an A/B you can deploy.
metric: cost_per_correct_answerstop: mSPRT, 14 daysClick to expand setup guidance ↓
Model Swap Under Budget
[?]A new open-weight model looks promising, or your frontier-model bill is unsustainable and you want to test the scaling-law prediction on your traffic.
Incumbent is control at k=1. Candidate runs at k values predicted to be cost-neutral (Knee Finder output is a good starting point). Verifier must be identical across arms.
Smaller models often fail on the long tail. Stratify by difficulty bucket. Aggregate wins that hide bottom-decile collapse will regret in production.
Drift-Corrected Sampling
[?]Sentinel queries plus CUSUM on pass@k. When the control chart signals, rerun the knee finder.
metric: pass@k driftstop: Always onClick to expand setup guidance ↓
Drift-Corrected Sampling
[?]Always. This is the one template that runs continuously. It catches input-distribution shifts that move the optimal k without touching the model or prompt.
Define a representative sentinel set with known answers. Schedule runs (hourly or daily). Apply CUSUM to rolling pass@k. Auto-trigger the Knee Finder when the chart alarms.
Sentinel staleness causes false alarms (or missed ones). Rotate the sentinel set quarterly or use a longer moving window.
Full lifecycle walkthrough with instrumentation schema and sample-index tracking in How many times should your model answer?
Beyond manual prompt engineering
Most teams still tune prompts by hand in a notebook and hope for the best. The Optimization Agent runs continuous experiments in production, using real user data and your evaluation criteria to find improvements you would never discover manually.
Put your optimization on autopilot
Start with Observer mode. Promote to Full Auto when you are ready. Free tier includes 1,000 optimization cycles per month.
Start Free