How many times should your model answer?

If you run a reasoning-heavy product, you have probably felt the itch already. You picked a frontier model because the benchmarks looked good. You run it once per query because that is the obvious default. Cost climbs. Quality is fine, not great. You wonder whether a smaller model, sampled four times with a verifier picking the best answer, would actually beat it on the bill.

That is not a hunch. It is a scaling law.

Roberts, Cho, Gao et al. (2026), Test-Time Scaling Makes Overtraining Compute-Optimal, formalize the intuition. They jointly optimize three knobs under one budget:

• N: model parameters
• D: training tokens
• k: inference samples per query

Training cost is 6ND. Inference cost per query is 2Nk. Most teams treat these as separate line items. T² treats them as one end-to-end budget and shows the optimum drifts hard once you do.

What the paper does not do is tell you how to translate that into a product. A scaling law gives you a prior. Your production traffic decides the posterior. The rest of this post is the lifecycle that connects the two, and the workflow templates that drop it into practice.

The hypothesis, confirmed

The working frame we want to ship under is: pretraining on a dataset with observability and statistical rigor, then in production validating and optimizing to total cost under a budget. Two pieces of the research land that point directly.

First, scaling-law fitting is itself an observability-and-statistics exercise. T² fits two functional forms against held-out runs. The first is an additive Chinchilla extension on task NLL:

L̂(N, D, k) = E + A/N^α + B/D^β + G/k^γ

The second is a Beta-Binomial pass@k accuracy form that fits per-question success probability directly. Both are parametric models over logged evaluation data. Nothing in the methodology requires you to own pretraining. What it does require is a held-out dataset large enough to fit the parameters stably, which is exactly the instrumentation posture the hypothesis demands.

Second, the paper is explicit that the forecast is a prior, not a verdict. Their Table 1 validation shows overtrained 37M-parameter models outperforming 901M Chinchilla-optimal models on simple reasoning (57.90% vs 18.40%) at matched inference-corrected budget (C_train = 2.56×10¹⁹, C_inf = 2×10⁹ FLOPs). The forecast works. But the forecast is over their eight benchmarks with their verifiers. Your workload has a different input distribution, a different verifier, and a different quality floor. You verify on your traffic, with cost tracked on the same dashboard as quality. That is the second half of the hypothesis, and it is the only way to close the loop.

So: pretraining-style observability gets you the prior. Production validation with live cost accounting gets you the posterior. Confirmed.

The three knobs

Before the lifecycle, a quick map of what is being optimized.

N (parameters)

Every call costs roughly 2N FLOPs. Dropping from 70B to 7B is a 10× inference discount. Also usually a pass@1 quality loss.

D (training tokens)

T² shows the optimum sits far into the overtraining regime compared to Chinchilla's 20 tokens-per-parameter rule. Small heavily-overtrained models hit the joint-optimal sweet spot. If you are not training the model, your proxy is picking a model that someone else overtrained (most recent open-weight releases qualify).

k (inference samples)

The knob most teams never touch. k = 1 is the default. But if the task has a verifier (tests pass, schema matches, answer equals expected), k > 1 with a selection rule is a direct quality lift at a known cost multiplier. Qwen2.5-Math reports RM@256 (256 samples picked by a reward model) as a headline metric, with the 7B-Instruct model solving 21 AIME 2024 problems under reward-model guidance versus 9 in greedy mode. That is a 2.3× improvement from sampling and selection alone.

The joint optimum is not obvious. Self-consistency work shows majority voting needs roughly 18.6 samples on average to match what a confidence-aware selector reaches with 10, a 46% compute reduction from choosing the selection rule better. ROC-n-reroll (Sun et al., 2025) formalizes the ceiling: the achievable accuracy is bounded by the geometry of your verifier's ROC curve, not by k alone.

Translation: you cannot tune k without also measuring how often each sample index wins and how good your verifier is. Which brings us to the lifecycle.

The lifecycle

Six stages. Each one's instrumentation is what makes the next one meaningful.

1. Instrument

Every LLM call logs a row per sample, not per request:

{
  request_id, variant_id, sample_index, k,
  model, prompt_version, temperature,
  output, verifier_score, quality, correct,
  latency_ms, tokens_in, tokens_out, cost_usd,
  user_id, timestamp
}

Two fields earn their keep: sample_index and verifier_score. Without them you cannot compute pass@k, cannot estimate the win-rate-by-index curve, and cannot run any of the templates below. This is the thing to do first and the thing most teams get wrong.

2. Baseline

Pick candidates (model, prompt, k). Run each on a held-out dataset. Record four numbers:

• pass@1 (quality without sampling lift)
• pass@k (quality with your selection rule)
• cost_per_correct_answer = (cost_per_call × k) / pass@k
• Verifier agreement: P(verifier_score > θ | correct) vs P(verifier_score > θ | incorrect)

The last one is your verifier's ROC. If it is flat, more k will not save you; fix the verifier first.

3. Experiment

mSPRT or Bayesian A/B between baseline and challenger. Standard experimentation, with two AI-specific wrinkles:

• The metric is usually a composite (quality minus λ·cost), not a single number.
• LLM variance is higher than most web-A/B variance, so power analysis has to include it.

4. Optimize

A multi-armed bandit (Thompson Sampling is the canonical choice) allocates live traffic to the better candidate, with the optimizer proposing new (model, prompt, k) triples. Reward is cost-adjusted, not raw quality. Raw quality alone pushes the allocator toward k = 16 on the frontier model and breaks the budget.

5. Track which sample wins

This is the stage most teams never instrument, and it is where the sampling question actually gets answered.

For each completed query with k samples:

• winning_index ∈ [0, k) — which sample the verifier picked
• winning_margin — verifier score gap between chosen and runner-up
• would_have_passed_at — smallest k' ≤ k at which the final answer would already be correct

Aggregate over a rolling window:

• win_rate_by_index[i] — fraction of queries where sample i was chosen
• pass@k vs k — the empirical saturation curve
• marginal Δquality / marginal Δcost — the derivative you actually want

Three patterns show up:

• win_rate_by_index[0] ≈ 1.0 and quality high: k = 1 is sufficient. Stop paying for extras.
• Uniform win rate across i: the verifier is uninformative. Either fix the verifier, or raise temperature / use diverse decoding so samples are less correlated.
• pass@k flattens past some k*: classic knee. Set k to the smallest value where Δpass@k / Δcost drops below your quality-per-dollar floor.

Every one of those is a decision. Every one of them requires the logs from stage 1.

6. Budget guardrails

Hard ceilings at two granularities:

• Per-query: the optimizer cannot propose a (model, k) whose expected cost exceeds your per-query budget. No exceptions.
• Portfolio: daily spend ceiling triggers automatic rollback to the cheapest candidate meeting the quality floor.

Soft alerts are useful. Only hard caps prevent the 3 a.m. incident where k = 16 on a frontier model ate the month's budget before lunch.

Six optimization templates

Winnow has six workflow templates for experimentation: Feature Launch, Canary, Prompt Test, Continuous Optimization, Cost Optimization, and Shadow. For the optimization side, the six templates below cover the full lifecycle. Each has a trigger, a target metric, a guardrail, and a stop rule.

1. Cost Per Correct Answer

Minimize (cost_per_call × k) / pass@k at a fixed quality floor. Grid over (model, k). Stop when the best cell's lower confidence bound beats the incumbent's upper bound.

• Trigger: budget overrun, or new cheaper model released.
• Guardrail: hard quality floor on pass@k.
• Stop: mSPRT on cost_per_correct_answer.

When to use. Your model bill is growing faster than your traffic, or a new cheaper model just dropped and you want to know if the economics work on your workload. This is the template that answers "can I pay less per correct answer?"

Setup. Define a quality floor (minimum acceptable pass@k). Supply the candidate grid: models × sampling budgets. Provide a verifier or reference answers so pass@k is computable. Reuse your existing eval suite if you have one.

Watch for. Verifier drift. If your verifier scoring shifts mid-experiment, cost-per-correct-answer becomes misleading. Run a Verifier Strength Audit first if results look surprising. Also: cheap models sometimes lose disproportionately on the tail. Stratify by difficulty before declaring a winner.

2. Pass@k Knee Finder

Fix the model. Sweep k ∈ {1, 2, 4, 8, 16}. Find the smallest k* where Δpass@k / Δcost < τ. This becomes your default k going forward.

• Trigger: first rollout, or verifier replaced.
• Guardrail: latency ceiling on total k × latency_per_call.
• Stop: knee found with confidence, or k = 16 exhausted.

When to use. First time you roll out multi-sample inference on a task, or after swapping the verifier, changing prompt structure, or moving to a new model family. Any time the pass@k curve might have shifted.

Setup. Pick the model. Pick your test set (your eval suite is the natural choice). Set τ, the quality-per-dollar floor, typically the current config's pass@k / cost. The sweep runs as a single batch; no live traffic needed.

Watch for. Latency. Parallel sampling (fan-out) does not hurt wall-clock much. Sequential sampling (retries with prior context) does, and the tail gets ugly fast. If you have strict SLAs, cap k_sequential separately.

3. Latency vs Accuracy Frontier

Two axes: k_parallel (batched, does not hurt wall-clock) and k_sequential (does). Trace the Pareto front.

• Trigger: latency SLA under pressure.
• Guardrail: p95_latency ≤ SLA.
• Stop: Pareto front stable over successive windows.

When to use. Your p95 latency is creeping up as you raise k, or you have hard SLA commitments and need to know the Pareto-optimal (k_parallel, k_sequential) combination for your accuracy target.

Setup. Define p95 and p99 latency ceilings. Configure both k_parallel (batched concurrent requests) and k_sequential (retries that see prior output). Pick a target accuracy. The template plots the frontier across runs.

Watch for. Provider-side rate limiting. Parallel sampling spikes request volume, and throttling inflates latency in ways static benchmarks miss. Validate under peak-traffic conditions, not 3 a.m.

4. Verifier Strength Audit

Compare pass@k with and without the verifier's selection rule. The delta is how much the verifier is buying you. If it is small, fix the verifier before spending more on k.

• Trigger: sampling lift smaller than expected; planning a verifier swap.
• Guardrail: none; this is diagnostic.
• Stop: fixed horizon.

When to use. Your sampling lift is weaker than the literature suggests (for example, doubling k only gets 2–3% accuracy improvement), or you are planning to swap the reward model or verifier and want a baseline.

Setup. Run the same queries twice: once with verifier-based selection, once with a baseline selection (random or first-sample). Compute the gap. This is cheap; you can do it on a small held-out set.

Watch for. A flat delta means the verifier is uninformative on your distribution. No amount of k will fix that; the ROC-n-reroll bound is binding. Fix the verifier (better reward model, stronger grader, process rewards) before scaling k.

5. Model Swap Under Budget

Smaller model at higher k versus incumbent at k = 1. Direct cost-per-correct-answer A/B. This is the T² recommendation packaged as a deployable change.

• Trigger: new model released, or incumbent cost exceeds target.
• Guardrail: hard quality floor.
• Stop: mSPRT declaring winner, or 14-day horizon.

When to use. A promising open-weight model just dropped, or your current frontier-model spend is unsustainable and you want to test the T² prediction on your traffic rather than on a benchmark. This is the headline test of the scaling-law finding.

Setup. Incumbent stays at k = 1 as control. Candidate smaller model runs at k values predicted to be cost-neutral (use the Knee Finder output as a starting point). Verifier must be identical across arms; otherwise you are measuring two things at once.

Watch for. Hard queries. Smaller models can lose asymmetrically on the long tail. Stratify quality by difficulty bucket. A win on aggregate means nothing if the bottom decile collapses.

6. Drift-Corrected Sampling

Sentinel queries plus CUSUM on pass@k. When the control chart signals, rerun the knee finder because the optimal k may have shifted with the input distribution.

• Trigger: continuous, runs in background.
• Guardrail: automatic rollback on quality breach.
• Stop: never. This one is always on.

When to use. Always. This is the one template that runs continuously rather than as a discrete experiment. It catches the case where an input-distribution shift moves the optimal k without touching your model or prompt.

Setup. Define a sentinel set: fixed, representative queries with known answers. Schedule them to run at a cadence (hourly, daily). Apply CUSUM to the rolling pass@k. When the chart alarms, auto-trigger a Knee Finder run.

Watch for. Sentinel-set staleness. If your sentinels no longer represent production, the chart will either over-alarm or under-alarm. Rotate sentinels quarterly, or use a longer moving window with tolerance for minor shifts.

What survives post-training

The T² finding holds through fine-tuning. RQ3 shows overtrained 37M–149M models still beat Chinchilla-optimal larger models on SciQ (66.80% vs 57.60%) after supervised fine-tuning. The effect is subdued but directional. Springer et al. (2025) note overtrained checkpoints are modestly harder to fine-tune, which damps the advantage without reversing it.

For practical purposes: the recommendation survives for every model family you actually serve. Qwen, DeepSeek, Llama, Mistral, and the recent open-weight reasoning releases are already in the overtrained regime T² forecasts. The question for you is no longer whether to overtrain. It is whether to sample.

The one-paragraph answer

Pretraining scaling laws built the last generation of serving decisions. T² builds the next one: jointly budget (N, D, k) under one spend ceiling, forecast the optimum, and verify on your traffic. The lifecycle is instrumentation first, scaling-law prior second, production posterior third, and a control chart on pass@k fourth. The six templates above are the surface area you need to cover. Everything else is bookkeeping.

Read the source

• Roberts, N., Cho, S., Gao, Z., Huang, T., Wu, A., Orlanski, G., Trost, A., Buchanan, K., Albarghouthi, A., & Sala, F. (2026). Test-Time Scaling Makes Overtraining Compute-Optimal. arXiv:2604.01411.
• Yang, A., et al. (2024). Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement. arXiv:2409.12122.
• Wang, X., et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171.
• Snell, C., et al. (2024). Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters. arXiv:2408.03314.

Try it

• Eval Builder with pass@k — set "Samples per input" above 1 on any new suite.
• Optimization dashboard — the six templates above are selectable when you create a new optimization.
• Previous post: the T² result in depth.