What I Keep Writing in Reviews August, 2026

Over the past couple of years I have reviewed for NeurIPS, the DL4C workshops at ICML and NeurIPS, SpecOps, and Amazon's internal ML conference. After enough review cycles you notice that you are writing the same paragraph over and over. Different papers, different authors, same handful of gaps. This post is my attempt to write those paragraphs once, publicly, so that they are useful before submission rather than after.

Two caveats before I start. First, nothing here is about any specific submission: review discussions are confidential, and everything below is a pattern I have seen often enough across venues that it belongs to no one in particular. Second, none of these are reasons I reject papers on their own. They are the things that turn a paper I want to champion into a paper I have to argue for with a caveat attached, and that caveat is usually what sinks it in the discussion phase.

1. Contamination is a claim, not an assumption

If your benchmark predates the model you evaluate, the burden is on you to argue that the model has not already seen the answers. This applies to almost every popular code benchmark by now. "We use HumanEval" is not a neutral methodological choice in 2026; it is a choice that requires a paragraph of defense.

What convinces me:

  • The construction date of the dataset, stated plainly, next to the training cutoff of every model you evaluate.
  • A temporally held-out slice, e.g. only repositories or issues created after the cutoff.
  • A contamination probe: ask the model to complete a canonical test from the benchmark given only its name, and report how often it reproduces it verbatim.
  • Results on a second benchmark that is clearly post-cutoff, even if smaller and noisier.

What does not convince me is a sentence in the limitations section acknowledging that contamination "may" be a factor. That sentence leaves the confound unmeasured while the headline number still rests on it.

2. The baseline that matters is usually the boring one

Papers are diligent about comparing against other recent neural systems and much less diligent about comparing against something dumb. For code tasks the dumb baseline is often startlingly hard to beat:

  • For retrieval and localization: BM25, or plain grep over the repository.
  • For bug finding: an off-the-shelf linter or static analyzer that ships with the language.
  • For anything with an elaborate scaffold: the same base model, same budget, no scaffold. Just the prompt.
  • For repair: the empty patch, and the patch that deletes the failing test. You would be surprised how often weak test suites accept both.

That last category is the one I care most about. If a paper proposes a multi-stage agent architecture, the single most informative experiment is the one where you remove the architecture and keep the budget. I ask for it constantly and receive it maybe a third of the time.

3. Report what it cost

A method that samples fifty trajectories and takes the best is not comparable to a method that samples one, and reporting both as a single accuracy number hides the entire tradeoff. This has gotten worse as agentic systems have gotten more elaborate, because the expensive parts are now buried inside a loop rather than exposed as an explicit k.

The minimum I want to see alongside any pass rate: 

tokens in / tokens out   (mean and tail, per task)
wall-clock latency       (mean and tail, per task)
model calls              (or reasoning cycles, or steps)
dollars                  (at published API prices, stated)

And then, if at all possible, a Pareto curve rather than a point. A method that is two points worse at a third of the cost is frequently the more useful contribution, and papers routinely bury that result because the headline table only has room for one number. I have never once objected to a paper for being honest about its cost. I have objected many times to not being able to tell.

4. Ablations that do not actually ablate

Three recurring versions of this:

  • Two things at once. The "without module X" row also changes the prompt format, or the context budget, or the number of retries. Now the delta is uninterpretable.
  • Side effects left in place. A component is removed but the thing it wrote into the context, or the ordering it imposed, survives. The ablation measures less than it claims.
  • Only the favorable ablations. Every removed component costs several points. Real systems have components that do nothing; a table where everything matters equally reads as incomplete rather than impressive.

A negative ablation result makes me trust the rest of the table more, not less. If one of your five ideas turned out not to matter, say so. It is evidence that you looked.

5. One seed is not a result

Sampling is stochastic, agent trajectories diverge, providers silently update endpoints behind a stable model name, and benchmark harnesses have their own flakiness. A single run of a single seed on a two-hundred-task benchmark, reported to two decimal places, is a number with error bars wider than most of the gaps being claimed.

Three runs with a standard deviation is enough for me. If compute genuinely does not permit that on the full benchmark, do it on a documented subset and say which. Precision without variance is the single most common way papers overclaim, and it is also the cheapest to fix.

6. Does the metric measure the claim?

This is the one I write the longest comments about, because it is the least mechanical. Some pairs I keep encountering:

  • The paper claims correctness and measures test-passing, on test suites that were never designed to be adversarial.
  • The paper claims repair and measures the reference patch was reproduced, penalizing correct alternative fixes.
  • The paper claims understanding and measures exact string match on generated identifiers.
  • The paper claims efficiency and measures final prompt length, without accounting for how many times the loop ran.
  • The paper claims something about software engineering and evaluates entirely on standalone Python functions.

None of these proxies are illegitimate. Perfect metrics do not exist here, and a well-chosen proxy is how the field makes progress. The problem is the silent substitution: stating the strong claim in the abstract and the weak proxy in Section 5, and letting the reader assume they are the same thing. Name the gap yourself. A paper that says "we measure test-passing, which overestimates correctness on this benchmark because the suites are thin, and here is a manual audit of thirty patches to calibrate that" is a paper I will fight for.

7. Generalization is claimed far past where it is shown

A paper evaluated only on Python has shown something about Python. If every repository in the benchmark has thorough tests, or every issue is one a human already closed, those conditions are part of the result whether or not the abstract mentions them. Any of these is a perfectly good scope for a paper. The title just has to stay inside it. When I see a general claim resting on one language and one repository shape, my review turns into a scoping argument, and scoping arguments are boring for everyone. Narrow the claim in the abstract and the same paper reads as rigorous instead of overreaching.

A checklist I would apply to my own submissions

Condensed to what I would actually check the week before a deadline:

  • Is every benchmark's construction date stated next to every model's training cutoff?
  • Is there a no-scaffold, same-budget baseline?
  • Is there a trivially dumb baseline, and does it lose for the right reason?
  • Are tokens, latency, calls, and dollars reported next to every accuracy number?
  • Does each ablation change exactly one thing?
  • Is there at least one ablation that did not help, reported honestly?
  • Does every headline number have variance across seeds?
  • For each claim in the abstract, can I point at the metric that supports it, and do I believe the proxy?
  • Does the title's scope match the experiments' scope?

Why I bother writing this down

Reviewing is mostly an unglamorous obligation, but it is the one part of the process where you read a large, near-random sample of what the field is currently trying. The patterns are visible from that seat in a way they are not from inside your own project, where you already know which corners you cut and why they seemed reasonable.

Almost everything above is a request for the paper to describe itself more precisely, not to do more work. The experiments are usually already run; what is missing is the sentence that says what they do and do not establish. Writing it costs nothing, and it is usually what decides whether a reviewer spends the discussion period defending the paper or picking at it.