Across 21 model-and-scaffold configurations, 78.25% of attempts produced a static-analysis checker that compiled. 73.65% produced one that fired on the vulnerable version of the code and stayed quiet on the fixed one. And 32.30% passed. That last gap is the finding, because almost nothing is lost between “it builds” and “it catches the bug” — the drop happens afterwards, on the requirements nobody writes a leaderboard around.
CheckerBench is an executable benchmark submitted on 6 October by He and eleven co-authors. Its task is narrower than the usual coding-agent exercise and considerably less forgiving: read a defect specification, explore a real repository, and write a reusable static-analysis checker that finds that class of bug. Not a patch for one instance. A rule. The paper describes it as “an executable benchmark of 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems,” with each task shipping “vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold.” Seven models were run across three agent scaffolds, three independent repeats each, inside a 50-minute generation budget. The full text is on arXiv.
How the attrition is distributed
The headline, from the paper’s own abstract, is that “mean Pass@1 is 32.30%, while the best reaches 45.33%” — that best being one frontier model on one particular scaffold. Half the tasks defeat the strongest configuration tested. But the aggregate number is less informative than its decomposition, which the paper gives as three successive gates.
Build is 78.25%. Differential success rate, meaning the checker fires on the broken revision and not the patched one, is 73.65%. Pass@1 is 32.30%. In the paper’s own arithmetic, “the gap between Diff. SR and Pass@1 is 41.35 percentage points, much larger than the 4.60-point gap between Build and Diff. SR.”
So roughly five attempts in a hundred write a checker that compiles but cannot tell the vulnerable version from the fixed one. Forty-one in a hundred write a checker that does tell them apart, and fail anyway. What they fail is the remainder of the acceptance gate: the paper lists “semantic fidelity, process quality, false-positive control, and normal completion.” One example in the paper is worth more than the category label — among the recorded failures, “a planned path-sensitive analysis becomes AST matching that omits locking and control-flow conditions.” The checker works on the two revisions it was shown. It is not the analysis that was promised.
What 4,360 failed runs were failing at
The paper’s failure analysis adjudicates 4,360 selected failed runs into eight categories. Reading them as a group rather than a ranking is what makes the shape visible.
Two categories are mechanical: “missing or invalid artifacts (1.83%), and compilation or registration (1.05%).” Together, 2.88%. The plumbing is essentially solved.
Two more are about detection: “differential detection or localization (8.17%)” plus “patch-specific overfitting or anchoring (3.56%).” Together, 11.73%.
Three are about the quality of the work itself: “inadequate synthesis process (22.96%), and inadequate semantic modeling (21.15%)” plus “excess false positives account for 15.96% of failures.” Together, 60.07%.
Sixty percent against under three. What the failures are made of is not tool operation, and mostly not bug-finding. It is whether the thing built is a defensible piece of analysis.
The counter-explanation, which is the largest single category
The eighth category is the one that complicates all of the above, and it is the biggest: “budget exhaustion (25.32%).” A quarter of failures are not bad work. They are unfinished work inside a 50-minute window. If that is the real constraint, the quality reading above is partly an artifact of the clock.
The paper tests this, and the answer is unusually clean. Retrospectively applying shorter cutoffs, “aggregate Pass@1 rises from 7.24% at 10 minutes to 26.05% at 30 minutes and 32.30% at 50 minutes, with diminishing gains: the 40-minute cutoff captures 93.02% of the successes observed by 50 minutes, and the final 10 minutes add 2.25 percentage points.” The curve is flattening hard at the point where it was cut. More clock buys very little.
It buys little, with one caveat the authors state themselves and I am not going to paper over: “performance under separately imposed shorter deadlines remains untested.” These are retrospective cutoffs on trajectories that knew they had 50 minutes. An agent told up front it has 20 would presumably budget differently. So the time curve weakens the budget explanation; it does not dispose of it.
The same holds in the other direction for false positives. If 15.96% of failures are excess false positives, a stricter-than-necessary precision gate would be an obvious suspect. The paper sweeps it from 10% to 50% and reports that “the aggregate acceptance rate increases from 32.05% at 10% to 32.30% at 25% and 32.83% at 50%, yielding a net increase of 0.78 percentage points.” The largest move for any single configuration across that entire range is 1.67 points. The threshold is not what is holding the number down.
Configuration matters more than the model ranking suggests
One finding here is awkward for anybody who reads model leaderboards as model properties. The three scaffolds average 36.95%, 33.81% and 26.14% — “a 10.81-percentage-point difference between the highest and lowest averages.” And the ordering of models changes depending on which one you use. One model falls from 38.67% on the best-averaging scaffold to 24.33% on the worst, which the paper records as “a 14.34-point gap.” The paper’s conclusion is that the pair, not the model, is “the appropriate unit for comparison and selection.”
It also reports that no configuration wins across the board: “no evaluated configuration leads simultaneously in synthesis success, process and semantic quality, and observed diagnostic precision.” The configuration with the best process and semantic scores of all 21 reached 25.00% Pass@1.
One more number resists the easy reading. Of the 300 tasks, 159 are C/C++, and there every model posts its worst language result, the leader reaching 24.53%. The aggregate is substantially a statement about C and C++ under one analyzer.
Most of the capability is the loop
The ablation is the most quotable result in the paper. Taking one model’s full configuration at 39.67% Pass@1, the authors strip components: “8.33% without skills and strategy references, 13.33% with execution feedback hidden, and 0.33% for one-shot generation without tools or a harness.” One-shot generation produced “invalid or incomplete submissions in 237/300 tasks.”
Nearly all of the measured capability is the loop, not the single forward pass. Though here too the authors decline the clean story: “differences between historical and new run conditions prevent attributing the gaps solely to the ablated components.” That sentence is doing real work and most write-ups of a result like this would drop it.
Contamination gets the same treatment. CVE publication postdates model release for only 3 of 300 tasks on one model and 1 on another, and a paraphrase-and-rename intervention cut exact CVE recall from 10.00% to 1.00% while costing only 2 to 3 points of Pass@1 — recognising the vulnerability was not what was carrying the score. Judging was done by a frontier model with human validation at a pooled Cohen’s kappa of 0.85, which is disclosed rather than assumed.
Where this leaves a reader
Against the prior art the benchmark beats, it does well: 80.3% valid checkers against 67.2% for KNighter on that tool’s own 61 Linux-kernel tasks, per the paper’s baseline comparison. Strip the skill library and the same method drops to 29.5% on those tasks, which is the ablation finding again from a different direction.
What I would take from it is narrower than the 32.30% headline and better supported. On this task, in this harness, the agents have stopped failing at mechanics and have not started succeeding at judgement, and the two sensitivity analyses that could have explained the shortfall away — the clock and the precision gate — both come back small, by the authors’ own measurement. The one explanation still standing is a quarter of failures that ran out of time under a protocol the paper is careful to say it has not tested against a real deadline.
Numbers here are the authors’ own, from a benchmark they built and evaluated, with a model as judge. The arithmetic groupings of the failure taxonomy are mine.