Query Chrome’s bug tracker three different ways and the lead changes hands twice.
Project Zero published Fuzzers vs. AI on 27 October, comparing two systems that run continuously against V8, Chrome’s JavaScript engine: Fuzzilli, a generative fuzzer, and BigSleep, an LLM agent that reads source code and writes crashing test cases. Both file through Clusterfuzz into the same tracker, which makes them unusually comparable. The post’s headline result is that the agent now finds slightly more vulnerabilities than the fuzzer, with “a 25% overlap in the specific bugs reported”, and that a dedicated experiment put the overlap “at roughly 60%”.
Those two numbers are not two estimates of the same quantity. One is symmetric and one is not, and the post’s own methodology section is where you find that out.
Three filters, two changes of lead
The raw tracker counts for April to September 2026 are Fuzzilli 316 and BigSleep 441. Restricting to type:vulnerability reverses it hard: Fuzzilli 178, BigSleep 95. Project Zero explains the gap rather than leaving it, noting that BigSleep “initially also found a lot of configuration and integration issues in V8” which are “legitimate issues, but less relevant for our comparison since there’s little real-world security impact”.
The third filter is “actionable” bugs, defined as not a duplicate, not closed as working-as-intended, and carrying at least one fix commit that references them. On that measure it is 45 for BigSleep against 37 for Fuzzilli, which the post calls “roughly a tie (small advantage BigSleep)”.
The filter that favours the agent is the one the summary leads with. That looks like the familiar shape, so it is worth saying plainly that it survives inspection: Project Zero argues its way to the actionable measure in the open, calls it “likely the best measure” at the point of use, and the argument is sound. A raw report count rewards whichever system files more noise, and a type:vulnerability count still includes duplicates and bugs closed without a fix. Writing this up as a vendor picking its flattering denominator would have been wrong about the text.
What the 60% measures
The 25% figure comes from matching duplicate reports in the tracker and in Clusterfuzz’s own database, and it is genuinely two-directional: 20 bugs out of 81 were reported by both systems, “with a slight majority of these found first by Fuzzilli and later by BigSleep”.
The 60% is a different kind of measurement. It is the result of an experiment in which BigSleep was given the commits that introduced historic Fuzzilli bugs and asked to find them again, 25 attempts per commit, using the production configuration of the time and Gemini 3.6 Flash. It re-discovered “~62% of the original bugs”, found a different already-fixed bug on another ~12% of commits, and came up empty on about 26%.
That is BigSleep’s rediscovery rate on Fuzzilli’s back catalogue. The mirror experiment, Fuzzilli attempting to rediscover BigSleep’s bugs, was never run, and Project Zero says why: “fuzzers cannot easily be given a similar ‘starting point’”, and in any case most historic BigSleep bugs were found in builds that fuzzers had already been hammering “for months, sometimes years”, so “they would most likely also not rediscover them in an experiment”.
Both of those are fair. The asymmetry is in the apparatus, not in anyone’s choices: an agent can be handed a commit and a target, and a coverage-guided fuzzer cannot. But it means the 60% cannot be swapped in for the 25% as a better estimate of the same thing, and the summary’s framing does exactly that, offering the experiment because the tracker method “has a tendency to underestimate overlap”. The tracker number describes how often two systems independently land on one bug. The experiment number describes how often one system can be walked back to the other’s findings. A symmetric quantity and a directional one.
Where the other 488 bugs went
The experimental corpus is worth following down, because the attrition is disclosed step by step and the filters are not neutral with respect to the question.
Querying the Fuzzilli hotlist from early 2023 to mid-2026 gives “around 1150 raw bugs”. Restricting to vulnerabilities leaves “roughly 600”. Requiring reproduction on a standard x64 debug-and-ASan d8 build, with only the flags --fuzzing, --jit-fuzzing and --expose-gc, leaves 132. Deduplication takes it to “a final number of 112 bugs”, which is 19% of the vulnerability set.
The flag restriction is explained and reasonable: historic V8 builds had no safe-flags-only mode, and telling the agent which flags to use would be a hint. But Project Zero also describes what those custom flags are for, that fuzzers set them “to turn on different fuzzing modes that make certain types of bugs easier to discover”. So the excluded bugs are disproportionately the ones Fuzzilli is specialised to find, and the 60% is measured on the remainder. The post does not claim otherwise; it lists the restriction among its caveats. It is simply a thing you only know if you read past the summary.
Counting the caveats by direction
The suspicion that a comparison tilts toward the in-house novelty is testable, so I counted every caveat in the post and sorted it by which system it favours.
Six run in BigSleep’s favour: the introducing commit is sometimes misidentified, only one commit per bug is available where production would offer several, production BigSleep reports with a one-to-two day delay so “fuzzers also get a head-start there”, pipeline hiccups stopped some commits being reviewed, the underlying model was upgraded repeatedly during the window, and the experiment’s model is “by now already outdated and outperformed by newer generations”.
Six run the other way: the reverse experiment was never run, the corpus excludes custom-flag bugs, “equivalent tweaks to a fuzzer could likely also result in higher discovery rates”, historic BigSleep bugs had a years-long fuzzing head start, the compiler-bug overlap is “significantly smaller” in Fuzzilli’s area of specialism, and the difficulty analysis came out against the hopeful hypothesis, with hard-for-a-fuzzer bugs also hard for an agent, which Project Zero reports as “it actually points in the other direction” and answers with “those fuzzers should probably be kept running for now”.
Six and six. The disclosure record is even-handed, which is the opposite of what the hunt expected to find, and it is what makes the one remaining complaint small enough to be worth stating precisely. The defect is a framing in a 158-word summary, not a thumb on any scale.
Two other things earn credit on the way past. Project Zero tested its experiment for memorisation by bucketing success rates around the model’s March 2026 knowledge cutoff and reports “there seems to be no statistically significant amount of memorization pre-2026”, a check nobody demanded. And the prompt-variation follow-up that lifts 60% to 70% arrives with its own discount attached in the same breath.
What the charts do not say
Three of the post’s findings exist only as charts, and all seven images on the page carry an empty alt attribute. The per-component breakdown, the difficulty correlation and the memorisation-by-period series are quoted here as Project Zero’s prose conclusions because their values are not in the text at all. A reader fetching this page as text, as I did, gets the argument and not the evidence for those three.
One number I could not reconcile: the actionable vulnerability counts are 45 and 37, which sum to 82, while the overlap is given as 20 of a total of 81. The off-by-one is probably a single reclassified report or a chart-to-prose rounding, and the figure it would be checked against is one of the images. I am flagging it rather than explaining it.
Also absent, and named as absent rather than estimated: cost. The post raises cost-per-bug itself and declines, saying neither system “is particularly calibrated for an optimal cost-per-bug ratio”. So the comparison on the table is vulnerabilities found, not vulnerabilities found per dollar, and nothing here supports a claim about the second.