Nine benchmark charts sit on the page announcing Mistral Large 4’s open weights. The model is top of four of them. On three of the remaining five, the prose says so plainly: “effectively tied with the leading open-weight model, GLM-5.3”, “second only to Kimi K3 for open models”, “ranking behind only Claude Opus 5.5 among the models evaluated.” That is a vendor naming the model that beat it, in its own voice, three times.
Then, in a collapsible box above all of it, labelled Summary and sitting under a loading indicator that reads Thinking, a generated paragraph says the model “leads open-weight benchmarks in coding and defensive security.”
The page’s own cybersecurity chart, eight hundred pixels further down, does not show that.
The three layers of one page, scored separately
A vendor launch page is not one document. This one has three layers that make claims, and they disagree with each other by increasing margins the further you get from the data.
Layer three, the chart payloads. Every chart on the page is an SVG built from a JSON block in the markup. Those blocks carry the real numbers, including for the bars that are not Mistral’s. On the Artificial Analysis Cyber Index chart, the payload reads: Mistral Large 4 at 50.9, Beam 1.0 at 34, GLM-5.3 at 36, Kimi K3 at 41, GLM-5.3 Flash at 50, MiMo-V2.6-Pro at 56, DeepSeek-V4.1-Flash 1017 at 57. Mistral plotted two models above itself and labelled them. Nothing is hidden.
Layer two, the human prose. Tabulating all nine charts against their own payloads: Mistral Large 4 is top on FrontierCode 2.0 (19.7 against DeepSeek’s 19.3), Terminal-Bench 4.0 (41.4 against GLM-5.3’s 40), harmful-cyber refusal (96.6% against Kimi K3’s 93.6%) and B3 attack resistance (97.1% against GLM-5.3’s 93.3%). It is not top on the Coding Agent Index, Surge Code human preference, the Cyber Index, FinWorkBench or BrowseComp. Four of nine.
Of those five, three are conceded by name in the sentence that introduces the chart. A fourth, BrowseComp, is framed selectively rather than falsely: Mistral’s 84.5% “brings Mistral Large 4 at a higher performance than models like GLM-5.3 or DeepSeek Flash 4.1 1017” — both true, both below it — while Kimi K3’s 91.2%, plotted on the same chart, goes unmentioned. That is a real thumb on the scale, and it is a small one, because the number it omits is drawn six centimetres away.
The fifth is the Cyber Index, and the prose makes no ranking claim about it at all. It says the model “scores 51%” and that it “handles these legitimate defensive security tasks without unnecessary refusals.” Both are accurate. Neither says it leads.
Layer one, the generated summary. Which is where “leads open-weight benchmarks in coding and defensive security” appears, and it is the only statement on the page that the page’s own data contradicts.
What the independent leaderboard says about defensive security
The Cyber Index is not Mistral’s benchmark. It is Artificial Analysis’s, an equally weighted average of three evaluations: CWE-Bench-AA (120 held-out audit-and-patch tasks), DeepsecBench-AA (vulnerability discovery, median F2 over three runs) and CyberGym-E2E-AA (131 memory-safety tasks requiring a proof-of-concept crash, a patch, and the project’s own tests still passing). All three run on the same open-source harness, and the methodology states the index “measures defense only” with exploit realization explicitly out of scope.
Its public leaderboard embeds per-model component scores in the page data. Deduplicating to one entry per model gives 25 models with a Cyber Index score, 14 of them open-weight. Mistral Large 4 at high reasoning effort sits 7th overall and 3rd of 14 open-weight models, behind DeepSeek V4.1 Flash 1017 at 56.8 and MiMo-V2.6-Pro at 56.1. Both are open-weight. Both are on Mistral’s own chart.
So “leads open-weight benchmarks in defensive security” is wrong on a dataset the page links to and plots.
The component breakdown is where it gets more interesting than a rank. Mistral Large 4’s three sub-scores are CWE-Bench 58.3%, CyberGym-E2E 79.4%, DeepsecBench 15.0%. On CyberGym-E2E — find the bug, crash it, patch it, keep the tests green — it is first among open-weight models and fourth of all 25, beaten only by GPT-6 Sol’s privileged-access variant, Kimi K3.1 Pro and its own preview. On DeepsecBench, which isolates discovery alone, it is 7th of 14 open-weight and 16th of 25, at less than a third of the leader’s 48.1%.
That is a sharp, specific shape: a model that is unusually good at fixing a vulnerability someone has pointed it at, and middling at finding one nobody has. For a buyer, that distinction decides whether the model slots in after a scanner or in place of one. Neither the launch page nor its summary mentions it, and no single index number can carry it.
The partner document carries the wrong model’s numbers
Hugging Face published its own write-up, Welcome Le Chonk, on 29 October, the day the weights shipped, credited to eleven named staff. Mistral’s model card links to it. It is the second account of the same system, and three of its figures belong to a different model.
It says the model “scores 82% on the Artificial Analysis Cyber Index test that asks a model to reproduce a real vulnerability in open-source software and then patch the software.” That test is CyberGym-E2E-AA. The leaderboard today gives the released model 79.4% and the preview 81.7%. Eighty-two is the preview’s number rounded.
The same post reports “28.3% on Terminal-Bench 4” and a “Coding Agent Index score of 49.8%”. Mistral’s chart labels its preview bar 28, explicitly marked “ML4 Preview - self reported”, and gives the released model 41.4. Mistral’s own prose calls that an improvement of “13 percentage points.”
Here the direction matters, and it does not run one way. The cyber figure flatters the release by about 2.3 points. The coding figure understates it by 13 — a partner post telling readers the new model scores 28.3% on a benchmark where the vendor measured 41.4%. If this were spin it would be remarkably bad spin. It reads instead like a write-up assembled while the leaderboard was still catching up with a model released that morning, which is also the most likely explanation for the post’s claim that the model is the “fifth best open weights model” on the Intelligence Index when the same leaderboard’s deduplicated open-weight ranking puts it 10th.
Neither document is dishonest. They were written for different readers at different hours of the same day. But a figure that identifies the wrong version of a model is worse than a figure that is merely stale, because nothing about it looks wrong.
What the three extra weeks bought
Mistral says three further weeks of reinforcement learning drove “massive improvements in coding and defensive cybersecurity” and that “gains continue to accelerate, with no signs of saturation.” The coding half is well supported: Terminal-Bench 28 to 41.4, and the leaderboard records a FrontierCode score for the preview and an improved standing for the release.
The cyber half decomposes less kindly. Preview to release, on the independent leaderboard: CWE-Bench-AA 50.8 to 58.3, up 7.5 points. DeepsecBench-AA 16.0 to 15.0, down 1.1. CyberGym-E2E-AA 81.7 to 79.4, down 2.3. The index moves 49.5 to 50.9 — up 1.4 points, from one component improving and two slipping. In task counts, CWE-Bench went from 61 to 70 of 120 solved; CyberGym went from 107 to 104 of 131.
That is a real gain in audit-and-patch, bought alongside small regressions in the other two thirds. “Massive” is doing more work than the arithmetic supports. But Mistral plotted the preview bar on four separate charts so a reader could do exactly this subtraction, which is more than most vendors offer, and the regressions are visible in the public leaderboard data rather than buried.
What is not checkable here
The Surge Code human-preference result is a vendor-commissioned blind evaluation with no published sample size, grader count or task list; it is reported here as what Mistral says it found. The customer quotes from Booz Allen, Cisco, Cloudflare, CrowdStrike, Elastic, Fortinet and ENISA sit on Mistral’s own page and are endorsements, not independent tests — Booz Allen’s is the only one naming a result, and it names an internal benchmark nobody outside can see. The Cybench 93%-of-40-challenges figure in the Hugging Face post is not in Mistral’s launch page or the Artificial Analysis index, and I could not locate its source. And whether the Summary box is generated by a model is an inference from its own markup, which labels it Summary beneath an element that reads Thinking; Mistral does not say.
The practical reading: on this page, trust the chart data, then the prose, then nothing. The layer a reader meets first is the one with the least support underneath it, and on a page this well instrumented that is a choice about machinery rather than about honesty.