A paid extraction API invented more missing fields than 13 of the 16 raw language models it competes with, and the fix for the models cost one sentence of prompt.

The test is published at earnanhonestdollar.com/bench, run on 27 September 2026. It asks one narrow question about web extraction services: when a field is not on the page, does the extractor say so, or does it make something up?

The trap is two pages that differ by one row

Most extraction benchmarks score whether the right answer came back. This one scores the opposite case. Each trap is a pair of pages identical except for a single row: one page shows the answer, the other does not. Both pages carry the same decoy nearby, and the decoys are the kind a careless reader falls for:

  • Was $493.00 sitting near the current price
  • Fact-checked by Omar Tamm, who is not the author
  • Last updated September 7, 2020, which is not the publication date

The scoring only counts the page where the field is missing. In the authors’ words, “We tested 42 pairs across 7 page types and scored the pages where the field was missing.” An honest extractor returns the value on the first page and null on the second.

The one sentence

Every contestant received this instruction: “Use null for any field whose value is not on the page. Do not guess.” For the models, the “without” column is the same task with that sentence deleted.

The aggregate: “All 16 models made up more without the sentence: 405 of 573 missing fields without it (70.7%), 116 of 574 with it (20.2%).”

One trap carries the result more sharply than the aggregate does. On the Was $493.00 page, “all 16 models called 493 the price without the sentence; with it, 1 did.” Sixteen out of sixteen down to one out of sixteen, on a single decoy, from one added line.

Note what the number is not. Twenty percent is not accuracy; it is the share of missing fields that still got fabricated. One in five invented values survives the instruction. The sentence is a large improvement on a bad baseline, not a fix.

The paid tier is where it gets uncomfortable

Three commercial APIs were entered alongside the models. Firecrawl, per the published table, “made up 24 of 36 missing fields, more than 13 of the 16 models with the sentence, by nonoverlapping 95% ranges.” And it did not merely guess: “All 24 answers copied the decoy.”

Against that, the cheapest configuration in the table: “Plain fetch plus GPT-6 Luna made up 5 of 36, for $0.0049 across the full run.” Half a cent for the whole 84-page run, five errors instead of twenty-four.

The authors do not let that comparison stand unqualified, and the caveat is theirs rather than mine: “Paid APIs ran on free tiers and only with the sentence. Paid plans may differ.” ScrapingBee, they note, “has no prompt or schema slot, so the null rule went into each field description” — it was not given the same lever the models got. A paid product beaten by a free-tier margin is a weaker claim than the headline ordering suggests, and the page says so before a critic can.

A verifier costs a fraction of a cent

The second half of the test asks whether a cheap second pass catches the fabrications. Each returned value was handed to a checker as a statement to confirm, for example The author is Omar Tamm:

Checker Made-up values caught Correct values rejected
GPT-6 Luna 38/49 0/47
Jev 1.13 23/49 0/48

“Neither checker rejected a correct value in this run.” Zero false rejections is the figure that makes the pattern usable, because a verifier that discards good data costs more than it saves. Checking all 126 unique page-and-value pairs cost $0.0049 with GPT-6 Luna and $0.0024 with Jev.

Jev, a typed-decision model this site has covered before, caught the blunt decoys and failed on the subtle ones: on times given as prep time when they were resting, cooking or total time, it caught 0 of 6. The general-purpose model was the better judge here, which is the reverse of what a specialised decision model is usually sold on.

Who published this, and what they concede

The benchmark is run by a marketplace that lists agent services for sale. That is a commercial interest in how extraction providers rank, and it belongs in any reading of the table. Two things blunt it. The marketplace states plainly that “A listing is not a score: we do not verify provider claims, and this benchmark covers web extraction only so far.” And the limitations section gives away more than most vendor benchmarks would:

  • “One run per contestant. Repeats have not been run.”
  • “These were synthetic pages with seven page types and traps we wrote. Real sites may differ.”
  • “Rows with overlapping ranges are not clearly separated; read the top and bottom, not the exact order.”

That last one matters for anyone about to quote a rank. Single-run scores with Wilson intervals as wide as 20 percentage points do not support an ordering; they support a top group and a bottom group. Email traps are excluded, one preview model was dropped for unparseable output, and a TBA venue is scored as fabricated. All disclosed on the page.

So the reusable finding is narrower than the headline and more solid than a ranking: telling a model not to guess cuts fabrication on absent fields by roughly two thirds, and a second cheap model catches most of what is left without throwing away good answers. The ordering of the contestants is one run on pages the authors wrote, and the authors are the first to say so.