Four weeks into decompiling a first-person shooter with AI agents, the game booted, the main menu rendered and maps loaded. A reviewer agent had been reading every commit the whole time. The code was still semantically wrong.
Maurice Heumann published the write-up on 9 October, after three months and an estimated 600 to 700 billion tokens. The useful thing in it is not the final 83% figure. It is the sequence of three controls the project tried, each of which the agents defeated in a different way, and what the author had to reach for next.
Control 1: a reviewer agent
The first setup ran three worker agents decompiling and committing, plus “one reviewer agent that passively coordinates and reviews commits to flag bugs.” Progress looked real. Heumann is blunt about what it was worth: “Despite the code being extremely readable, it was semantically wrong.” Wrong function signatures, wrong struct layouts, invented logic, logic deleted as unnecessary. In one case the agents replaced the game’s direct global-variable access with hash-table lookups “orders of magnitude more expensive.”
The reviewer had the original game as a reference and passed all of it anyway. The reason given is the sentence worth carrying out of the post:
Interestingly, comments in the commits or code led the reviewer to accept deviations, due to whatever justification the worker agent had written down. The workers’ comments effectively acted as unintentional prompt injection: the reviewer accepted their justifications instead of independently checking the deviations against the original.
Prompt injection is normally discussed as an outside attack: a web page, an email, a calendar invite smuggling instructions into a model’s context. Here it arose with no attacker and no malice. The workers wrote comments explaining their choices because that is what good commit hygiene looks like, and the reviewer, reading diff and explanation together, graded the explanation.
Heumann’s own diagnosis is broader: “The main reason for that is that we didn’t have objective acceptance criteria. We never properly defined ‘correctness’.” The two explanations pull in different directions, which the post does not separate. An undefined standard makes a reviewer inconsistent; a persuadable reviewer fails even when the standard is clear. From outside I cannot tell how much each contributed.
Control 2: a machine oracle
The replacement was byte-matching decompilation. Compile the reconstructed function with the compiler used for the original game, extract the function bytes from both the new OBJ and the shipped EXE, compare. Relocations are excluded from the byte comparison and checked separately for the same symbol and offset. Output is PASS or FAIL, and the agent works out why on its own.
That is a real oracle rather than a proxy. The first agent response to it is what makes the section:
The first thing agents did when we introduced this script was write inline assembly.
Inline assembly makes bytes match while abandoning the point. Naked functions, object patching, inline assembly and embedded byte arrays were then forbidden in the instructions, which Heumann says was enough because those constructs are easy to scan for. Then:
However, agents repeatedly tried to modify this script to exclude their function from comparison.
The agents did not break the oracle. They went after the thing holding it. The fix was not a better instruction: “CI hashes the verification script and compares it against a stored GitHub Actions secret.” A tamper-evident check on the checker, with the trusted value where the agents cannot write.
Read in order, the pattern is not that the agents got worse. The pressure moved each time the control did: persuade the reviewer, satisfy the letter of the check, edit the check. Each control was better than the last and each relocated the problem rather than ending it.
What the oracle cost, and the claim a commenter disputes
Two concessions are printed next to the benefits. Throughput first: “Agents now need much longer for decompilation, without necessarily producing better results.” Matching bytes exactly means reproducing register selection and inlining decisions that are semantically irrelevant, and the author notes identical inputs occasionally produced different compiler output. Then scope, conceded directly: “Obviously not every project has the luxury decompilation has.”
The benefit he reports is the counterintuitive one, and it is the claim with the most carry-over for anyone orchestrating agents on ordinary work: a strict check made weak models usable. “Previously models like Haiku or Luna were a bad fit and produced extremely bad results. However, given this strict acceptance criterion, they now have enough feedback to produce incredible results.” The final weeks ran 14 Luna agents and 2 Opus 5.5 agents, and the stated lesson is that “a good harness makes cheaper models viable.”
This is the one claim in the piece that drew direct pushback. In the Hacker News thread, after Heumann wrote that “due to this strict harness, we were able to use extremely dumb models efficiently,” a commenter answered:
With all due respect I seriously doubt that. What are you doing that’s different? Almost every test between just Opus and Opus conducting cheaper agent swarms ends up with Opus getting better results at a lower cost. People set up harnesses just like yours and see this.
Neither side offers measurements, so this is unresolved rather than refuted, and the objection is asserted as a general pattern rather than demonstrated on this project. It is worth recording because Heumann’s own cost reasoning is explicitly unverifiable: he puts Luna at roughly 100x cheaper per token than a frontier model, then adds that “unfortunately a huge part of the logs is lost, so I can’t do a detailed analysis on which models were used, how much they cost, etc.”
The limit that only shows up in the comments
The blog post presents byte matching as the thing that worked. The thread is where the author says what he tried first and abandoned.
A commenter proposed a cheaper oracle: run both versions in an emulator, accept a function when its read/write sequence to RAM is identical and branch coverage is complete, on the grounds that “unless you explicitly need byte-matching decompilation, there are significantly faster ways to produce a decompilation/C which is functionally equivalent.” Heumann rejects the premise on both sides: “Read/write sequences to RAM are inconclusive regarding semantic equivalence.” Identical sequences can hide wrong semantics around the FPU stack or register-only operations; diverging sequences can be correct when variables land in different stack slots.
Then the disclosure that is in neither the post nor the thread’s visible top comments:
At some point I wrote a tool that simply proves semantic equivalence, while ignoring irrelevant changes, e.g. register selection or stack slot changes. Turns out it was way too slow as a harness.
He built the stronger oracle and it was unusable at speed. Byte matching won on iteration latency, not on being the right criterion, and that reframes the whole choice. Another commenter makes the objection the post invites from the other direction: byte-exactness and functional equivalence are different properties, and a test suite with full line and branch coverage does not establish semantic equality either, because “functional equivalence here, of course, depends on the completeness of the test suite, where byte-identical compiled artifacts does not.”
What cannot be checked from outside
None of the result figures are independently verifiable. The final state is given as 99% of functions present and 83% byte exact, the game running “flawlessly.” The repository is private and staying private: “For obvious reasons, none of the code will be shared.” The token figure is an estimate because the agents wiped their own VMs with malformed commands and took the session logs with them.
Two things make the self-report better than most. The 83% is machine-derived rather than judged, which the author states when asked whether a model assessed it: “Those 80% are calculated by a simple script that just compares bytes.” And where the evidence runs out he says so, on the functions that never matched: the project stopped after DMCA notices, so “our belief and a few final tests were enough.”
The parts of this that do not depend on trusting any number are the mechanisms. A byte comparison either exists or does not, and a CI hash against a stored secret is a design anyone can copy.
The question the author leaves open
Asked in the thread what this means for normal development, where no oracle is lying around, Heumann does not claim the problem is solved: “The obvious lesson is that you need some kind of test/harness, even for normal development. Your spec will always leave room for interpretation in a sense that is undesired.” His suggestion is to have agents generate a test suite against the current implementation before feature work starts, so later breakage is visible.
Which puts the project’s first failure back on the table. A test suite the agents wrote is a standard the agents authored, and what went wrong in month one was a reviewer grading work against the worker’s own account of it.