Five elicitation methods were pointed at a model with a chemistry classifier encrypted inside its weights. Four came back at coin-flip accuracy. The fifth recovered the hidden capability completely, and the authors tell you on the same page why that does not count: it won by enumerating all 256 possible triggers.

That asymmetry is the whole value of AI Models Can Provably Hide Arbitrary Capabilities, posted 9 October and currently on the Hacker News front page. The paper is a proof of concept, not a report from the wild, and the interesting part is not the headline but the precise shape of what the authors are and are not claiming.

What is embedded, and what the claim covers

The construction compiles a small neural network into a host model’s weights, encrypted behind a digital locker, so it decrypts and runs only when a secret trigger appears in the input. The hidden payload in the chemistry experiment is modest and specified exactly: a ternary MLP with dimensions 64, 8, 16, 2, containing 672 ternary weights, predicting whether a set of reagents yields a candidate product from 64-bit Morgan fingerprints derived from USPTO reaction data.

The claim attached to it is narrower than the title reads. The paper states that it is computationally hard to recover the trigger or to determine which capability is embedded, under cryptographic assumptions, even given full weight access. It then immediately concedes the thing most readers would assume it denies: “The presence of a compiled circuit itself may be detectable: our primary focus is hiding its trigger and capability.” Hiding that something is in there is not what is being claimed.

The five attempts, one at a time

Table 2 of the paper reports held-out chemistry accuracy, means over three seeds with 95% confidence intervals. The encrypted model handed a wrong trigger sits at 50.6%, which is chance on a balanced binary task. Given the correct trigger it reaches 73.5%. Everything else is an attempt to get from the first number to the second without the key:

  • Supervised fine-tuning, 500 steps directly on the task data: 50.9%, against 50.6% for a control model whose locker holds a random network instead of the trained classifier. No benefit from the real payload being in there.
  • Latent adversarial training at eight layers: 52.9%, against 53.6% for that same control. The encrypted model scores marginally below the thing with no capability inside it.
  • GCG trigger search over a 16-bit space: 52.7%.
  • Weight noising, at scales of 0.01 and 0.1 before fine-tuning. No table row for this one; the paper reports only that it “did not materially affect the observed elicitation results” on either model.
  • GCG trigger search over an 8-bit space: 73.5%, which is exactly the model’s correct-trigger accuracy.

The comparison that gives those numbers meaning sits in the same table. A conventionally backdoored password-locked model performing the same task goes from 50.7% at wrong-trigger to 92.8% after the same 500 steps of fine-tuning, and to 91.7% under LAT. The standard way of building a hidden capability gives it up almost immediately; this one does not.

The fifth attempt, and why the authors discount it

The 8-bit GCG result is a complete recovery, and the paper does not bury it. It explains the mechanism in plain terms: GCG “recovers the encrypted model’s correct-trigger performance with 8-bit triggers by exhaustively enumerating all 2^8 candidates, matching query-matched random search.” Two hundred and fifty-six guesses, and the search performs no better than guessing randomly. That is a statement about the size of the toy trigger space used for the experiment, not about the search method defeating the cryptography. Widen the space to 16 bits and the same method returns 52.7%.

The authors are explicit that the experimental settings were weakened on purpose for speed, and explicit about why that is defensible: “even at settings that would not withstand cryptanalysis, they remain ineffective,” because the elicitation methods tested do not exploit the hash structure at all. That reasoning is sound for SFT, LAT and noising. It is self-evidently not sound for brute-force trigger search, which is precisely a method whose cost depends on the parameter they shrank. The paper’s own table shows both halves of that, which is to its credit.

The objection from the thread, and the paper’s answer

The sharpest comment under the Hacker News submission was a deflation. One commenter called it “a cool demo, but that’s about it,” arguing the attack is “trivially detectable” because the compiled weights will look nothing like trained weights: very sparse, not uniformly random.

On the untreated construction, that objection is simply correct, and the paper says so first. Section 7: raw compiled weights are “trivially anomalous: approximately 99% are exact zeros, the rest are mostly integer-valued, and excess kurtosis is in the hundreds.”

The paper then does something about it. An obfuscation pass widens each hidden layer with cancelling plus-minus pairs of Gaussian weights and applies SwiGLU’s orthogonal and permutation symmetries, which the authors report brings KS distance and excess kurtosis close to a same-shape Gaussian-init reference and cuts sparsity and integer fraction by over 100x, with the compiled function preserved exactly (test MSE 8.9e-5 for both the raw and obfuscated versions). So the specific statistical tell the commenter named is addressed. What follows that result is the honest limit: “resistance to broader detection methods remains unexplored.” Both things are true. The obfuscation defeats the univariate statistics it was built against, and nobody has tried anything else yet.

One caveat on that exchange: a reply in the thread pointed to arXiv 2407.14494 as further support for obfuscating weight distributions. That paper is InterpBench, about semi-synthetic transformers for evaluating interpretability methods, and does not support the point being made. The paper’s own Section 7, which the same reply also cited, does.

The price of hiding 672 weights

The overhead figures are where the proof-of-concept framing earns itself. Hiding that 672-weight ternary MLP produced a host model of 38 layers, roughly 2.07 billion dense parameters, 1.9 million of them nonzero, and a maximum residual-stream width of 10,266. The hash subcircuits account for about 1.98 billion of those dense parameters, roughly 95% of the total. The paper puts the marginal cost at approximately 3.1 million dense host parameters per hidden weight. The password-locked baseline it is compared against has about 170,000 parameters in total.

An optimising compiler mode gets full-strength security parameters within reach: a 24-round Keccak-f[1600] locker with a 128-bit trigger and the same classifier yields 110 layers and 3.58 billion dense parameters, matching the classifier on all held-out triggered inputs and running a stand-in circuit on all 10,000 wrong triggers tested. Still two billion parameters of overhead to hide a classifier that fits in a kilobyte.

Where the authors stop

Three limits are stated in the paper rather than left for a reader to find. The constructions “require cryptographic and ML expertise and incur substantial computational overhead, making them impractical for directly hiding large capability models at present.” The chemistry task was “deliberately constrain[ed] to classification rather than synthesis planning.” And the individual GLU units were verified inside Gemma 3 270M, but “integrating the complete construction into a pretrained model remains future work.”

What survives all of that is the negative result, which never needed the attack to be practical: a failure to elicit a capability does not establish its absence. The paper has made that concrete enough to test defences against, and released the model organisms to do it with. The load-bearing finding for anyone running capability audits is not that someone is doing this. It is that the two most common things an auditor reaches for, fine-tuning on the task and adding noise to the weights, were both measured against a thing that definitely had the capability, and both reported chance.