A category with one product in it is a company’s marketing. A category with two products benchmarking against each other is a category. Nine days after TypeSafe AI launched Jev, an open-source challenger shipped with a comparison table aimed squarely at it — and the table is more interesting for what it reveals than for what it claims.

What laya is

Laya is “an open-source, non-autoregressive System 1 decision model” that takes a state and typed questions and returns typed answers with calibrated probabilities, never generating text (model card: Hugging Face). That is the same pitch as Jev: decisions software can consume directly, no string to parse.

The architecture is unremarkable in a useful way. A ModernBERT-large backbone at 395M parameters, fully fine-tuned, plus a decision head trained from scratch — 421M total. Every option is scored at its own mask-marker token, then softmaxed within that question. All questions in a request are evaluated in one forward pass, in “~33 to 38 ms on GPU”. It was trained on “100% human-annotated real-world datasets, zero synthetic shortcuts”, per the card, in under two hours of fine-tuning.

The comparison, and the problem with it

Laya’s card publishes a head-to-head table against “TypeSafe Jev (Published)”. The headline figures: 38.4 ms p50 latency against Jev’s “~400 ms avg”, described as “~10.4x faster”; and 83.8% accuracy against Jev’s 67.8%, described as “+16.0% higher overall accuracy”.

The accuracy claim does not hold up as stated. Laya’s own evaluation section, four paragraphs further down the same page, reports two numbers: in-task macro accuracy 0.838, and “zero-shot (held-out task families)” macro accuracy 0.651 with macro ECE 0.207. The 83.8% in the comparison is the in-task figure — performance on task families it was trained on. Against Jev’s 67.8%, which the table attributes to “4 production workflows”, the like-for-like comparison is arguably 65.1%, which is slightly worse, with calibration error more than three times higher.

To be precise about the limits here: the 67.8% attributed to Jev could not be independently verified — it appears in Laya’s table, not in TypeSafe’s own published documentation, which describes the primitives without publishing accuracy figures (https://docs.typesafe.ai/introduction). So this is one vendor’s number for a competitor, compared against the vendor’s own most favourable number.

What does survive scrutiny

Three claims are structural rather than benchmarked, and those are the ones worth taking seriously.

The licence: Apache 2.0, against Jev’s closed weights and metered API. That is checkable on the model card and it is the most consequential difference for anyone deciding between them.

Deployment: 421M parameters runs on “commodity GPUs, Mac MPS, or CPU”, which makes air-gapped and on-device use possible. Jev is an API. For anything touching regulated data, that is not a performance question at all.

Latency: some of the 10x gap is architecture — a 421M bidirectional encoder in one forward pass against an API round trip — and some of it is simply the network. A local model beating a hosted one on latency is close to a tautology, but it is still the number that decides whether you can put a decision inside a request path.

Why this matters

The useful signal is not that Laya wins. It is that a second team built the same shape of thing, independently, and cared enough to benchmark against the first. Nine days is fast for a category to acquire a competitor.

Laya’s own card is unusually candid about where it should not be trusted — “evaluate on your own distribution before full automation”, and keep “arithmetic, counting, date comparisons, and multi-hop index lookups” in deterministic code. That candour is worth more than the comparison table, and it points at the same conclusion from the other direction: typed decision models are a real tool with a narrow, well-understood job, and the honest version of the pitch is smaller than the marketing version.