We ran three of them against a real tool surface, 48 requests, one machine. Choosing the correct tool was close: 14 of 24 for a 121M on-device model, 15 of 24 for a 151M encoder-based one, 10 of 24 for a 144M multilingual one. Refusing a request that no tool could serve was not close at all: 3 of 24, 16 of 24, and 0 of 24. And when we handed the third model an explicit "none of these" option, it ignored it on every single one of the 24. That last result cost us a claim we had already written down, which is the most useful thing in this post.

The Question the Model Card Never Answers​

In an agent, a decision model sits in front of the tool list and answers "which of these applies?". The interesting case is not the request that matches a tool. It is the request that matches none of them, where the only correct output is an empty result.

That framing is inherited, and it is worth naming where from. A tool from an earlier project on this site audits technical claims against the hard ceiling of their own physics, and the single rule that made it work was keeping a control set of real products the tool must never flag. Without it, three separate domains shipped a confidently wrong bound. The same discipline applies here with the sign flipped: the control set is 24 requests that no tool in the set can serve, and the correct behaviour on every one of them is silence. A decision model that always returns an answer is not a router. It is a guessing machine with a probability attached.

So: 11 tools, 24 requests each way. The tools are the real ones registered in our own Rust agent library, read from the source rather than invented for the test:

What We Ran, and One Control First​

Needle 3 (Cactus Compute, 121M parameters, Apache-2.0) is the only one of the nine that ships a callable artifact: a static C engine plus a 2-bit container. We linked libneedle.a into a Rust binary, loaded the 35,335,380-byte weight file, and drove the C API directly. The whole runtime — weights, engine, tokenizer, config — is 36.7 MB. No GPU, no Python, no network.

Verdict / OpenJev (151M on a ModernBERT encoder, Apache-2.0) is a fine-tune of a GLiClass zero-shot classifier with an explicit abstention candidate in its schema. It exports to ONNX, so it is reachable from Rust, though we ran it through PyTorch. Its weights are 605.5 MB in fp32 — 16× the whole Needle runtime, which is the price of the encoder design.

The first thing we did with Verdict was not run our test. It was to run theirs. An earlier pass of our own — 12 labels described as bare imperatives — produced a near-uniform distribution that looked exactly like a broken model, and it was actually a mis-shaped instrument: the engine renders every option as the literal sentence It is {description}, so "Run a program or shell command" was never a sentence fragment that template could use. Their README example caught it in one run: 2 of 2 correct on its home domain, and it abstains cleanly on an off-topic request. Only then was our test fair. We discarded that first pass rather than publish its number.

Julia-1 (Supersonic Labs, 2026-09-26) is the most carefully documented release in this set and the reason is worth stating plainly: it publishes its own failure. A 144.3M-parameter model built on jhu-clsp/mmBERT-small, Apache-2.0, trained to choose among 2–20 supplied answers, running on a CPU. It ships its evaluation as machine-readable metrics, a provenance.json, an inference-policy.json, and a reproduction script pinned to a dataset revision — and we verified the weights before using them: model.safetensors is 577,189,056 bytes and its SHA-256 is df853bf7fe42…, matching the hash in both their policy file and their reproduction script.

Two things in that paperwork made it the obvious third engine. It reports coverage alongside accuracy, and its Banking77 result lists {"ok": 99, "abstained": 1} — so this is a release that tracks abstention as a first-class outcome. And its inference-policy.json sets "calibration": null, meaning it explicitly does not claim calibrated probabilities. Both are rare. Its card states the weights occupy 550.5 MiB, and they do: 577,189,056 bytes is 550.5 MiB exactly. Running it cost us an afternoon: 577 MB, no GPU, and the same PyTorch stack they used.

The Result​

The consequence for an agent is not symmetric. A wrong tool call is a bad turn. A router that answers 21 of 24 unusable requests is a system that cannot tell the user "I can't do that here" — and this is not a small model doing its best under a hard constraint, because the same model refuses correctly some of the time, which we will come to.

Verdict's failures are the mirror image, and it publishes them: four of its misses were false abstentions — it declined requests that did have a tool, including "build the project with cargo build" and "remember that the store is SQLite with FTS5". Its own documentation puts abstention recall at 97.5% on held-out in-domain data with precision down to 89.45% and 23 false abstentions. That is a calibration trade, and it is the right trade for a router: a false "I can't" costs one clarification, a confident wrong tool call costs the turn and possibly the session.

Confidence Is Not a Refusal Signal​

This is the part we did not expect, and it is the most useful thing in the whole exercise.

Needle's three refusals arrived at 0.986, 0.990 and 0.990 confidence. Its most confident wrong call was 0.891. Whatever the refusal path is measuring, it is not the model's uncertainty: it declines when it is certain there is no tool, and calls when it is uncertain but has a plausible match. So confidence is not merely useless as a scope gate — in the tail it points the wrong way.

Concretely: a gate set at 0.7 would have acted on 9 of 24 requests that no tool could serve. Confidence does separate right from wrong on average — 0.860 for correct calls against 0.622 for wrong ones — and that separation reproduced exactly across two sample sizes. It is simply a different quantity from "is this request in scope", and the two get conflated constantly because both are reported as a number between 0 and 1.

Eleven Releases, Five Families​

The releases are not variations on one design. They are five different bets:

Only one of these families scores 16 of 24 on the refusal test, and it is the one whose schema carries an explicit insufficient evidence route. That correlation is real; the explanation we first gave for it was wrong, and the third engine is what showed us. We had written that a schema of five options makes "none of them" unrepresentable, "no matter how good the weights are". Julia-1 does not have that limitation — and it still refused nothing. Representation is necessary and nowhere near sufficient: a model refuses only if the option is there and it was trained to use it. Verdict's route is trained (its card names the objective, RLCD); the others have never seen an abstain option, so handing them one puts it out of distribution.

The most valuable sentence in any of the eleven releases is Verdict measuring itself against the thing it reimplements: on TypeSafe's public benchmark OpenJev scores 48.07%, against 90.80% for Jev and 88.43% for a 26B diffusion model. A small model honestly reporting that it is half as accurate as the model it copies — alongside a cardinality study showing its own top-1 accuracy falling from 97% at three candidates to 72% at 25 — is worth more than any launch chart. Julia-1's equivalent is a Banking77 pilot at 64 of 100 against Jev's 87%, published next to the three tasks where it wins.

The Option That Did Not Help​

This is the experiment we would run first, if we were starting again, because it converts a correlation into a cause. On the same 24 off-topic requests and the same 11 tools, we ran Julia-1 twice: once with the 11 tool options exactly as every other engine saw them, and once with a twelfth option described as "a request that none of these tools can serve".

Offering the option changed nothing. On all 24 unusable requests it picked a tool anyway — 16 times execute_command, plus three web_fetch, three read_file and two web_search — and on "what is the capital of France?" it reached for a shell command. Adding the option did not merely fail to help; it also cost two points of tool accuracy, which is what you would expect if the extra label is noise to a model that has no idea what to do with it.

The conclusion is narrower than the one we published first, and more useful. A reimplementation can copy the interface in a day, but the interface is not what makes a decision model safe to put in front of a tool list. What makes it safe is a route that is present and trained, and that is a data and objective decision made long before anyone writes a launch post.

The 80 MB File Called "8B"​

CLM arrived with a claim that does not survive its own file listing: an "8B" model whose only weights file is 75.6 MB. Eight billion parameters at fp16 is about 16 GB, so those numbers cannot describe the same object. They don't, and the model card says so plainly: CLM is "two small projection heads on top of a frozen Qwen3-8B encoder".

The arithmetic confirms it exactly. Each head is a projection 4096 → 1536 → 1536 → 512, which is 4096×1536 + 1536 + 1536×1536 + 1536 + 2×1536 + 512×1536 + 512 = 9,443,840 parameters, so the pair is 18,887,680 in fp32 = 75,550,720 bytes against a file of 75,557,149. The remaining 6,429 bytes — 0.0085% — are torch archive overhead, not weights.

So the file is real, correctly described, and not the good news either. The base it requires, Qwen3-8B, is 8,190,726,144 parameters at bf16 = 16,381,452,288 bytes, which is 216.8× the size of the artifact you download. Deploying CLM means serving an 8B encoder on a GPU with vLLM; the 19M of heads are a rounding error on top. It is the opposite of the on-device proposition, and it will never be small.

Two more caveats are in the card rather than the announcement. The headline agentic results — DeepSWE 81.6%, Terminal-Bench 2.1 87.6% — come from fine-tuned heads in a separate repository, not from this checkpoint, which is the starting point for that fine-tune. And its "up to 9×" is the card's own "13× faster with ~1k candidates", which is a caching result.

And the structural point that matters here: CLM cannot abstain at all. Searching its schema, server, engine, client and fine-tuning guide for any notion of abstention or refusal returns nothing. Its yes/no type is strictly false/true, and its choice type is a softmax over whatever candidates you supply — the card's own words are "no generation: CLM only scores the candidates you give it, and its probabilities are relative to that set". On our 24 controls it would return an answer every time, 0 of 24 by construction, and adding a "none of the above" option is a modelling decision the user has to make because the API does not make it.

The Idea Worth Stealing​

CLM's speed claim is not about its architecture being clever. It is that states and actions are encoded separately, so the action side is computed once and reused while the state changes. For a fixed action set that is exactly the right decomposition — and it describes our problem precisely: an agent's tool list is fixed, and what changes per turn is the request. Encode the 11 tools once at startup, encode only the state per turn, score. That works with any scorer in the table above, needs no new model, and is the cheapest real improvement in this entire body of work.

Candidate Count Matters, and Not Monotonically​

One more measurement, because the candidate menu turns out to be a variable rather than a constant. Same 24 inputs at three menu sizes:

Refusal falls hard from 5 labels to 9, then rises at 12. Three points at n = 24 cannot explain that, and we are not going to pretend otherwise — but it is enough to say the effect is real and not simply "more options, worse", which is consistent with Verdict's own note that its abstention "depends heavily on the seen prompt structure". Their published curve, on 100 samples per slice, is the version to trust: 97% → 96% → 91% → 78% → 72% accuracy as the menu grows from 3 to 25. An 11-tool router is a 12-candidate problem, which is where their own data already says to be careful.

A Month Later, the Field Has a Leaderboard​

Everything above measured ten reimplementations, none larger than 151M parameters. That is no longer the shape of the field. Decision 3.0, released on 10 October by the vLLM Semantic Router team, is six Apache-2.0 decision models from 0.59B to 26.09B, built on Qwen3.5 and Qwen3.8 backbones, multimodal (images and video), and published as zero-shot-classification rather than text generation. Its card states the design intent in one sentence: the model "returns a probability for every answer, without generating text."

So the interface we said takes an afternoon to copy is now a category label, and the category has a public number attached. The Jev Decision Index 0.3.1 board runs on a suite of 140,178 requests:

The reference implementation is now fourth on a board Perplexity and Fastino have entered. The column that should interest anyone building on this is New-domain, where even the leader reaches only 56.6 — that is the generalisation gap we could not measure with 24 hand-written controls and they now attempt with 140,178 requests. The size ladder tells its own story too: the same family scores 28.82 at 0.59B against 65.45 at 27B.

The Line to Interrogate​

Both cards lead their highlights with the same sentence: "all 140,178 public requests answered, none unsupported."

Read that against the test above. A decision model that never declines is a guessing machine with a probability attached — that is precisely what we measured, where three engines refused 3, 16 and 0 of 24 requests that no tool could serve. Whether "none unsupported" is an achievement or a confession turns on one question the card does not answer: does the 140,178-request suite contain cases where the correct answer is "unsupported"? If it does not, the index cannot measure the behaviour that separated the engines here, and the claim is unfalsifiable rather than good. That is the control-set rule from the claimaudit post, pointed at the industry's own benchmark — and it is the first thing we would check before trusting the board.

Can These Models Emit a Word at All?​

We had asserted "no", so we measured it. The artifact settles half the question: a released Jev-style checkpoint carries no language head. Of the four pieces ModernBertForMaskedLM needs to produce a vocabulary distribution, three are absent — head.dense, head.norm, decoder.bias — leaving only the tied embedding table, which is a shared input tensor. Nothing in the file outputs a word distribution; the model returns one scalar per marked option.

The other half needed a control. We loaded the parent jhu-clsp/mmBERT-small, which does carry the full MLM head, then swapped in only the post-trained encoder — 134 of 134 tensors changed:

Three things fall out of that. The instrument works — the parent produces clean continuations including at the end of a sentence, so a masked LM answering "the next word" is a legitimate query and the distribution-shift objection we expected does not apply at single-step. The post-training did not destroy the language ability — a guard lowering his gun is a better answer than the parent's "horse". And it is unevenly degraded: "the north ___" collapses to america and beauty, so the encoder has been specialised away from free-form completion.

So: yes, mechanically — and only with a language head bolted back on, which costs about 98.7M parameters for the parent's untied decoder. That is more than double the 42.2M in all 22 encoder layers, entirely because the vocabulary is 256k. And it is single-step only: generating means re-encoding the whole prefix for every word, with no cache. Which is the conclusion the vendor reached from the other side, in the sentence quoted at the top of this section.

The Conformal Set That Abstains on Everything​

One more claim deserved the same test, because it is the one that would matter most. A poster for a 0.8B competitor, vega, advertises a conformal set at a chosen risk level α, plus abstain and unbound flags — abstention by construction, with a coverage guarantee, rather than a hope that the model learned to decline. If that worked it would answer this whole post.

We ran it on the same 24 in-scope and 24 off-topic requests, the same 11 tools, at three risk levels — a conformal set shrinks as α rises, so one α is not a result:

abstain fires on 22 of 24 requests on both sides. Every row below 0.4 confidence abstained — 38 of 38 — and 6 of the 10 above it did too. Mean confidence separates off-topic from in-scope by 0.030 (0.298 against 0.328). The flag is a confidence gate, not a scope judgement, and it is the same wall this post documented for Needle — now in the model whose headline feature is principled abstention.

The conformal set is a cliff, not a curve. At α ≤ 0.1 it returns a singleton or a pair: never empty, and never wide either, so it cannot say "I can't tell". At α = 0.20 it returns None for all 48. There is no α at which the set does the job.

The likely reason is in the card: "each adapter route carries its own calibration vector." This checkpoint is built for a public-intents label space, and an 11-tool router is outside that route, so its calibration and conformal tables do not apply. The package ships fit() — five-fold, and it refuses a head at or below chance — which is the documented way to adapt it. So the honest conclusion is narrower than "it does not work":

Conformal abstention is not portable to a new vocabulary without refitting.

That is this post's own lesson a third time: a route that exists is not a route that works. It matters because "conformal set + abstain: yes" reads as a capability you get, not one you have to fit.

On speed we will not score a point we did not earn: the run took 551 ms on an Apple M1 with 16 GB, but both fast kernels were missing and the runtime says so — "correct but much slower" — so that is an upper bound, not a like-for-like test of the poster's 75 ms. The contradiction worth recording needs no hardware of mine: the model's own card says 280 ms on Apple M-series.

What It Cannot Do​

• It is 24 hand-written requests per side, not a benchmark. The differences are large enough to act on; they are not precision measurements, and the sweep above shows how sharp the sensitivity to framing is.
• One tool surface, one language, one working domain. These models are classifiers over text. Every number here is out-of-domain for the encoders involved.
• Nothing was fine-tuned. All three engines ran zero-shot. The largest advertised gains in this field come from per-product fine-tuning, which we did not do.
• No engine was tested inside a live agent loop. This measures whether the decision is right, not whether a turn improves.
• CLM was read, not run. Reaching it means vLLM plus 16 GB of Qwen3-8B, which is not worth an afternoon to confirm something its own schema already settles.
• Julia-1's own abstentions do not come from the model. Its metrics report coverage and an abstained status on Banking77, but the word does not appear in any code file it ships, and its reproduction script always returns a choice. Whatever produced those abstentions lives in a Banking77-specific harness that is not in the repository.
• Julia-1's probabilities are declared uncalibrated and are then cosmetically sharpened. Its policy file sets "calibration": null, and a presentation step rounds a winner above 0.95 with all others under 0.045 to a one-hot 1.0. That is a display rule, disclosed in the docstring, and it is why confident answers read as exactly 1.000 in our table. Do not read those ones as measured certainty.
• Decision 3.0's own numbers are its own. Its card marks d3's results internal evaluation while the competitors' come from live board data, and the card does not say who operates the board. Treat the ordering as informative and the absolute values as unverified.
• The index's floor is unstated. 28.82 at 0.59B against 64.7 at 26.09B is a large gap, but with no published floor neither number means anything on its own, and we are not going to guess one.
• Decision 3.0 loads with trust_remote_code=True. That runs code shipped in the repository, so it is a supply-chain decision rather than a model download, and it is worth treating as one.
• A conformal set is only as portable as its calibration. vega's tables are fitted per adapter route, so on a vocabulary it was not fitted for the set is a singleton or empty rather than a measure of doubt — measured above, at three risk levels.
• A refusal is not a safety property. "Nothing applies" is a scope judgement. It is not a check that a request is safe, and nothing here should be put in a security path in place of a deterministic one.

What We Learned​

What worked: three things, all cheap. The first is the control set — 24 off-topic requests that no tool can serve — which is the only reason "3 of 24" is a number at all rather than a story about a model that seemed fine. The second is running the vendor's own example before judging it on ours; that single control separated "broken model" from "mis-shaped test" in one run, and the first number we would otherwise have published was wrong. And a third, quieter one: sometimes the answer is in the source rather than the weights. CLM's inability to refuse is not a measurement, it is a grep, and it took a minute. Julia-1 made the same point from the other direction — its published SHA-256 let us verify 577 MB of weights before running anything, and its own policy file told us not to trust its calibration before we had formed an opinion about it.

What didn't — nine of our own instruments, all caught before publication. A C API whose return value is not a byte length, which silently truncated every response into what looked like a model that refuses everything. An off-by-one in our own validator that marked every call as ungrounded and turned a real signal into noise. A 12-label test that was mis-shaped for the model's label template. A summary line that reported "4 refusals" in a run where zero occurred, and whose first fix reported 3 — both wrong, in a log that a reader would have believed instead of the rows. A sample of 12 that let us write "it never refuses" when 24 makes it refuse three times. A sweep we declared unreproducible and marked do-not-cite because we had read its log file while the job was still running; the file was complete, and we had condemned valid evidence on a stale read. A version pin we ignored: Julia-1 declares transformers>=5.0,<5.1, our shared environment had 5.17, and their encoder patch broke in a way that looked like a model failure until we built the environment they actually specified. A language-head probe that returned pure subword soup on every prompt — which was briefly read as "the post-training destroyed the vocabulary geometry", until a control on the parent model showed that we had skipped the prediction head ModernBERT applies before the decoder, and the soup was our own. And a parser that assumed a field was never null: the conformal probe crashed at α = 0.20 because prediction_set comes back as None there, and took two completed alphas down with it — fixed by normalising the answer and writing results after each one. Three of those generalise. A background job's output is not evidence until the job has exited. When a release pins a dependency range, that range is part of the artifact. And an optional field is optional: normalise the answer before you parse it, and never let a late failure discard work you have already done.

The uncomfortable part: we published an explanation and the next release falsified it. Our claim was that abstention is a property of the schema — put a "none of these" option in the contract and the model has somewhere to put the answer. The first half is true and the second half is not: Julia-1 was offered exactly that option and used it zero times in 24, then lost accuracy for having been given it. What we got wrong was treating representability as sufficient, when it is only necessary — Verdict's route works because it was trained with an objective that rewards honest abstention, not because the label existed. That is a narrower claim than the one we first made, and it is the one the evidence supports. It is also the reason to run a control you did not design the model for: the 24 requests could only ever have shown us that the engines differ, and it took adding one option to a third engine to show us why.

Every number above is re-read from committed run artifacts by scripts/check-openjev-post.py — the Needle counts are parsed out of the harness logs, the Verdict and Julia-1 counts out of the per-request JSON, and the CLM arithmetic is re-derived from its config. It fails if any figure drifts, or if a figure appears in this post with no derivation behind it. scripts/mutation-test-openjev-checker.py corrupts the post ten ways and requires every corruption to be caught, so that the checker is not merely a thing that passes. --live re-fetches the pinned vendor claims, which are the parts that can change under a published post.

Updating this post. This is meant to be re-run, not rewritten: the harness, the evidence rows for all three engines, and the checker are all committed. To add a model, run it on the same 24 + 24 requests against the same 11 tools, add its rows, add one line to the family table, and let the checker decide whether the prose still holds.

Updated 2026-09-26: Julia-1 added and measured, which required correcting the explanation this post first gave for why abstention separates the engines. The correction is in place above rather than in a footnote, because we would rather the post be wrong in public for a day than quietly right later.

Updated 2026-10-11: the field acquired a leaderboard. Decision 3.0 and the Jev Decision Index are added above, with the one line on that board we would not accept without a control set — plus our own measurement of whether a Jev-style model can emit a word at all, which is the question the whole category is defined by refusing to answer. Nothing in the measured core changed.

Updated 2026-10-18: we tested the strongest version of the abstention idea — a conformal set at a chosen risk level, which would abstain by construction rather than by training. On our 24 + 24 surface it abstained on 22 of 24 requests on both sides, and its conformal set was a singleton or nothing at all. The mechanism is real; the calibration behind it does not travel to a new vocabulary without refitting. That is the same lesson as the rest of this post, so it belongs here rather than in a separate note.

The Question, and the Wrong Answer​

The prompt was a video arguing that rare earths aren't actually rare. That is true, and it led somewhere unexpected. The interesting question underneath was whether a solo person with a PC and no lab could take an abundant, undervalued material and transform it into something valuable.

No. Materials work is gated three ways: knowledge (a simulation can sometimes close this), a wet lab (it cannot be closed from a keyboard), and capital (a plant is not a weekend project). Almost every candidate lands behind at least two of the three. Rare-earth-free magnets are the clean example — the physics is fine, every contender has adequate intrinsic magnetism, and the blocker is synthesis and phase stability, which is bench work.

But three independent research passes looking into it all landed on the same bottleneck, and it wasn't materials. It was that nobody verifies the numbers. Claims circulate through press releases, trade coverage, and citation chains, and almost nobody does the two-line arithmetic that would show a claim cannot be true.

That is a PC-first, zero-capital activity with a real intellectual core, so it became the project.

The Rule That Made It Work​

The tool audits a claim by computing the hard ceiling its own physics imposes, from published constants, rather than by comparing it to competitors. Three design rules turned out to matter more than the physics:

• Derive the ceiling, don't cite it. Shockley-Queisser isn't quoted from a textbook; it's computed by detailed balance over an embedded copy of the real ASTM G173 AM1.5G spectrum. Integrating that embedded table returns 998.8 W/m² against the 1000.4 W/m² the standard documents — 0.16% apart, which is the check that the spectrum and the integrator are both right before any efficiency is computed.
• Keep a control set of real shipping devices the audit must never flag. This is the rule that did all the real work, and it is the one we would have skipped.
• Keep the naive bound as a regression test. Every domain starts with the wrong version of its own ceiling. Keeping it around, and asserting that real hardware breaks it, documents exactly why the rigorous version exists.

Plus a fourth: verdicts that refuse. When a bound doesn't apply, the tool says so instead of quietly widening until the claim passes.

Three Domains, Three Different Failures​

The three domains were chosen so that each tests something the previous one couldn't.

Magnets. The remanence ceiling Br²/4μ₀ gives 56.3 MGOe at Br = 1.5 T, but that ceiling only applies if the coercivity is high enough. With coercivity included, the binding limit is Br·Hcj, and at the 2.1 kOe coercivity actually reported for a 36 MGOe claim, the ceiling is 31.5 MGOe. The claim exceeds it by 12% and is flagged IMPOSSIBLE. At the company's own higher target coercivity the ceiling rises to 49.5 MGOe, the claim fits — and a second check then fails it on a different ground: reaching it requires a squareness of 2.91× the straight-line relation, against a best-ever-measured 2.79×. The best peer-reviewed measured result in that family, 20 MGOe at 1.91 kOe, passes cleanly at a 28.7 MGOe ceiling. Three claims in one family, three different verdicts, all from arithmetic.

Solar. A widely reported story said solar cells had hit 130% quantum yield, "smashing a 65-year limit." As a power-conversion efficiency that would be impossible — the single-junction ceiling peaks at 33.7% near Eg = 1.34 eV, and 130% exceeds it roughly fourfold. But quantum yield is not efficiency. It counts excitations per absorbed photon, and it legitimately exceeds 100%: singlet fission turns one singlet into two triplets, so the real cap is 200%, and 130% sits comfortably under it. The same number gets opposite verdicts depending on which quantity is meant, which is why the tool has a NOT_APPLICABLE_FOR_SQ verdict and uses it.

Heat pumps. This is the domain that inverted the tool's posture. The dominant public error here is not an overclaim but a naive debunking: a heat pump rated COP 4.8 gets reported as "480% efficient" and dismissed, because nothing can exceed 100%. But COP counts heat moved per unit of work, and the heat comes from outside the building. The naive rule rejects all 8 of the real, manufacturer-datasheet heat pumps in the control set — and the second law accepts all 8. The best real machine reaches 57% of its Carnot allowance; the tool flags claims above the envelope, not above 100%.

The domain-specific trap is subtler than the naive rule. The ceiling is only as tight as the temperatures you feed it. Use indoor room air as the hot reservoir and a claim of COP 12 sails through; use the delivery-water temperature the machine actually produces and the same claim is IMPOSSIBLE. The tool therefore refuses to audit a COP claim that doesn't state its temperatures. In the wild, that is the most common form — "COP up to 5.1" — and it is unverifiable as written.

The same corpus also contains this domain's version of the magnet failure: manufacturers' seasonal ratings reach 5.65, while the government's own monitored field trial median is 2.80 across 742 homes. Both numbers are honest. They are not the same quantity, and presenting the first as the second is the same move as presenting a development target as a measured result.

The Control Set Is the Product​

The finding worth keeping is not any of the physics. It is that every domain's first bound attempt false-positived on real shipping hardware.

The naive magnet bound, Br·Hcj/4, condemns shipping sintered ferrite. A single-junction-only solar check condemns certified tandem cells that legitimately run above the 1-junction curve. The naive heat-pump rule condemns every heat pump ever built. In each case the physics was fine and the bound was the thing that was wrong, in the direction that makes a tool look impressive and be useless.

Only the control set caught it, every time. And that is why the same failure mechanism never repeats across domains while the shape of the failure always does — so this generalises as a per-domain discipline with a control set, not as one clever general tool.

Then We Asked Who Would Pay​

At this point there was a working tool, 49 passing tests, and four of those tests defending real devices against false positives. The obvious next move was a fourth domain. The useful move was to stop and ask whether anyone buys this.

So we wrote the kill criteria down first — committed, dated, before any research — because a criterion written after seeing the data is a rationalisation. The question was narrow: is there a buyer who will pay real money for auditing a technical claim against the hard ceiling of its domain? Split into four independently falsifiable parts: pain, payment, reachability, scalability.

The verdict is kill. Three of three stopping conditions met; two of three continuation conditions, failing the one that matters.

What the Money Actually Looks Like​

Verification is paid for, generously, and the price ladder is the clearest thing we found:

Every one of those is a different job. Not one of them is "tell me whether this claim is physically achievable." The best-funded adjacent service is Exponent Inc., a Nasdaq-listed company whose explicit business is technical due diligence, billing a rate ceiling that rose about 40% in three years. That is unambiguous evidence the skill is valuable — and it is sold through firms and credentials, contracted enterprise-to-enterprise, never self-serve.

Meanwhile the open-ended version of the job is done for free at scale: we catalogued 28 free artifacts doing this work (expert blog posts, trade-press critiques, forum arithmetic, YouTube debunks, publicly funded null replications) against 13 paid services in adjacent slices and zero paid products selling the open-ended verdict to a general buyer. The public-interest version is philanthropy-funded: Retraction Watch ran the leading claim-verification database as, in its own words, "a volunteer activity" for 12 of its 13 years, then abandoned its licensing revenue model entirely in September 2023 and handed the database over to be made free.

The position is unoccupied for individuals and fully occupied for institutions — and the institutions are behind accreditation, professional-indemnity insurance, tender turnover thresholds, court qualification, or expert-network seniority screening. Identifiable buyer, unreachable buyer.

Why It Is Dead​

Three reasons, and the first one is the one that would be hardest to fix.

The tool is pointed at the wrong failure mode. A ceiling audit diagnoses sincere over-belief — someone who genuinely thinks their impossible thing works. The documented losses are overwhelmingly intentional misstatement: the SEC charged fraud, requiring scienter, in case after case, and in one instance a company claimed 87 vehicles sold when the number was zero. Fraud is caught by document forensics, whistleblowers and short sellers. None of those is a physics service.

The sophisticated buyers already verify in-house and walk away. When a major automaker diligenced a hydrogen-truck startup with an "army of people," it restructured the deal and dropped the equity stake rather than buying an audit. When another ran battery cells through its own lab before investing, that was in-house capability too. The people with money to lose and the discipline to check already check.

The free supply is fast and correct. A contested superconductor claim went from announcement to expert consensus in about two weeks, for free. An impossible propulsion claim died to a single null replication from a publicly funded lab. Envia's battery claims died to a customer's own test bench plus trade press. Competing with €0 supply that is fast and usually right is not a business.

What It Cannot Do​

• It cannot tell you a claim is true. CONSISTENT means internally consistent with the physics, not verified. A claim can pass every bound and still be a lie.
• It cannot audit an under-specified claim. No temperatures, no conditions, no numbers — no verdict. That is the majority of marketing claims.
• It cannot see the thing that actually costs money. Intentional misstatement is invisible to a physics bound.
• It cannot scale across domains for free. Each new domain needed its own derivation and its own control set. The reusable part is the method, not the code.
• It did not find a buyer. The two load-bearing negatives — nobody asks, and no well-funded product died in this space — are absence of evidence, not proof.

What We Learned​

What worked: deriving the ceiling rather than citing it, and controlling against real hardware. Both were cheap, and together they are the whole method. The specific rule that paid off most is keeping a set of devices the tool must never flag — without it, three separate domains would have shipped a confidently wrong bound that condemned real, shipping products. A tool that only ever says "no" is a debunking machine; the control set is what makes it an auditing capability.

What didn't: the assumption that a working, rigorous, error-finding tool is adjacent to a business. It isn't adjacent to anything. Building three domains taught us nothing about willingness to pay, and the honest sequencing is the reverse of what felt natural — write the criteria down, then check, before the sunk cost makes the question feel like an attack on the work. Once the tool existed and worked, the temptation to reinterpret a negative result as a reason to build a fourth domain was very strong, and the only thing that blocked it was the criterion already being committed to git with a date on it.

The uncomfortable part: the market we found is real and well paid, and it is a labour market rather than a founder market. Engineering consultancies bill hundreds of dollars an hour for exactly this judgement; expert networks pay experts a few hundred an hour and charge clients several times that. The work is worth a great deal per hour and nothing as a solo product, because the buyer contracts with a firm and a credential, not with a person who has neither. That is a different problem from "the idea was bad," and it is not solvable by executing the same idea better.

The tool survives as a capability, and it has already earned its keep twice — once by showing a magnet headline was arithmetically impossible at its own reported numbers, and once by showing that a solar "breakthrough" was a definition being swapped rather than a limit being broken. Both were public claims that had travelled a long way on citation. Neither needed a lab, a budget, or anyone's permission.

The numbers in this post are read out of committed run artifacts and re-derived by scripts/check-claimaudit-post.py, which re-runs the audit tool's own ceilings, re-integrates the AM1.5G spectrum, and checks every market figure against the evidence document it came from. It fails if any of them drifts.

The audit tool and its research streams are not published. The evidence behind this post lives with the tool in docs/VERDICT.md and docs/evidence/ — four research streams with every claim labelled fact, inference or not-found.

First, What Is Actually Established​

Everything below distinguishes three tiers of evidence, because the interesting part of this story is how they differ.

Primary, from Taalas (product page, Bajic's launch post): 815 mm², TSMC 6nm, 53 billion transistors, 17k tokens/sec/user on Llama 3.1 8B, 1k-in/1k-out. Also on the record: the quantization is a custom 3-bit base type with 3-bit and 6-bit parameters mixed, which Bajic states "introduces some quality degradations relative to GPU benchmarks."

Primary, from AMD (newsroom): a definitive agreement, "subject to customary closing conditions and regulatory approvals." The release does not state a price, and the deal is not closed — so the shorts are slightly ahead of reality.

Where the comparison numbers live, and a correction. The competitor figures come from a chart image on Taalas's own site, which means they cannot be quoted from page text and have to be read. We read it: Taalas HC1 16,960 · Cerebras 1,981 · SambaNova 932 · Groq 594 · Nvidia B200 353 · Nvidia H200 230 tokens/sec/user. The B200 figure is Taalas-measured; the Groq, SambaNova and Cerebras figures are sourced on that chart to Artificial Analysis. An earlier draft of this post used 350 and 200 for the two Nvidia baselines, taken from secondary coverage rather than the chart. Those were low: the correct baselines are 353 and 230, which makes the head-to-head ratios 48× and 74×, not 48× and 85×. Taalas's own written claim is the more modest "nearly 10X faster than the current state of the art" — which checks out against Cerebras at 8.6×.

Third-party: The Next Platform got the only substantive technical interview. The founder history is worth knowing: Ljubisa Bajic founded Tenstorrent, a general-purpose AI accelerator company, then left to build Taalas around the opposite bet; all three co-founders came from Tenstorrent, and before that AMD.

Not established: there is no independent third-party benchmark. The B200 baseline was run by Taalas.

The Ceiling That Explains Everything​

LLM decoding is not compute-bound. To emit one token you must touch every weight exactly once, and at 16-bit precision that is 16.06 GB of data that has to be hauled out of DRAM per token, forever. This puts a hard ceiling on any conventional accelerator, from any vendor:

That table is the whole story, and it cuts both ways. Read one way, it says modern GPUs are nowhere near the theoretical limit. Read the other, and it says they are already at 71–77% of a wall that physics put there — which is a well-engineered chip doing its job, not a chip that can be optimised into 17,000 tok/s. You cannot get to 17,000 by making the memory faster. You have to stop reading the weights.

The self-consistency here is worth noting, and it survives the correction above. Taalas's published Nvidia baselines sit at 71% and 77% of the ceilings computed from published bandwidth — that is, they are generous to Nvidia. A vendor inflating its own ratio publishes low competitor numbers; Taalas published high ones. Their 48× is therefore a more conservative claim than it needed to be.

A Weight That Is Not Data​

The mechanism is a mask ROM. In a mask ROM a bit is not a charge in a cell — it is the presence or absence of a contact at a wire crossing. Value is geometry in a metal layer. Bajic's description of the consequence:

"We have got this scheme for the mask ROM recall fabric — the hard-wired part — where we can store four bits away and do the multiply related to it — everything — with a single transistor. So the density is basically insane. And this is not nuclear physics — it is fully digital. It is just a clever trick that we don't want to broadcast."

The lever is that a weight is a constant, fixed at fab time. Multiplying by a constant is not really arithmetic — it is a lookup. So there is no multiplier array to build (Bajic calls multipliers "the big boy piece of the computer") and no weight movement at all. Only activations move, and only across the die. The architecture pairs that immutable ROM fabric with a mutable SRAM fabric holding the KV cache and LoRA adapters — the model's skeleton is silicon, its mutable state is not.

Attributed where it belongs, and still unconfirmed. Taalas is explicit that it is not disclosing the fabric ("we don't want to broadcast"). The circuit-level story that circulates in secondary coverage — that the chip pre-computes all sixteen products of a 4-bit slice at manufacturing time and lets pass transistors select the correct pre-wired channel — is exactly that: secondary coverage, not a Taalas statement. It is a plausible reading of "one transistor stores four bits and performs the associated multiply," and it is not ours to assert. The functional claim is Bajic's, on the record. The mechanism behind it is not.

Two physical consequences follow, and they are why 200 W is credible. Multiplication by a constant bit decomposes into a conditional wire, so the dominant energy term disappears; and a ROM cell that is not being read does not toggle, so dynamic power tracks only the activations moving through. The resulting power density is 0.25 W/mm² (200 W over 815 mm²) — low enough that standard air cooling suffices, and low precisely because the data is not moving.

Does the Transistor Budget Add Up?​

This is the one part of the design that is externally checkable, so it is worth checking. Take Bajic's own ratio — four bits stored and multiplied per transistor — and Taalas's own quantization statement (3-bit base, 3-bit and 6-bit mixed, so call it ~4 bits per parameter on average):

The arithmetic closes on the stated figure, which is not nothing — a fundamentally different storage scheme that came out three orders of magnitude off the announced transistor count would be a problem. Note what the 15% is and is not: it is the transistor share, not the area share, and ROM cells and logic have different densities, so the two are not interchangeable. What it does say is that the weights are a minority of the silicon: the rest is the compute fabric and the mutable state, and the KV cache has to live in that budget — which is why Taalas's stated flexibility ("configurable context window size and LoRA fine-tuning") is bounded by how much SRAM was sized at tapeout.

A useful contrast: the B200 carries 208 billion transistors. Its die area is roughly 1,600 mm² across its two dies, which puts it near 130 MTr/mm² against HC1's 65 MTr/mm² — about double. That area figure is our estimate, not a published NVIDIA number, and the comparison is order-of-magnitude only. Even loosely, though, it makes the point: the ROM chip is less dense, not more. Density was never the problem being solved. Not moving the data was.

Three Numbers Called "Efficiency"​

This is where the headline claims have to be taken apart, because three quite different figures get flattened into one.

Latency: ~74×, and it is structural. 17,000 tok/s is 58.8 µs per token, which across 32 transformer layers is 1,838 ns per layer — for RMSNorm, QKV projection, attention and the MLP combined. For contrast, take the same ceiling table and ask what one layer costs on an H200: 502 MB of weights per layer at 4.8 TB/s is 105 µs of pure memory time, before a single multiply. That is a 57× gap in memory time alone, and no amount of batching closes it, because it is per-token serial latency either way. End to end against the H200's 230 tok/s the ratio is 74×; against the B200's 353 tok/s, 48×. It is the most defensible number in the whole story.

Energy per token: a few times, not a thousand. HC1 spends 11.8 mJ per token at the stated 200 W per card. An H200 serving one stream at 230 tok/s spends 3,043 mJ per token — 259× more. That is the comparison the headline invites, and it is the wrong one, because batching exists to amortise exactly this. A batched H200 at 20,000 tok/s aggregate is already at 35 mJ per token, only 3× behind. That number is the load-bearing assumption in this post, so here is the sensitivity rather than a single figure:

And a second caveat running the other way: 200 W is the per-card figure from the Next Platform interview, while the product page quotes 2.5 kW for a ten-card server — which is 250 W per card including its share of host, NICs and fans. On the higher figure HC1 is 14.7 mJ/token and the honest ratio becomes 2.4×. So the real answer is a range of roughly 2.4× to 3×, and it is a range because both inputs are.

The 1000× is a utilisation figure, and it has a range too. Take the B200 at 353 tok/s. That is 353 × 16.06 GFLOP = 5.67 TFLOP/s. Against a dense FP4 peak of 9,000 TFLOP/s that is 0.063% of peak — invert it and you get roughly 1,600×. But the denominator is the one place this arithmetic can be pushed, because the B200 never executes in FP4 at single-stream: at FP8 the peak is 4,500 TFLOP/s and the ratio is ~800×, and at FP16, the precision actually being executed, the peak is 2,250 TFLOP/s and the ratio is ~400×. So the honest reading of "1000× more efficient" is: it is the reciprocal of GPU utilisation at batch size 1, and it brackets between 400× and 1,600× depending on the precision you pick. That range is a real result — the GPU genuinely runs at a fraction of a percent of peak on a batch-1 decode, because it is starved for bandwidth — and it is not a claim about joules. On joules per FLOP at peak, the B200 is the better chip: 9.00 TFLOP/J against HC1's 1.37 TFLOP/J.

The physical bedrock under all of this is Horowitz's ISSCC 2014 numbers, and they are brutal:

Reading a single weight from DRAM costs many times more than the arithmetic it feeds, and a weight is sixteen bits. Delete the read and most of the bill disappears. That is the entire thesis, and it is a real one.

What It Cannot Do​

• One model per chip. A new model is a new tapeout. Only two metal layers change, so it is far cheaper and faster than a full design — Taalas quotes two months from an unseen model to deployable cards — but it is a hard discontinuity, not a config file.
• Quality is knowingly traded. Taalas's own words on the 3-bit/6-bit mix: "some quality degradations." Comparing a degraded 8B model at 17k tok/s to a clean one at 230 is not comparing like with like.
• Long context is the weak axis. The KV cache grows per token and lives in fixed on-chip SRAM. Window size is configurable within that budget, but the budget itself is fixed at tapeout.
• Mixture-of-experts inverts the economics. This is our argument rather than a sourced claim. MoE activates a small fraction of weights per token, so a GPU's bandwidth bill scales down with the active fraction. A ROM chip must bake all the experts into silicon to serve any of them, then traverse only some — the full area cost, a fraction of the benefit. It is the architecture that most directly attacks the premise.
• One die is one die. At 815 mm² the part is already at 95% of the 858 mm² reticle limit — the standard single-exposure field for 193i lithography. Anything larger than a mid-sized model has to be pipelined across chips.
• No independent verification. Vendor-run benchmarks, as of the launch.

Why AMD Bought It​

Two things are happening at once. The obvious one: The Next Platform reported that Nvidia had just bought Groq for ~$20 billion, and AMD has no equivalent asset and no CUDA moat to defend. Buying a decode-specialised chip is differentiation by a different route, and AMD's stated plan — integrate it alongside Instinct GPUs in Helios racks — is a division of labour rather than a replacement. GPUs keep prefill, training and flexibility; the hardwired part eats the serial-decode latency problem that GPUs cannot structurally solve.

The less obvious one is who is being bought. Bajic was an AMD architect on hybrid CPU-GPU designs before founding Tenstorrent; his co-founders came through AMD and ATI as well. On the economics, the framing from Taalas is that hardening a model costs about one hundredth of training it. That only works if you need to serve that specific model for long enough to amortise the tapeout — a bet that models stop churning every three months. Llama 3.1 8B is still being served today, years after release, and frontier release cadence has been lengthening. It is a bet with evidence behind it, which is not the same as a safe one.

What We Learned​

What worked: the derivation. Almost every dramatic number in this story is reproducible from constants that are published — parameter count, HBM bandwidth, TDP, transistor count, die area — and doing that arithmetic first is what separated the three "efficiency" claims from each other. The 299 tok/s ceiling is not Taalas marketing; it falls out of dividing two public specs, and it is why the 17,000 figure is plausible rather than absurd. Likewise the transistor budget closing at 15% weights and 85% everything-else, and HC1's power density landing low in a way consistent with the claim that nothing is moving.

What didn't — and what an independent audit caught. Our first draft took its Nvidia baselines from secondary coverage instead of from Taalas's own chart: 200 and 350 tok/s where the chart says 230 and 353, which turned a 74× ratio into a claimed 85×. The baselines were not on the page as text; they only exist inside a chart image, and we had not read the image. Three other things did not survive review either: we had attributed the "1000×" claim to the AMD announcement when it is only on Taalas's homepage; we had described the pre-computed-product mechanism as our inference when it is actually circulating in secondary coverage and should have been attributed as such; and we had credited Taalas with a first version built on general-purpose silicon, conflating it with Tenstorrent, which is a different company. All four are now fixed above. The lesson is narrow and repeatable: when a number only exists inside an image, a text extraction will silently substitute someone else's version of it, and every ratio computed from it inherits the error.

The uncomfortable part: the design removes one constraint and promotes another. Once nothing is bandwidth-bound, what is left is per-layer serial latency, power delivery, and the reticle limit — and then the deepest constraint of all, which is that the model is frozen. Whether that is a good trade depends on a prediction about how fast models will keep changing, which is a prediction about the world rather than about silicon. The hardware is sound. The bet is the interesting part.

The arithmetic in this post is re-derived, not transcribed: every figure is recomputed from the published constants above by scripts/check-taalas-post.py, which will fail the build if the numbers drift.

Why Bother Asking​

The dataset is real, published, and awkwardly large in a specific way: the weights table has 151,856,684 rows, and only 25,563,197 of them survive filtering to neurons whose reconstruction has actually been proofread — 16.8%. The rest is segmentation debris. FlyOS converts the survivors into a compact binary format (198 MB) with byte-identical reproducibility: two independent implementations, in different languages, agree on every neuron, edge, and synapse count.

On top of that sits a simulator where the same input produces the same output every time, down to a state hash. That is not a nice-to-have. Without it, every comparison below is meaningless, because you cannot tell a structural difference from a random one.

How to Ask It Without Fooling Yourself​

The honest comparison is not "real brain versus random noise" — that is a strawman that will always be won. It is random rewrites of the real brain that preserve different amounts of its structure:

Each rung keeps a bit more of the truth, so instead of one dubious comparison you get a ladder — and you can attribute a behaviour to the kind of structure responsible. Five replicates per family, all byte-reproducible from seeds.

What the Ladder Found​

Ignition threshold rises monotonically as structure is destroyed:

And the response sharpens at the same time. The real brain behaves like a dimmer — raise the input and activity climbs gradually. Every randomised rewrite behaves like a switch: dead, then abruptly 97% active. The ordering matters more than any single comparison, and it only exists because there are five rungs rather than one.

Testing whether this was all just excitatory/inhibitory balance eliminated that explanation — the sign-preserving control has the same threshold as the degree-matched one. What it revealed instead is that the mechanism splits into independent axes: threshold is set by the coarse cell-type layout, the runaway ceiling by E/I balance, and oscillation strength by fine wiring.

The Result That Stood Out​

Feed the network a 20-millisecond pattern of activity, then let it run free. 400 ms later you can still tell which of eight patterns it was — 80 to 95% accurate. Random rewrites of the same brain, at matched activity: 7.5 to 25%.

The confound we had to rule out first was mundane: with only 200 stimulated neurons against a 5,000-neuron readout, chance overlap is 6 neurons — 0.121% of dimensions. That cannot produce 80% against a 12.5% ceiling. The decoded state is propagation, not the stimulus.

Five Times We Were Wrong​

This is the part worth reading carefully, because it is the difference between this and a demo that always works.

One. We reported that the network's ~108 ms rhythm "requires biological wiring". Then we noticed every comparison had used the same absolute input level, while the graphs ignite at different levels — so the controls were below their own threshold, silent by construction. Compared fairly, the rhythm is near-threshold behaviour every graph shows. Retracted.

Two. We reported the real brain's threshold as 2.0× lower than a degree-matched rewrite. A finer measurement grid gave 1.71×. Corrected.

Three. We reported that only the biological network could encode a stimulus at all. That was our decoder being too weak — with common-mode removal and a ridge classifier, the controls reach 100% immediate accuracy. What survives is narrower and more interesting: everyone encodes; only the real brain still has it 400 ms later. Retracted.

Four. We hypothesised the memory came from a property of the connection matrix, the standard result in recurrent network theory. It doesn't. The prediction missed by 25×, and the ordering inverted: graphs with virtually identical spectral fingerprints — the four degree-preserving families sit within 0.7% of each other — differ 5× in how well they remember. Hypothesis dead.

The pattern is consistent and worth naming: we kept over-claiming in the direction of "the biology is special", and only better instrumentation caught it. Not once did a result get stronger under scrutiny.

The Test the Survivor Passed​

The memory result had an obvious vulnerability. Every neuron had been given identical parameters, and identical units synchronise artificially — so the memory could have been an artefact of that uniformity rather than of the wiring.

So we jittered every neuron's decay and threshold by ±10%, ±20% and ±30% — five runs, each measured at three activity offsets, so fifteen measurements. (Three independent random draws exist only at ±20%.) If it was an artefact, it should collapse.

It survived everything. Then we attacked the other idealisation the project's own assumptions flagged as most distorting: conduction delay. Real neurons take time to signal; in this model, signals arrived instantly, which the assumptions recorded as overestimating synchrony.

Retention survives a 30× range of the assumption most likely to destroy it, and the contrast holds: at 10 ms, the real brain retains 60–77.5% where sign-preserving, class-preserving and degree-matched rewrites reach 7.5–20%, 10–15% and 12.5–15%.

The most interesting thing in that table is a dissociation. Delay makes encoding trivial for every graph — all of them hit 100% at immediate readout — while conferring none of the persistence. What biological wiring does is demonstrably not what delay does.

What the Memory Actually Is​

Knowing that the wiring remembers is not knowing how, and a property you can't explain isn't one you can reuse. So we asked the non-statistical question: what does the retained state physically consist of?

Four candidates, each with a distinct signature, fixed before looking: a self-sustaining attractor, a circulating loop, a wave still travelling outward, or a passive decaying echo.

The first answer was a limit cycle with a 160 ms period. The pattern-specific state appeared to recur at exact multiples of 160 ms — 160, 320, 480, 640 — at correlation r = 0.742, while intermediate lags sat near zero. The story was that the input selects which phase the network occupies, and an attracting cycle holds that phase, so the selection persists instead of fading.

That was a satisfying answer. It was also wrong, and the way it was wrong is the most useful thing in this post.

The retraction​

We only found it because we varied the measurement instead of the network. Every test up to that point changed the graph and left the analysis fixed at one configuration: 20 ms bins, an 800 ms window. The recurrence analysis reported its peak lag in bins, and 8 bins × 20 ms happens to be exactly 160 ms.

So we changed the bin width and nothing else — same graph, same operating point:

A real period is fixed in milliseconds no matter how you bin it. This one isn't. At 5 ms bins the correlation at 160 ms is +0.272; at 40 ms bins it is −0.039, which is nothing at all. The "comb at exactly 160/320/480/640 ms" was a comb of multiples of the peak lag in bins, which at 20 ms binning lands on multiples of 160 ms. The limit cycle was an artifact of the analysis grid.

The tell was there and we talked ourselves out of it twice. The "period" had already survived a 10× change in the membrane time constant, a 4× change in conduction delay, an 8× change in the refractory period, and an 8× change in network size — without moving. Nothing physical is invariant to four independent parameters. A null against a well-established law (delay sets oscillation frequency) is a reason to suspect your instrument, not the law.

What the correction leaves​

The contrast was never the artifact — only the period was. Measured at three bin widths with activity matched (biological 2.06%, block-301 2.70%, sign-401 2.44%):

Only the location of the peak follows the bin grid. The gap over the matched controls — 5× to 159× — holds at every bin width.

So the honest statement is weaker than a limit cycle and better supported: the biological graph's pattern-specific state stays correlated with itself over hundreds of milliseconds, and the randomisations decorrelate. That is persistence with structure, not a clock.

It is also a better explanation, for a specific reason. "The input selects which phase of a cycle" was a separate claim from the retention measurement, so it could be wrong on its own. "The state retains its pattern-specific structure" is the same fact the decoder is measuring directly. The mechanism no longer sits on top of the result — it is the result.

One more thing worth knowing: the code is distributed, not a labelled line. Five neurons decode at 52.5%, twenty-five at 57.5%, a hundred at 80%, and five hundred saturate at 82.5%. The neurons doing the work sit about one hop from the stimulus — the direct targets — rather than spread across the network.

Which half of that is the wiring, and which is the weights?​

"Maintains coherence" is a description, not yet an explanation. Two things could be responsible: the wiring (which neuron talks to which) or the weights (how many synapses on each connection). Every randomised control we'd built so far moved both together, so none of them could tell them apart.

So we built one that moves only the weights — permuting the synapse counts while leaving the wiring byte-for-byte identical — and, separately, rewired edges gradually in a ladder.

The result splits the mechanism cleanly in two.

The wiring is surprisingly hard to break. Rewiring 4% of edges costs essentially nothing. Rewiring 18% — moving 4.5 million connection endpoints — still costs essentially nothing: decoding 85%. Then between 18% and 63% it falls off a cliff to 25%. The substrate that carries the memory doesn't degrade gracefully, it fragments.

The weights decide how much information rides on it. With the wiring held byte-identical and 78% of the synapse counts reassigned, decoding halves, 87.5% down to 40%.

That is the part that corrected our own story: the same wiring, carrying less than half the information. Whatever the retained state is made of, its content depends on how the synapse counts are arranged, not only on which neurons are connected.

the wiring sets how much can be retained · the weights set how much actually is

Caveat, stated plainly: our original version of this section claimed the recurrence signature "survives completely intact, same 160 ms period" under the weight permutation. That half is void now — it was measured with the affected metric, at one bin width. The information half (87.5% → 40%) is a decoding measurement and stands. And the re-run is now done: at three bin widths, with both graphs at steady state, the recurrence signature survives a topology-identical weight permutation (0.412 / 0.836 / 0.917 against the biological 0.565 / 0.750 / 0.804) while retention falls from a mean of 78.3% to 44.2%. So the dissociation holds — as a statement about the recurrence signature, not about a cycle.

Did the result survive being tested properly?​

Everything above rested on one control graph per condition. That is not enough to claim anything — with n = 1 the minimum achievable p-value is 0.5, and the field's standard is around a thousand replicates. So we ran it properly.

146 independent randomisations, all at a matched operating point:

Three independent runs agreed to within a point — 12.9%, 13.5%, 13.7%. The effect is real, and it is large. Chance is 12.5%, so the randomisations are sitting at chance while the real wiring is six times above it.

Then we went looking for what does it structurally​

This is where the story turns. We had four candidates for a structural property that explains the difference, and all four died:

Four structural statistics, all present in the randomisations too. None of them could be what makes the difference — and that was starting to look like the real answer: maybe no structural statistic can see it.

Then one did. The spectral radius of the weighted connectivity separates the connectome from every single randomisation by 2.3 to 3.1×:

That is the first structural quantity in this whole project that detects the difference at all.

And its mechanism turned out to be measurable rather than mysterious: the connectome aims its heavy synapses at well-connected targets. The correlation between an edge's weight and its target's degree is +0.063 in the real brain, decays steadily as you rewire it, and is −0.0003 once you shuffle the weights. That single correlation tracks the spectral radius almost perfectly.

And then it failed — in the most useful possible way​

Two of those graphs have essentially the same spectral radius: 1934.2 and 1926.0 — a difference of 0.4%.

They retain at 44% and 12%.

A 3.7× difference in behaviour between two networks with the same structural number. So the spectral radius cannot be what determines the behaviour, no matter how well it correlates.

That failure is the finding:

The connectome has two independent structural properties. The spectral radius follows the weight arrangement. Retention follows the topology. They can be destroyed separately, and when you do, the two measurements move independently.

Rewiring breaks both — which is why the spectral radius looked like the answer for a while. Shuffling the weights with the wiring held byte-identical breaks only the first, and retargets the question: the weights control how much information rides on the structure, and the structure decides whether it survives.

One number that was wrong for a good reason​

We also wrote down the wrong mechanism first. Having measured the spectral radius, we explained it as a "degree-to-weight correlation" — which sounds right and is false.

Degree-preserving rewiring swaps whole edges, so every neuron keeps both its number of connections and its total synaptic weight. That correlation is exactly invariant by construction — we measured 0.7899 for seven different graphs, identical to four decimal places, while the spectral radius moved threefold. A quantity that never changes cannot explain one that does.

The correction took ten minutes to test and is now in the record. It is the third time in this project we caught ourselves stating a mechanism we had not actually measured, and we have written the fix into the process rather than promising to be more careful: a mechanism claim must name the quantity it asserts, and that quantity must appear in a table. The first version did not. The corrected one does.

What We Learned​

What worked: the method. Pre-registering the decision rule meant the answer was decided before the data arrived. The control ladder localised each effect to a specific kind of structure instead of producing one uninterpretable null. Every permutation null sat at chance (11–15% against a 12.5% floor) throughout, which is what makes the low control numbers trustworthy rather than suspect. And running each claim through a deliberate attempt to break it is what separated four retractions from a confident wrong post.

What didn't: most of our first explanations. Five claims had to be narrowed or withdrawn, and the surviving one is narrower than we originally stated: not "biology encodes and others don't", but "everyone encodes, only the biology retains" — and that retention is not a decaying trace but persistent, structured evolution of the pattern-specific state, which the controls show is the difference between remembering and not, since they are just as active. Our first explanation of that persistence — a 160 ms limit cycle — did not survive being measured two different ways.

The one thing we still can't explain: which structural feature of the wiring sustains the persistence. We now know more than we did — the spectral radius separates the connectome from every randomisation, its mechanism is weight-to-hub targeting, and it is demonstrably not the control parameter. But the quantity that actually carries the memory is still unidentified. Four aggregate statistics and one spectral quantity have been eliminated. That is the honest edge of the result.

The uncomfortable part: our errors were almost all in one direction. Better instrumentation never strengthened a claim; it only ever weakened one. That is a useful prior to carry: if a result about biological specialness feels satisfying, that is a reason to attack it harder, not to write it up.

What we would not claim: that this is a fly brain. It is an anatomical connectome plus an assumed dynamical model — no plasticity, no per-edge delays, no gap junctions, no neuromodulation. The retention result is a statement about what this model does with this wiring. Whether it survives morphology-derived delays is the open question, and uniform delays are not a substitute for them.

FlyOS is at github.com/brsbyrk/flyos — code, data, and the full experimental record, including every retraction in the commit that made it.

The Hypothesis​

One sentence, one metric, falsifiable:

At matched parameter count and matched compute, a sparse graph-routing model predicts byte streams at lower bits/byte than a dense model of the same budget.

The appeal of routing is that it should be cheap. A dense transformer spends a fixed amount of compute per byte forever. A router picks a path through a neighbourhood and does small local work, so most of the model is dormant most of the time.

The mechanism: 32 stations arranged in a graph, and a puck that visits them. Each station carries an anchor (where it sits), a force vector (which way it pushes), a 16×16 context matrix, and a fixed byte-level instruction. The puck has position and velocity, and momentum — v = 0.5·v + 0.5·F. At each step it scores its four neighbours by alignment times reachability:

G(m) = cos(F_m, v) · exp(−‖a_m − p‖² / 2σ²)

and takes the argmax. Hard selection, geometric scoring — deliberately not a softmax, because the point of the experiment was to test whether discrete routing pays for itself. A shared head then reads the puck's context state and emits the next-byte distribution. Arm A: 34,624 parameters. Arm B: a parameter-matched dense transformer, 33,184 parameters, 4% smaller.

It Lost​

Routing was beaten at every context length, and the margin grew with context:

Dense also won on equal wall-clock — and that is the harder test, because at 1024 context the transformer costs 8.4× the FLOPs per byte. It won anyway, while paying for its own compute.

Worth saying plainly: both arms are worse than gzip (2.846 bits/byte). At ~35,000 parameters these models are far too small to be competitive compressors. This was never a product; it was a question about representation.

We Removed the Special Parts​

The decisive evidence wasn't the loss — it was the ablations. If routing's distinctive machinery is load-bearing, removing it should hurt. We replaced each defining feature and measured at 10 MB, three seeds:

Nothing moved. Removing the momentum, the hard selection, and the per-node instructions changed nothing measurable. If you can unscrew the engine and the car drives identically, the engine wasn't connected. The kernel was decoration; a shared emission head was doing all the work.

Then We Found the Bug​

Chapter II's follow-up asked whether that result was really about per-node capacity rather than routing at all — by varying the one thing the first experiment never varied: how much per-node compute exists. That experiment added a manipulation check, which is a fancy name for a simple idea: before you interpret an ablation's null result, prove the ablated thing actually ran. A null from a mechanism that never fired means "it never ran", not "it didn't help".

The check failed immediately, for a reason nobody expected:

'the cat sat on the mat'   -> node path 0, 2, 10, 11, 19, 12, 10, 11, 19, ...
'ZZZZZZZZZZZZZZZZZZZZZZ'   -> IDENTICAL
22 random bytes            -> IDENTICAL

8 walkers reading 8 DIFFERENT byte streams -> distinct paths: 1

The routing rule never observed the data. Its score consumed only the graph, the anchors, the forces, the position and the velocity — none of which depend on the input. Every byte the model read went into the context state, which reached the emission head and nothing else. The puck's path through the graph was a pure function of its parameters: a fixed schedule, pre-computed, the same for a love letter and random noise.

That reframes the whole first experiment. Routing wasn't weak — it was a fixed wiring wearing routing's clothes. The momentum and hardness ablations were noise because there was nothing for them to contribute: the path carried zero information about the input. The verdict was right, but my explanation of it was wrong, and the correction is now in the repo.

It also explains the coverage numbers. A fixed schedule visits 12 of 32 stations, so only 10 of 16 instructions ever executed, and the counting instruction — the one the whole follow-up rested on — lived on two stations the walk never reached.

Making It Real​

The fix was small. Make the puck's heading depend on what it has read:

u = v + W·c          W is 16×16, +256 parameters

c is the context state — the only thing on the walk that carries history, and by the time routing happens it already contains the current byte. Momentum is kept, the geometric scoring is kept, the hard argmax is kept. Only the direction the alignment is measured against changes.

The result was unambiguous. Same model, same everything else:

Those are ranges across six seeds, not single readings: with a fixed schedule the walk deterministically reaches 12 stations, and once routing depends on the data it reaches 26–32 depending on how the route trains. The 15th and 16th instructions are no-ops, so "all real ones" is the honest bar — the no-op occasionally goes unreached, which cannot matter by definition.

For the first time, the channel was genuinely alive.

It Still Lost​

The follow-up was a 2×2 factorial: a rich 16-instruction set versus the same model with every instruction replaced by a no-op, crossed with two ways of reading the instructions' output. 1 MB of enwik8, three seeds. If instructions matter, the rich arm should beat the do-nothing arm.

+0.0014 bits/byte against a 0.10 threshold. Sixteen working instructions, including a counting table that demonstrably filled, are indistinguishable from instructions that do nothing. The pre-registered prediction — that richer instructions would only help with a wider read-out — failed too.

What We Learned​

What worked: the experimental method, eventually. Pre-registering the kill criterion meant the answer was decided before the data arrived. The manipulation check — added specifically because the first experiment's nulls were uninterpretable — found a structural defect on its first use, before 12 runs were spent on a dead channel. And each stage removed an alternative explanation: it wasn't the routing rule, wasn't the instruction set, wasn't the read-out width, and wasn't the context window.

What didn't: the idea. At ~35,000 parameters, the shared emission head reading a 16-dimensional recurrent state accounts for essentially all of the model's behaviour. The graph structure, the geometric routing, the per-node instructions — none of it contributes anything measurable on top.

The uncomfortable part: the sequence was null → find a defect → fix the defect → still null. That is a much stronger negative than the first one, because the mechanism was verified working when it produced the null. It's also a reminder that a negative result is only as good as your ability to show the mechanism ran.

The one lever never touched: scale. Everything here ran at tens of thousands of parameters on 1–10 MB. That constraint is capacity, not architecture, and it is the honest reason both arms sit above gzip. Testing it needs real compute and a different budget — so this is where Chapter II stops.

Chapter II is closed as a negative result, and the negative is documented to the same standard as a win would have been: the design, the defect, the fix, the numbers, and every ablation that failed to matter. Three experiments, two code revisions, all on CPU.

Carnot is open-source under MIT. github.com/brsbyrk/carnot

rANS: Replacing the Arithmetic Coder​

The original pipeline ended with an order-1 arithmetic coder from PAQ. It was accurate but slow — arithmetic coding maintains a high-precision interval and rescales it per symbol, an inherently sequential process. rANS (range Asymmetric Numeral Systems) replaces interval arithmetic with a single integer state update: divide by the symbol frequency, multiply by the total, add the cumulative offset. Division and multiplication instead of interval logic — streaming-friendly and far faster.

carnot-rans is a pure-Rust crate with order-0 and order-1 adaptive models. Frequencies update as symbols are encoded, so the model tightens over the stream. The key trick is 16× interleaving: 16 independent rANS states encode interleaved symbols, breaking the data dependency chain and enabling pipelined throughput. Each symbol lands on a different state in a stride of 16; decoding reads them back in the same order — no parallel hazard, just ILP-friendly sequential throughput. On enwik8, rANS encodes at 180-220 MB/s and decodes at 350-400 MB/s — 50-100× faster than the PAQ arithmetic coder, with identical compression ratios.

GPU rANS Kernel​

Can rANS run on a GPU? Encoding is symbol-parallel (nice for GPUs) but decoding is inherently sequential — you can't decode symbol N+1 until you know state N. We split the problem: histogram on GPU → table build on CPU → encode on GPU → decode on CPU. The kernel uses CubeCL 0.10, a Rust GPU compute framework targeting wgpu — one codebase, four backends (Metal, Vulkan, CUDA, DX12). Each workgroup builds a local histogram before atomically merging into global counts, and the encode pass scatters encoded symbols into a global output buffer. On a 1MB input, the GPU encode pass takes ~0.5ms; PCIe transfer is the real bottleneck.

GPU BWT via Prefix Doubling​

The June BWT scaffold was CUDA-only. To go cross-platform, we implemented prefix doubling + bitonic sort in CubeCL: assign each suffix an initial rank from its first character, repeatedly double the comparison window, and sort rank pairs with bitonic sort over log₂(n) iterations. The bitonic sort maps perfectly to GPU workgroups — it's a fixed comparison network with no branching, ideal for SIMD lanes.

The approach works correctly. The crossover point tells the whole story:

In release mode, CPU SA-IS is so fast that GPU doesn't win until ~1MB. For Carnot's 64MB blocks, CPU is faster. The kernel is a working reference, not a practical accelerator — we're archiving it as such.

Multi-threaded BWT​

While the GPU path matured, we parallelized the CPU path. carnot-bwt uses libsais-rs with rayon for block-level parallelism, controlled by CARNOT_BWT_THREADS. Each 64MB block gets its own SA-IS invocation on a dedicated thread, with multi-block pipelines overlapping BWT, MTF, and rANS stages. On a 4-core machine: 2.5× speedup over single-threaded, no ratio penalty.

VMBC Extensions​

The bytecode VM gained two new pattern detectors. The arithmetic progression detector recognizes sequences like 0, 5, 10, 15, ... and encodes them as a single (start, step, count) program — a 1MB arithmetic sequence compresses to ~30 bytes, roughly 9× smaller than zstd on this class of data. The counting pattern detector extends this to nested arithmetic progressions, where each "digit" follows its own step within a wrapping modulus (think base-N counters with varying stride per position). These are niche wins — dramatic on synthetic structured data, zero contribution to natural text. That's the VM's trade: high-ceiling, narrow-domain compression.

Final Benchmarks​

Carnot is now a multi-block, multi-threaded, rANS-backed BWT compressor. We dropped RePair from the default pipeline — MTF + zero-run captures enough digram redundancy on the rANS backend that RePair's overhead stopped paying for itself.

At 1MB, Carnot beats gzip by 14%. At 100MB, it edges past bzip2 and lands within 3% of LZMA2 — but 4× faster than the June PAQ-based pipeline. The GPU pipeline achieves identical ratios to CPU when MTF is active (0.40 vs 0.40 on enwik8 1MB). The transforms dominate; the entropy coder faithfully captures what they produce.

What We Learned​

What worked: rANS is the right entropy coder — fast, precise, and interleaving eliminates the sequential bottleneck for all practical purposes. CubeCL delivered cross-platform GPU compute with one codebase and no CUDA dependency. Multi-threaded BWT scales cleanly with libsais + rayon. The VM pattern detectors are real wins for structured data, even if they don't move the needle on text benchmarks.

What didn't: GPU BWT via prefix doubling can't beat a tuned SA-IS on CPU for realistic block sizes — the GPU's advantage is throughput on huge inputs, not latency on 64MB chunks. GPU rANS decode remains CPU-bound; the sequential dependency means the kernel only helps on the encode side, and PCIe transfer eats most of that gain on small-to-medium inputs.

Why we're archiving: Carnot set out to explore how close we could get to the true compression limit by trading time for ratio, with GPUs as a speculative accelerator. We found a solid answer: BWT + MTF + rANS gets within 3% of xz with a clean, maintainable Rust codebase. The GPU work proved the approach viable but not yet practical at these block sizes. The remaining ~38% gap to PAQ/CMIX requires long-range context modeling — a different problem, for a different project.

The repo stays open. The dead ends are documented. The code compiles, benchmarks run, and anyone curious about compression, GPU compute, or rANS can pick up where we left off.

Carnot is open-source under MIT. github.com/brsbyrk/carnot

Carnot is an open research project named after Sadi Carnot — who defined the theoretical maximum efficiency of heat engines. The project asks the same question in a different domain: how close can we get to the true compression limit if we stop caring about throughput?

It has grown into three tracks across a Rust workspace: pattern-scale decomposition compression, GPU-accelerated transforms, and a cryptographic accumulator on BLS12-381. It's a lab notebook, not a product — 21 experiments so far, with failures documented as carefully as successes.

Pattern-Scale Decomposition​

The key insight, formalized in docs/cf-unification.md: every structured byte sequence decomposes into two orthogonal components — the pattern (generating rule: constant, alternating, counting, repeating) and the scale (length, repeats, frequency). Standard compressors conflate both channels; Carnot separates them.

The math uses continued fraction rational approximations of byte sequences. CF convergents find the infinite-limit structure in small terms, but encode length implicitly through a huge final term costing O(n). The bytecode VM is the practical fix — same pattern detection, but with an explicit O(log n) scale channel.

The Bytecode VM​

A 17-opcode stack VM with loop stack encodes structural patterns as tiny programs approaching the Kolmogorov complexity floor — the shortest possible description of the data:

Programs are discovered via beam search (width 50, sampling 4KB windows) rather than exhaustive enumeration. The adaptive pipeline selector auto-routes between 7 strategies: random pass-through, delta encoding for sorted integers, BWT+MTF+RLE for text, sparse encoding for low-entropy data, dedup for repeated blocks, RePair grammar compression, and the VM bytecode path for periodic/structured input. Candidate pruning gives a 1070× speedup on uniform data.

The BWT Pipeline​

The best general-purpose result comes from a five-stage Burrows-Wheeler pipeline:

Raw data → BWT (SA-IS) → Move-to-Front → Zero-run → RePair → Order-1 Arithmetic Coding

Each stage targets a different redundancy type:

The evolution is visible in the experiment log — five attempts to get from a Python prototype at 0.2971 ratio to the Rust champion. The breakthrough was sentinel-based SA-IS BWT + incremental RePair, refined from 0.2512 ratio and 29 minutes down to 0.2352 at 2 minutes via flat-array order-1 modeling.

On enwik8 100MB:

At 1GB (10× concatenated enwik8), xz pulls ahead — 0.1737 vs Carnot's 0.2403 — because LZMA2's 64MB sliding window exploits cross-copy repetition that BWT, confined to each 64MB block, misses. Carnot wins on single files by 5.3%; xz wins on concatenated copies. Both approaches have their regime.

Multi-threading and Speed Improvements​

Multi-threaded BWT gives 2.6× speedup at a 3.8% ratio cost on 100MB enwik8. Multi-block parallel compression scales near-linearly at 4×. The whole pipeline processes 64MB blocks sequentially to stay within RAM limits — SA-IS suffix array on 1GB needs ~5GB for the SA alone. A production run would need either 32GB+ RAM, out-of-core BWT, or pre-deduplication.

GPU BWT Acceleration​

BWT suffix array construction is the pipeline's bottleneck. carnot-gpu-core offloads it to the GPU via libcubwt, targeting a GTX 1060 6GB (Pascal, CC 6.1, CUDA 12.2). The FFI is scaffolded with cudarc device management and VRAM budgeting — libcubwt needs 20.5n bytes for n-byte input, capping at ~300MB on the 6GB card. A CPU fallback runs suffix sorting directly for testing. The carnot-gpu-cli provides info and bench-bwt subcommands.

Bilinear Accumulator on BLS12-381​

The carnot-accumulator crate implements a bilinear accumulator for constant-size set membership proofs — 48 bytes for the accumulator and each witness, regardless of how many elements are in the set:

Setup:  s = random secret
        pk = g₂^s

Accumulate:  acc = g₁^{∏(s + hash(x_i))}
Witness:     w   = g₁^{∏_{i≠j}(s + hash(x_j))}
Verify:      e(w, g₂^{hash(x_j)} · pk) == e(acc, g₂)

Key implementation details:

• Pairings via blst on the BLS12-381 curve
• NTT-based polynomial multiplication — tree-based product expansion in O(n log n) over a field with 2^32 roots of unity
• Batch witness generation via prefix-suffix products in O(N)
• Merkle tree integration for byte-level membership — each leaf is SHA-256(index || byte), so a verifier can brute-force all 256 byte values against a single Merkle proof
• Powers-of-tau trusted setup with chunked verification and parallel brute-force checking

This isn't directly about compression — it's infrastructure for verifiable computation. The polynomial math (NTT, pairings, finite fields) connects both domains.

What Didn't Work​

Failures are preserved as terrain maps for future explorers:

The Structural Gap​

The project's central metric:

H₀ - compression_ceiling = structural_gap

H₀ is byte-frequency entropy (Shannon). The compression ceiling is the best ratio standard tools achieve on a given file. The gap measures latent structure that current compressors miss — the hunting ground:

Periodic signals and repeated blocks have H₀ ≈ 8.0 — byte-frequency counters see noise. But standard compressors exploit their structure easily. The gap quantifies what's left on the table.

The bit-plane experiment reached 99.7% of its theoretical bound, confirming intra-byte structure is well-understood. The remaining ~38% gap to PAQ/CMIX on enwik8 requires inter-byte context — longer-range models, match models, word-level prediction. Not one missing trick, but an architecture difference.

Philosophy​

Three principles drive the project:

1. Measure first. Every claim backed by the standard test corpus (10 synthetic types × 4 sizes).
2. Preserve the journey. Failed approaches stay in the repo — they're the map of what's been tried.
3. Trade time for compression. CPU-hours are abundant; novel algorithms aren't.

The project achieved its goal: designing and benchmarking a novel compression pipeline that beats standard tools on structured data. The GPU and accumulator work extend the same mathematical foundation into new domains. Carnot remains open for exploration — the remaining gaps are documented, the architecture is modular, and the dead ends are mapped.

Carnot is open-source under MIT. github.com/brsbyrk/carnot