A recurrent model,
folded flat onto
a dataflow NPU.

The name and the mark

The logo is a valid schematic.

The constraint

What makes it hard.

A GPU runs whatever kernel you hand it. A dataflow accelerator runs the graph it was given, in the shape it was given, or it hands the work back to the CPU, and nothing tells you that happened except the power draw.

Most of the difficulty follows from four facts. Shapes are static and baked at compile time. Arithmetic is fp16 or narrower. There is no control flow inside the graph, so a loop must be unrolled or moved to the host. And placement is silent: a graph that cannot be lowered still runs, just slowly, on the wrong processor, without the energy advantage that was the point.

That last fact is the one that bites. Getting the model to produce correct tokens on the Neural Engine is easy. Getting every op to actually execute there is the work.

The shape of the model

Three layers that forget, one that remembers.

A Gated DeltaNet layer keeps a fixed-size state and updates it with a gated delta rule -- a bounded amount of memory, written in place, no matter how long the conversation gets. Full attention keeps every key and value it has ever seen. Qwen3.5 alternates them three to one.

That ratio is a gift to a dataflow engine, and it took a while to see why. Three quarters of the trunk has no growing tensor at all -- the state is the same size at token 1 and token 200,000, so its shapes can be baked at compile time and it can live permanently on the accelerator. Only the remaining quarter has the unbounded key-value cache that a static graph cannot express.

So the split is not a compromise. It falls out of the architecture: the 18 recurrent layers go on the Neural Engine and stay there; the 6 attention layers hand their keys and values to the host, which pages them and does the softmax merge in fp32. The accelerator never sees a tensor whose size it could not predict.

The conversation is a place, not a transcript

It never re-reads the chat from the top.

Most chat systems are stateless behind the scenes: every time you send a message, the whole conversation is fed back through the model from the beginning, like someone re-reading an entire email thread before typing each reply. Gadeon does not work that way -- it cannot, and that turns out to be a feature.

Those recurrent layers carry their memory as a small fixed-size state that was shaped by everything it has read, the way you remember a book without storing its pages. There is no way to rebuild that state except by reading everything again, so Gadeon simply never throws it away. Each turn, only the new words are fed in -- an append-only conversation. A ten-minute chat costs the same per reply as a two-hour one.

And when something should be un-said, it is rolled back rather than re-read: before each turn the engine drops a bookmark, and its private scratch -- the step-by-step reasoning, the raw mechanics of a web lookup -- is rewound to that mark once the answer is out, so it never piles up in memory. Press Stop mid-thought and the same bookmark restores the conversation as if the turn had never started.

The lowering · in order

Five moves that made it place.

Each of these was forced by the hardware rather than chosen. They are numbered because the order mattered -- none of the later ones was reachable before the earlier ones landed.

Weights are palettized to 6 bits with k-means, grouped per channel in groups of 16. The quantizer is not the interesting part of this project -- it is table stakes for fitting in memory. The placement is the interesting part.

Why it costs less

The joules are in the movement.

A matrix multiply costs about the same arithmetic wherever it runs. What differs is how far the operands travel to reach it. A GPU decode step pulls weights across the memory bus, computes, writes back, and synchronises, and at batch size one the arithmetic is small next to the traffic.

A dataflow engine is built the other way round: the compiled program keeps its weights near the compute units and streams activations through a fixed pipeline. There is no kernel launch, no scheduler round trip, and no per-step re-fetch of a weight that has not changed since the last token. The cost of that design is the constraints above (static shapes, no control flow, a strict placement rule); the payoff is that the expensive movement mostly does not happen.

The second-order effect matters just as much on a laptop or a phone: the GPU stays idle and cool. Inference stops competing with the compositor, the browser, and whatever else is drawing to the screen.

One honest caveat. That energy figure is a single measurement -- prefill, 0.8B, one machine. Per-model and decode-phase energy have not been benchmarked, and the throughput table below is throughput only. Treat 4.4× as evidence that the approach is sound, not as a spec sheet.

Measured

Throughput.

Two machines, release builds: a MacBook Air (M3, 24 GB) and an M4 Pro (48 GB). A 512-token prefill followed by 128 greedy decodes, one warmup pass -- the same protocol as llama-bench -p 512 -n 128.

The last two rows run on a different backend. The ternary Bonsai models (1.7B is a dense Qwen3; 27B is built on Qwen3.6 27B) run on the GPU, and the reason is measured rather than assumed -- it is the open question below. They are kept as a comparison point, driven by the same harness as the Neural Engine models, so their quality and speed can be read side by side. The 1.7B row prefills 508 tokens rather than 512: its vocabulary segments the same prompt differently. The two ternary rows were not re-run on the M4 Pro, so those cells are blank rather than estimated.

Speculative decode, and a second stack.

The M4 Pro run also measured two things the table above cannot show: what self-speculative decode does to the decode column, and how the whole thing compares against llama.cpp on the same machine -- its Metal GPU backend, 4-bit weights, same prompt, back to back.

Two readings. On prefill the Neural Engine beats the GPU at 2B, 4B and 9B, and the lead widens with model size -- the GPU pays a fixed launch cost per kernel that the batched, fully-placed prefill trunk amortizes away. Only the smallest model loses. On decode the GPU wins by roughly two times across the board. Speculative decode is the lever that narrows it, and it pays only where decode is already slow enough to amortize a verify pass: it costs the 0.8B and the 2B throughput, turns positive on the 4B at +16%, and is worth +49% on the 9B. The crossover sits between the 2B and the 4B, and the slower the model, the more it earns.

That comparison is not apples to apples, in two ways that both matter. Our weights are 6-bit and llama.cpp's are about 4.5-bit, so the faster decode is also reading fewer bytes per token. And the accelerators differ in kind: the Neural Engine is the low-power path and leaves the GPU and CPU free, while Metal is peak throughput at high power. "The GPU decodes twice as fast" is a throughput-at-full-power claim, not an efficiency one.

Hardware: 16-core Neural Engine rated 18 TOPS, 10-core GPU, 8-core CPU, 100 GB/s unified memory. Debug builds are roughly 100× slower; every number here is a release build.

Still open

Ternary weights and the Neural Engine.

Two of the models above run on the GPU, and we would rather they did not. Getting the ternary lineage onto the Neural Engine is the question this project keeps returning to. So far the hardware has answered no, and the shape of that no is worth writing down.

A ternary weight stores three values in a little over two bits, which should make a small model very cheap to move. Decode is bound by movement, not arithmetic: one token at a time, the accelerator reads the entire weight set and does almost nothing with each byte, so bytes per token is the whole cost model. On paper a 2-bit 1.7B is the ideal Neural Engine model, and a 7.5× smaller weight set should read 7.5× faster.

It does not, and the reason is where the expansion happens. We stacked real ternary layer pairs out of the 1.7B and measured them three ways on an M3:

A 2-bit weight handed straight to a matmul does not place on the accelerator at all, in either operand layout. Split into codes and scales it places whole, and the compiled artifact really does hold 2.13 bits per parameter on disk -- and it still runs a third slower than the dense fp16 version it is supposed to beat. The arithmetic settles the reading: the time is what moving the fp16-expanded weights would cost, not what moving the stored bytes would. The engine expands to fp16 before the DMA, so the compression buys disk and nothing else.

Per token, the same 1.7B moves 457 MB as ternary on the GPU, 1.29 GB palettized to 6 bits on the Neural Engine, and 3.44 GB as 2-bit on the Neural Engine. That is a property of the weight format rather than of the model size, so it holds all the way down the lineage: there is no small ternary model that wins here, and the 1.7B result is not the 27B's residency problem reappearing at a smaller scale.

Three things we are still turning over. Whether a later generation streams sub-4-bit natively, in which case the decomposition that already places is exactly the shape that would benefit, and that work is banked rather than wasted. Whether palettizing to the 6-bit format the engine does stream is worth the quality it costs -- 1.29 GB a token is still nearly three times the GPU's traffic, so the honest guess is no, but joules per byte are not equal on the two units and we have not measured that end. And whether one model on one unit is the wrong frame altogether: prefill is compute-bound and places beautifully, decode is movement-bound and does not, so a ternary model that reads on the Neural Engine and talks on the GPU is a split we have not built yet.

Meanwhile the GPU path earns its place for a reason that has nothing to do with speed. The 1.7B needs no Neural Engine at all, which makes it the one model that runs on the small and older iPhones where the Core ML decode graph cannot be lowered -- the devices the rest of this page leaves out.

Why the prefill column is the one that matters

An assistant that uses the web mostly reads.

Decode speed is how fast the model talks. Prefill speed is how fast it reads, and once a model starts using tools, reading is most of the job.

Ask Gadeon to research something and it searches, then pulls in what it finds: a Wikipedia article, a news story, a page of search results, each of them thousands of words the model must ingest before it can say anything useful. Every one of those words goes through prefill, not decode. The answer you finally read might be two hundred tokens; the material behind it was ten thousand.

That is what a few thousand prefill tokens a second buys in practice: a fetched article lands in about a second, and a whole research detour (a search, a few articles, the headlines) in a handful. At slower prefill rates the same detour takes long enough that looking things up stops being worth it. Fast reading, more than fast talking, is what makes tool-using chat usable on a laptop.

Standing on

Attribution.

GaDeoN is a lowering, not a model. Everything it runs was trained, designed, or compressed by someone else, and the interesting parts of this page are only possible because that work was published openly.