Field notes
Six weeks of running a frontier-scale MoE on a single 16GB GPU - and the two times I had to retract in public.
Originally published on LinkedIn — read the original.
It started as a stubborn question: could I run a 744B-parameter mixture-of-experts model on one consumer graphics card?
The card is an RTX 5080 with 16GB of VRAM. The model, GLM-5.2, has 744 billion parameters spread across 78 layers, 75 of which route each token through 8 of 256 experts. Quantised hard, the weights still need somewhere north of 100GB. The arithmetic does not work. It works anyway, because of a streaming engine called colibri that keeps the hot experts resident in system RAM and VRAM and pulls the rest off NVMe as the router asks for them. Whether it feels usable comes down to one number: how often the expert the router wants is already in memory.
That number is where the story begins.
After a release upgrade, my expert hit rate fell off a cliff. On the E8 quantisation it went from 97.4% to 84.2%; resident set collapsed from 106GB to 50GB. The engine was pinning 1,352 experts where it used to pin 3,588. Decode throughput dropped about 19%.
Reading the pinning code, the shape of the bug suggested the budget was being divided once globally instead of per container. If that was right, a fix should restore roughly 2,400 pinned experts on one quantisation and roughly 3,500 on the other. I posted those two numbers on the issue thread before any patch existed, precisely so I could not move the goalposts afterwards.
When the fix landed, it measured 2,406 and 3,536.
That should have been the happy ending. It was not, because throughput did not follow. Residency was restored and the model was still slow. So I built the pull request's own merge base, ran the same sweep, and found the gap sitting almost entirely on one profiler line: expert matmul, 20 seconds in the old build against 29 in the new one. I wrote it up as a second, separate regression that had arrived in the fifty-odd commits between the two points, and published it.
It was not real.
I had compared builds I compiled that morning against numbers I had recorded a few days earlier. Same commit, same script, same toolchain - but not the same machine state. I rebuilt the anchor commit fresh and it measured 29.29 seconds where it had previously measured 23.64. The commit had not changed. The box had.
I rebooted. The same commit, same binary recipe, came back at 24.21.
So I went back to the thread and retracted it. Then I found a second error in what I had published: I had claimed the runs invoked the engine binary directly and never the launcher, which was the basis for eliminating one of the candidate commits. My harness does go through the launcher. I ran an A/B to show the launcher made no difference to the result - the conclusion survived - and corrected the claim anyway, because a conclusion that happens to be right for a stated reason that is false is not evidence, it is luck.
I still cannot fully explain the drift. It was not a background process pinning a core - the same process sat at 100% of one core out of 24 during the good runs too. It was not build flags; the compiler line was byte-identical across every build. What I can say is that this machine accumulated roughly a 20% CPU penalty over a long uptime and shed it on restart, with resident set and hit rate completely unchanged throughout - the signature of pure compute drift rather than anything about memory or I/O.
Then I measured how fast it accumulates. I ran five full sweeps back to back in one session, with the same anchor commit running first and last as a bracket. The two anchors disagreed by about 5% across 105 minutes, with resident set and hit rate identical at both ends. This box degrades measurably inside a single session under sustained load.
Two consequences followed, and I care more about the second one.
The first is procedural: benchmark numbers from this machine taken more than a couple of hours apart should not be compared, and across days they should not be compared at all without rebuilding an anchor commit first. That applies retroactively to some of my own earlier published numbers, which I said so on the thread.
The second is that the drift ran in my favour and I refused to use it. The four builds ran in an order that put the winning build last, under the most degraded conditions of the session - so every gap I reported was a floor, not an estimate. I could have divided the drift out. The arithmetic is trivial and it widens the result in the direction I wanted. I did not, because two anchor points bound the magnitude of drift, they do not establish its shape, and a modelled correction dressed up as a measurement is exactly the mistake I had made twelve hours earlier.
There was one more correction to make, and this is the one worth reading. On the degraded machine, I had credited the contributor's fix with a ~4% throughput gain. Measured clean, in the same session, thirteen minutes apart, it was +10.0% on one quantisation and +9.5% on the other. My sloppiness had understated someone else's work by more than half. Being wrong in public is uncomfortable when it costs you credit. It is worse when it costs someone else theirs.
The regression is closed. Current development builds are at parity or slightly ahead of the last good release on every arm.
With the engine healthy, a different question got interesting. A mixture-of-experts model does not use all of itself for any given token. If I could see which experts a prompt actually wakes up, could I map what lives where?
The model has 19,456 routed experts across its MoE layers. I built a probe set of sixteen topic categories - GPU architecture, history, code, mathematics, biology, economics, literature, cooking, law, music, geography, science, and four in Arabic - four prompts each, greedy decoding, and captured the router's per-token selections through a trace hook. One critical setting: the engine has a cache-aware routing mode that biases expert selection toward whatever is already resident. That had to be off. Otherwise I would be measuring my own cache, not the model.
The run bit me twice, which is by now the pattern of this project.
First, the engine has an environment variable that is documented as disabling the write-back of usage history. I set it. My usage profile - the ranked history the engine uses to decide what to pin - got overwritten anyway. Reading the source, that variable is implemented in two of the engine's model backends and not in the one I was using, which calls the save function unconditionally at five separate points. The documentation describes it as shared. So the probe prompts were quietly contaminating the profile the whole time. I restored from a master copy, wrapped every run in a snapshot-and-restore, and verified the file's checksum before and after every sweep from then on.
Second, just this week, a launcher timeout made me think a run had died when it had not. I started a second one. Two 744B engines streaming concurrently on one machine, both writing to the same trace file. I killed both, restored the profile from the verified master, deleted the contaminated traces, and re-ran clean. That batch cost twelve minutes instead of six. The alternative was publishing a number produced under conditions I had not accounted for, which is the thing this whole project has been teaching me not to do.
The atlas covers 19,168 of 19,200 traceable experts across 3.48 million routed selections. Two findings stand out.
Specialisation is real but modest, and it deepens with depth. Measuring how concentrated each expert's usage is across topics, the early layers score 0.134, the middle layers 0.230, the late layers 0.204. Early layers are doing something closer to general token processing; specialisation emerges in the middle stack.
And topics separate more cleanly than languages do. The overlap between the hottest 500 experts of any two English topics averages 0.197; between two Arabic topics, 0.311; across the language boundary, 0.144.
That last number is what made the final experiment worth running. I took four cybersecurity questions in Arabic and translated them word for word into English, so language was the only variable. If the model routes primarily by language, the two should look like any other cross-language pair - around 0.14.
They overlapped at 0.410. Twice the within-English baseline. Taking the 300 most security-enriched experts in each language, 183 are the same experts - 61% of each core, concentrated in the middle and late layers. The strongest ones are near-symmetric: one expert in layer 54 shows a 162x enrichment over the control topics in English and 162x in Arabic.
The model, as far as I can measure it, has a cybersecurity region that is substantially language-independent. Which means a domain profile built from English probes should transfer to Arabic - and that the entire union of both cores is 417 experts, about 5.9GB. On a 16GB card, that is a pinnable working set.
The technical result I am proudest of is not the atlas. It is that I published a wrong number, found it myself, retracted it in the same thread, and then found a second error inside my own retraction and corrected that too.
Three things I would keep:
All of this is on public issue threads, raw logs included, mistakes and all. That is the part I would not change.