Z.ai, a Chinese AI lab, released GLM 5.3 Flash on 25 August 2026. It's close to frontier level, its weights are open and free to download under the MIT licence, and it runs on hardware you can buy in a shop. It takes some effort, though.
Flash is the smaller sibling of GLM 5.3, released the same day. The big one has about 750 billion parameters and reads only text. Flash has 320 billion and reads images too, the first model in the GLM-5 series that does. On the six benchmarks Z.ai reports for both, Flash lands close to its big brother, and on two it comes out ahead:
| Benchmark | GLM 5.3 Flash | GLM 5.3 |
|---|---|---|
| Terminal Bench 2.1 | 84.3 | 88.2 |
| DeepSWE v1.1 | 63.4 | 66.9 |
| Agents' Last Exam | 26.3 | 28.5 |
| HLE (Humanity's Last Exam) with tools | 55.3 | 62.5 |
| AutomationBench v1.0.6 | 48.8 | 48.2 |
| GDPval-AA v2 | 1773 | 1769 |
Z.ai's own figures, from the Flash and GLM 5.3 model cards.
My machine is a Mac Studio with an M3 Ultra and 256 GB of unified memory. The top model has 512 GB, but I appreciate 256 isn't in every home either. This is how I got a frontier-ish model running on it locally, fast enough that I don't have to go to sleep between responses.
What it takes to run the full thing
A model is a very long list of numbers, its parameters. Flash has 321 billion of them (Z.ai rounds it to 320). At full precision (bf16), each one takes 2 bytes, so the weights alone come to 643 GB, before you've typed a word. Only 18 billion of them do any work on a given word, because Flash is a mixture of experts: each word goes through 8 of its 288 small expert networks, plus one that always runs. But all of them have to sit in memory, ready. The biggest Mac Studio has 512 GB. In a data centre, 643 GB is more than eight of the 80 GB GPUs they use, put together.
Z.ai's main release already squeezes each number into 1 byte (FP8), which makes it 328 GB. That's still too big for 256 GB, and it's built for those data-centre GPUs rather than a Mac.
So people in the community quantise it. It's a way to fit the weights into a smaller space: each number is stored in fewer bits, usually 4 instead of 16, and each small group of numbers keeps a scale that turns them back into close-enough full values when the model runs. The most sensitive parts are often left at full size. You lose a little precision and save most of the space.
Start with the 4-bit version
The Mac versions come from other people. The day after release, OrcaRouter quantised it for Apple's MLX in 2-, 3-, 4- and 6-bit builds. A month later mlx-community republished the 4-bit one with a small config fix, and that's the one I used, run with mlx-vlm, the MLX library for models that read images. It's 204 GB, which leaves about 50 GB for the conversation and everything else on the Mac. That's the best starting position: the 6-bit build is 296 GB and too big, and the 3-bit one only saves another 20 GB for less precision.
Once I had it running, it thought at about two tokens a second. That's slower than you read, from a model that likes to think for thousands of tokens before it answers. The digital equivalent of watching paint dry. My first test question ran for an hour and twenty minutes and still hadn't finished.
So I decided to figure out whether that was the ceiling or whether it could go faster. It turned out there were several bugs: four in all, counting the two that had to be fixed before it would run properly at all. With all four fixed it runs at 20.5 tokens a second, about ten times faster, and gives the same right answers. Since then, a dust-out, two settings and the model's own draft head have taken it to about 26.
Here's each bug, how to fix it, and how to check whether your own models have the same problem. At the end there's a benchmark: the same 200 questions on my copy and on Z.ai's own Flash in the cloud, to see whether the 4-bit squeeze and all the speeding up cost it anything.
If you just want the fixes, they're all in one file further down.
Bug 1: it won't load
The first attempt fell over before it wrote a word:
Expected shape (4096, 512) but received (4096, 768)
This is the only bug that announces itself. OrcaRouter kept a few parts at 6 bits instead of 4, and wrote that down in config.json under those parts' names. mlx-vlm glues two of those parts together as it loads and gives the result a new name. It then looks the new name up, doesn't find it, and assumes 4 bits.
Why does the wrong bit count break the shape? Because small numbers are packed into 32-bit boxes. A row of 4,096 numbers fills 512 boxes at 4 bits and 768 at 6. Hence: expected 512, got 768.
Fix: tell mlx-vlm the glued part uses the same setting as the parts it was made from. It's the first block in the patch file below.
Bug 2: it nearly took the Mac down
The model loaded, sat at 186 GiB, and started on the first question. Then free memory dropped from 35% to under 5% in five seconds.
I'd put a watchdog in the launch script: a few lines that check free memory and kill the job before the Mac runs out. It stopped the job at 1% free. On the retry I made it check every second and log as it went, and the log showed MLX jumping from 186 to 226 GiB in two seconds, while it read a 300-token prompt.
What was happening: some of this model's attention layers (the part that looks back over the conversation) have a fast, purpose-built GPU program, a kernel, that only accepts 16-bit numbers. The numbers were arriving as 32-bit, so the kernel said no, and the code fell back to a slow route that makes a separate copy of about 2,000 earlier positions for every token of the prompt. That's 40 GiB or more, on a machine with about 50 to spare.
Fix: convert the numbers to 16-bit on the way into the kernel. The extra memory went from 23.6 GiB to 0.12 on a test prompt.
Two things worth knowing from this one:
- MLX's memory limit is a guideline. Its own documentation says so, and it went straight past the one I'd set. The watchdog is what saved the Mac.
- Eight layers are enough to find a bug. Load the model lazily, keep the first 8 layers, and test those. That needs about 16 GiB instead of 190, and the Mac stays usable while you work.
And that "32-bit arriving where 16-bit was expected" turned out to be the story of the whole weekend.
The float32 tax
Computers store numbers with a decimal point in a few standard formats, called dtypes. Three matter here:
| Format | Size | Good at |
|---|---|---|
| float32 | 4 bytes | Range and precision. The safe default. |
| bf16 | 2 bytes | Huge range, less precision. What this model runs in. |
| fp16 | 2 bytes | More precision, smaller range. |
bf16 and fp16 are the same size but split their bits differently, so neither can hold every value the other can. When MLX is asked to multiply one by the other, it does the careful thing: it upgrades both to float32, which can hold either exactly.
Careful, and four times the work. The exo team, who make software for running models across several Macs, hit this on the same model and put it plainly: MLX "runs the quantized matmul in float32 at about 4x the cost." (A matmul is a matrix multiply, which is most of what a model does.)
Everything runs. The answers stay right. You just pay four times over for every calculation, silently. That's the float32 tax, and this model was paying it twice.
Bug 3: the tax, the first time
Friday night, with bugs 1 and 2 fixed, I sent one test question and watched what the model was doing, to see where the time was going. Memory steady, disk quiet, nothing swapping. And 1.9 tokens a second. At that speed you'd ask a question and come back after dinner. After an hour and twenty minutes it was still thinking, and the script only saved the reply once it finished, so there was nothing to keep. I abandoned it at 22:10. There had to be a better way than waiting. (Every run after this saved its reply every 512 tokens. Make long runs save as they go.)
The better way was to time a few layers on their own, the 8-layer trick from bug 2, which takes seconds instead of hours.
The cause: most models pass one running total from layer to layer, the residual stream. This one passes four, which Z.ai calls hyper-connections, and mlx-vlm keeps them in float32 so the mixing between them stays stable. Fair enough. But it then handed those float32 numbers straight into every layer, and every bf16 weight they touched got upgraded to float32 with them.
Fix: convert to bf16 on the way into each layer, and keep only the mixing in float32. On the 8-layer test, each layer got 7 to 10 times faster.
How do you know a speed fix hasn't quietly broken the model? You compare what comes out before and after. We used cosine similarity, a score for how closely two long lists of numbers match, where 1.0 means identical. This fix scored 0.999993.
On the whole model, speed went from 1.9 to... 2.7.
Something else was still eating the time. At 22:25 I parked it and let a short test run at that crawl instead: two questions, each asked two ways. Before you make something fast, you want proof it works, start to finish, at the real settings, with every answer read. If Flash had been getting answers wrong or thinking in circles, speed wouldn't matter. Those answers would also be the yardstick for every fix after: same questions, same answers. And the model and the speed tests couldn't share the Mac anyway; the memory only fits one at a time.
It finished at 23:26. Four answers, four correct, no loops. Seventy-three minutes.
Bug 4: the tax, the second time
To shrink a model to 4 bits, the converter splits each block of weights into groups of 64 numbers. Each group is stored as small whole numbers plus a scale that turns them back into real values. Every 4-bit multiply reads those scales.
On Saturday morning we wrote a script that records the number format going into and out of every part of the model for one token. Float32 was still leaking. The file headers showed why: all 37,338 sets of scales are stored as fp16. The model runs in bf16. So every single 4-bit multiply, in every expert, was a bf16 × fp16 pair, and every one was upgraded to float32.
This one we should have seen sooner. Exo had written it down on 1 November, and we'd read their notes on Friday night for bug 3. The bit about the scales was in the same paragraph. (Exo's version never had bug 3, which is why their "before" was 6.6 tokens a second and ours was 1.9.)
The same audit found one more leak at the very end: the final layer, which scores all 154,880 possible next tokens, was getting float32 too. It took 117 milliseconds per token. Fed bf16, it takes 2.
Fix: run each 4-bit multiply in fp16, the scales' own format, so the scales stay exact, and hand the result back as bf16. Feed the final layer bf16. On the 8-layer test, one token went from 34.7 ms to 9.0.
Why ten times faster overall, if the tax is four? Four is the rate per multiply. On top of that, float32 numbers are twice the size to move around, the final layer was hit far harder than four times, and the two leaks overlapped. That's also why fixing bug 3 alone barely helped: bug 4 was still holding the same multiplies in float32.
The result
At 06:22 on Saturday, with every fix in, I loaded the full model and ran the same test as Friday night. The questions are from MMLU-Pro, a well-known multiple-choice benchmark; each was asked plain and with a long system prompt from my own experiments.
| Question | Friday (bugs 1–3 fixed) | Saturday (all four) |
|---|---|---|
| Health, plain | 45.5 min · 2.6 tok/s | 5.0 min · 20.5 tok/s |
| Health, with system prompt | 18.7 min · 2.7 tok/s | 6.9 min · 6.6 tok/s |
| Maths, plain | 2.3 min · 2.8 tok/s | 0.8 min · 14.4 tok/s |
| Maths, with system prompt | 5.7 min · 2.7 tok/s | 1.2 min · 16.1 tok/s |
| All four | 73 min, 4/4 correct | 15 min, 4/4 correct |
Same answers both times. I read all eight.
So why 20.5 on one and 6.6 on another? I assumed the long prompt. Wrong. The slow stretches lined up with other things using the GPU: an embedding job in Ollama, and macOS Photos doing its own image analysis in the background. Plus fifteen Chrome processes that had been pinning the processor for days.
A 190 GiB model at 20 tokens a second needs the whole machine. Before you measure anything, quit what you can and open Activity Monitor → Window → GPU History. Otherwise you're benchmarking your Photos library.
Whose fault was it?
I wanted to know, mostly so I'd know whether the next model will do the same thing.
The download was fine: the files add up to exactly the byte count the model's index lists, and a corrupt file gives wrong answers or a crash. It's the software, and it's a pile-up of three sensible choices:
- fp16 scales are a reasonable choice for whoever converts the model (OrcaRouter, here).
- Float32 hyper-connections are a reasonable choice for whoever writes the code.
- Upgrading mixed pairs to float32 is a reasonable choice for MLX.
Support for this model was written within days of its release and checked for right answers, which is what people check first. Each piece makes sense on its own. Stacked together, they cost a factor of ten, and each one looks fine from where it sits.
Hugging Face's MoE handbook has a training-side version of the same story: two of their runs quietly fell back to plain network connections and ran about 25 times slower. The giveaway was one line in a log.
I didn't debug this alone. Claude Code (Anthropic's coding agent) did the digging with me. Fable 5.1 read the code and file headers as a second opinion and found the same two leaks. And I asked GLM 5.3 Flash itself, the cloud version, what was wrong with it. It missed the scales, but it suggested two of the speed ideas further down.
And a big thank you to Alex Cheema and the exo team. Their write-up in pull request #2517 had already described bugs 3 and 4 on this model, and it told us we were on the right track.
The fixes, in one file
Save this as patches.py and import it before you load the model, every time. Without bug 2's fix, the very first prompt can run a 256 GB Mac out of memory, so don't try a run without it "to see the difference". It wraps mlx-vlm 0.7.12's own functions and leaves the library's files alone. It's written for exactly that version: if you upgrade mlx-vlm, check the speed and the answers again before trusting it.
"""Fixes for GLM 5.3 Flash (mlx-community/GLM-5.3-Flash-4bit) on mlx-vlm 0.7.12. Import before load()."""
import json, os
import mlx.core as mx
import mlx.nn as nn
MODEL_DIR = os.path.expanduser("~/models/GLM-5.3-Flash-mlx-4bit")
# Bug 1 (won't load): mlx-vlm fuses each shared expert's gate_proj + up_proj into gate_up_proj, but the
# checkpoint lists their bit widths under the separate names. Point the fused name back at gate_proj,
# after checking gate and up always share the same settings.
import mlx_vlm.models.glm5_next.glm5_next as _g5
_q = json.load(open(os.path.join(MODEL_DIR, "config.json")))["quantization"]
for _k, _v in _q.items():
if isinstance(_v, dict) and _k.endswith(".gate_proj"):
assert _q.get(_k[:-len("gate_proj")] + "up_proj") == _v, f"gate/up quantization differ at {_k}"
_g5.Model.quantization_path_aliases = lambda self, p: (
(p.removeprefix("language_model.").replace(".gate_up_proj", ".gate_proj"),) if ".gate_up_proj" in p else ())
# Bug 2 (memory spike): sparse attention reached its fused kernel in float32, which it refuses, and fell
# back to a huge copy. Hand the kernel bf16, give the result back in the original type.
import mlx_vlm.models.glm5_next.language as _gl
_sparse_orig = _gl._sparse_prefill_attention
def _sparse_bf16(q, k, v, indices, scale, **kw):
bf = mx.bfloat16
return _sparse_orig(q.astype(bf), k.astype(bf), v.astype(bf), indices, scale, **kw).astype(q.dtype)
_gl._sparse_prefill_attention = _sparse_bf16
# Bug 3 (float32 into every block): the hyper-connection streams are float32, so attention, MLP and
# expert blocks all ran in float32. Feed each block bf16, and hand its output back as float32 so the
# hyper-connection mixing stays float32 (half the drift of bf16 streams, same speed).
for _cls in (_gl.Glm5NextLinearAttention, _gl.Glm5NextAttention, _gl.Glm5NextMLP, _gl.Glm5NextMoE):
def _mk(orig):
def f(self, x, *a, **k):
r = orig(self, x.astype(mx.bfloat16), *a, **k)
return (r[0].astype(mx.float32), *r[1:]) if isinstance(r, tuple) else r.astype(mx.float32)
return f
_cls.__call__ = _mk(_cls.__call__)
# Bug 4a (fp16 scales x bf16 inputs -> float32): run each 4-bit multiply in fp16, the scales' own
# type, so the scales aren't rounded, and hand back bf16.
from mlx_vlm.models.switch_layers import QuantizedSwitchLinear as _QSL
for _cls in (nn.QuantizedLinear, _QSL):
def _mk16(orig):
def f(self, x, *a, **k):
if self["scales"].dtype == mx.float16 and x.dtype == mx.bfloat16:
return orig(self, x.astype(mx.float16), *a, **k).astype(mx.bfloat16)
return orig(self, x, *a, **k)
return f
_cls.__call__ = _mk16(_cls.__call__)
# Bug 4b (117 ms per token): the final hidden state reached the bf16 output layer in float32.
_tm_call = _gl.Glm5NextTextModel.__call__
_gl.Glm5NextTextModel.__call__ = lambda self, *a, **k: _tm_call(self, *a, **k).astype(mx.bfloat16)
# Bug 5 (draft head won't load; only matters if you use it): load_drafter() renames the guessing layer to
# "mtp_block.*" but looks its bit widths up under the old name, so its 5- and 6-bit parts are read as 4-bit.
# Copy each entry to the new name (and the fused gate_up_proj name, as in bug 1).
from mlx_vlm.speculative.drafters.glm5_next_mtp.split import Glm5NextMTPSplitter as _MTPS
_mtp_quant = _MTPS.quantization
def _mtp_quant_aliased(self, weights, source_config, text_config, quant_opts):
q = _mtp_quant(self, weights, source_config, text_config, quant_opts)
if not isinstance(q, dict):
return q
q, n = dict(q), int(text_config["num_hidden_layers"])
for k, v in list(q.items()):
for pre in (f"model.language_model.layers.{n}.", f"language_model.model.layers.{n}.",
f"model.layers.{n}.", f"layers.{n}."):
if isinstance(v, dict) and k.startswith(pre):
rest = k[len(pre):]
q["mtp_block." + rest] = v
if rest.endswith("shared_experts.gate_proj"):
q["mtp_block." + rest.replace("gate_proj", "gate_up_proj")] = v
return q
_MTPS.quantization = _mtp_quant_aliased
Setting it up
You'll need a 256 GB Mac (the model peaks around 192–194 GiB while it runs, a few GB more with the draft head), about 204 GB of free disk, and Python 3.11 (brew install python@3.11). Use a fresh virtual environment, and pin these versions, because the patch wraps 0.7.12's functions by name:
python3.11 -m venv ~/venvs/mlx-vlm-glm5
~/venvs/mlx-vlm-glm5/bin/pip install \
"git+https://github.com/Blaizzy/mlx-vlm.git@7230940302cbe602054650f18a2f77db945bc57a" \
"mlx==0.32.4" "transformers==5.22.1"
~/venvs/mlx-vlm-glm5/bin/hf download mlx-community/GLM-5.3-Flash-4bit --local-dir ~/models/GLM-5.3-Flash-mlx-4bit
Then save this as run_flash.py next to patches.py:
import mlx.core as mx
import patches # must come before load()
from mlx_vlm import load, stream_generate
mx.set_memory_limit(205 * 2**30) # a guideline for MLX, not a hard stop
mx.set_cache_limit(2 * 2**30)
model, processor = load(patches.MODEL_DIR)
tok = processor.tokenizer if hasattr(processor, "tokenizer") else processor
text = tok.apply_chat_template([{"role": "user", "content": "Why is the sky blue?"}],
tokenize=False, add_generation_prompt=True)
for r in stream_generate(model, processor, text, max_tokens=4096, temperature=0.7, top_p=0.95):
print(r.text, end="", flush=True)
print(f"\n\n{r.generation_tokens} tokens at {r.generation_tps:.1f} tok/s, peak {r.peak_memory:.1f} GB")
~/venvs/mlx-vlm-glm5/bin/python run_flash.py
Expect a long think before the answer: the model's chat template turns thinking on at maximum effort by default.
The other scripts (the layer-by-layer memory test, the number-format audit, the 8-layer speed test, and the launcher with the watchdog) are in the repo for this post.
Is yours paying the tax?
Check without loading anything. This reads only the small header at the top of each model file. Seconds, almost no memory:
import collections, glob, json, os, struct
d = os.path.expanduser("~/models/GLM-5.3-Flash-mlx-4bit")
c = collections.Counter()
for f in glob.glob(f"{d}/model-*.safetensors"):
with open(f, "rb") as fh:
n = struct.unpack("<Q", fh.read(8))[0]
for k, v in json.loads(fh.read(n)).items():
if k.endswith(".scales"):
c[v["dtype"]] += 1
print(c) # this model: Counter({'F16': 37338})
If the scales say F16 and the model runs in bf16 ("dtype": "bfloat16" in its config.json), you're paying the tax unless something converts for you. It's worth running on every shrunk MLX model you download.
Know your number. On a quiet, cool M3 Ultra with these fixes, expect about 20 tokens a second on a short prompt, and about 25 with the speed setup further down. At 2 or 3, you're paying the tax somewhere. At 6 to 15, check what else is using the GPU first, then the heat.
Check the answers as well as the speed. Every fix here was compared against the slower version's output before it went near the full model, and the full model had to get the same questions right afterwards. A fast wrong model is just a differently broken model.
The pilot, and the dust
Four questions is a smoke test. On Saturday night I ran a pilot: 48 questions, each asked plain and with the long system prompt, 96 answers. It finished at 03:52 with every answer saved and the scores where they should be. The speed was off. The first ten minutes ran at 19 to 22 tokens a second, then it dropped and sat at a flat 11 for the rest of the night, on short answers and long ones alike.
On Sunday morning I ran the 8-layer speed test back to back. It held 111 tokens a second for about five minutes, then slid: 105, 99, 81, 57. After a rest it started fast again and fell within seconds. Fast when cool, slow under load. That's heat.
macOS keeps quiet about it: pmset -g thermlog had recorded nothing. powermetrics shows it, if you run it while the model is working:
sudo powermetrics --samplers gpu_power,thermal -i 2000 -n 5
A minute and a half into the load, thermal pressure read Nominal, with the GPU at 1,380 MHz drawing 35.7 W. Four minutes later it read Heavy: 1,144 MHz, 27 W. macOS slows the GPU down to protect it, and a model that needs the whole GPU for hours feels every bit of that.
The fix was a clean. The Mac was overdue a de-dusting. Afterwards the same test held 110 to 112 tokens a second for ten minutes with no slide, and every long run since has kept its speed for hours.
If your speed drops after a few minutes and comes back after a rest, check the thermal pressure before you check the code.
Past 20
The Saturday result left room. Adding up the file headers, the Mac was reading about 18 GB from memory for every token. At 20.5 tokens a second that's around 365 GB/s, and the M3 Ultra's memory bandwidth is about 800.
About 9.9 GB of that 18 sat in parts the conversion had left at full 16-bit size: the attention in 34 of the 45 layers, and the output layer that scores every possible next word. I quantised them as the model loads, which leaves the files on disk alone, and checked each size on the 8-layer slice against the unchanged model:
| Attention stored at | Same top next word | 8-layer speed |
|---|---|---|
| 16-bit (as downloaded) | — | 113 tok/s |
| 8-bit | 100% | 145 tok/s |
| 4-bit | 92.6% | 159 tok/s |
The speeds include the output layer at 8-bit, which moved the model's predictions even less than the 8-bit attention did.
At 4-bit, the model would pick a different next word about one time in thirteen. That's a different model, so I went with 8-bit. On the full model it took the speed from 20.5 to about 25 tokens a second, with the same four test answers right.
My back-of-envelope sums had said the low-to-mid 30s. They only counted bytes. After the change the Mac reads about 13 GB per token, which at 25 a second is about 330 GB/s, still well under the 800. The rest is fixed cost: each layer takes between half a millisecond and a millisecond per token however little it reads, and there's another 1.6 ms per token outside the layers. Shrinking the weights further mostly shrinks the part that had stopped being the problem.
The draft head. Flash ships with an extra layer trained to guess the next token. With speculative decoding, the cheap guess goes first and the full model checks it, so you can get more than one token per pass. mlx-vlm 0.7.12 already has a drafter for it. It wouldn't load: the same mix-up as bug 1, this time with the guessing layer's bit widths filed under its old name (bug 5, below). Once loaded, it adds about 5 GB.
| Question | 8-bit | 8-bit + draft head |
|---|---|---|
| Health, plain | 24.6 | 25.9 |
| Health, with system prompt | 24.5 | 25.5 |
| Maths, plain | 25.3 | 31.5 |
| Maths, with system prompt | 24.8 | 30.4 |
On these four, the maths went up by about a quarter and the health answers by about 4%. Across the 200 questions later, almost every subject came out between 24 and 27 tokens a second, maths included, so overall the draft head is worth a few per cent here. mlx-vlm's own test (pull request #2127, on a 512 GB M3 Ultra) went from 31.2 to 43.7. My guess at the gap: I sample at temperature 0.7, and a sampled token is harder to guess than the single likeliest one. I haven't tested that.
Fewer experts. Each token normally goes through 8 of the 288 experts. Using 6 means a quarter less expert weight to read. On the 8-layer slice it was 4% faster and changed the top next word 3.7% of the time, about as much as 4-bit attention, which I'd already turned down. It changes what the model computes, for very little speed, so it stays off.
So: about 26 tokens a second on long answers, from 2 at the start. Someone has gone further with a different engine. DwarfStar (ds4), antirez's small engine for Macs, has its own 4-bit GLM 5.3 Flash at 178 GiB, small enough for a 256 GB Mac. On r/LocalLLaMA, u/IngeniousIdiocy shared an M3 Ultra branch of it, tuned to this one chip. It takes DwarfStar from 29 to 40 tokens a second on short prompts, and from 24 to 38 with 62,000 tokens of conversation behind it, with the same outputs. In a long Claude Code session they averaged over 38. Their Mac has 512 GB; I haven't tried it on 256 yet.
The speed setup is one more small file. Save it as speed.py next to patches.py:
"""Optional speed setup for GLM 5.3 Flash on mlx-vlm 0.7.12. Import after patches.py; call after load()."""
import mlx.core as mx
import mlx.nn as nn
def shrink_attention(model, bits=8):
"""Store the linear-attention layers and the output layer at 8-bit, in memory. Files on disk are untouched."""
lm = model.language_model
def pick(path, m):
if not isinstance(m, nn.Linear) or isinstance(m, nn.QuantizedLinear) or ".self_attn." not in path:
return False
if path.rsplit(".", 1)[-1] not in ("qkv_proj", "o_proj"):
return False
layer = int(path.split("layers.")[1].split(".")[0])
return {"group_size": 64, "bits": bits} if lm.config.layer_types[layer] == "linear_attention" else False
nn.quantize(lm.model, class_predicate=pick)
lm.lm_head = nn.QuantizedLinear.from_linear(lm.lm_head, group_size=64, bits=8)
mx.eval(model.parameters()) # do it here, on the main thread: generation runs in another one
And in run_flash.py, after load():
import speed
from mlx_vlm.speculative.drafters import load_drafter
speed.shrink_attention(model)
draft, kind = load_drafter(patches.MODEL_DIR, kind="mtp") # needs bug 5's fix in patches.py
mx.eval(draft.parameters())
# then add draft_model=draft, draft_kind=kind to the stream_generate() call
200 questions
Speed is only worth having if the answers hold. The fair test is the same model run properly: Z.ai serves Flash in the cloud as its own FP8 release, and Ollama can call it. Both answered the same 200 questions from MMLU-Pro, across 14 subjects from law to engineering, each with up to ten options. Every question went in twice, plain and with the long system prompt, on the same settings: thinking on, temperature 0.7, up to 16,384 tokens of thinking. Answers that ran out of room ran again with up to 32,768.
| Right, out of 200 | Plain | With system prompt |
|---|---|---|
| My 4-bit Flash, on the Mac | 178 | 179 |
| Z.ai's own Flash, in the cloud | 178 | 176 |
On the questions both finished, they gave the same letter 184 times out of 197 asked plain, and 183 out of 197 with the system prompt. Neither gap is bigger than chance would give. After the four fixes, the 8-bit attention and the draft head, the 4-bit copy on my Mac answers these the way Z.ai's own does. Its replies run to about the same length, too, thinking included: a median of 570 tokens asked plain, against the cloud's 576.
For a second reference point, I'd already run the same questions on the big, text-only GLM 5.3. Asked plain, on the questions both finished, it scored 90.8%, and so did my Flash.
Five of my answers still ran out at 32,768 tokens. One was going round in circles. The other four were still reasoning when they hit the limit, 21 to 23 minutes in, at about 24 tokens a second to the end. Z.ai's ran out on two.
Where it stands
- About 26 tokens a second on long answers and 22 with a long system prompt, on a 256 GB Mac with the dust cleaned out.
- Over 200 questions it scores the same as Z.ai's own Flash in the cloud.
- I've reported all five to mlx-vlm (issue #2714). As of 11 November its code for this model is unchanged, so until a fix lands, a fresh install will hit all five.
- Long prompts are barely tested: most of mine were under 2,000 tokens.
If there's more, I'll post it.
If you're seeing 2 tokens a second
You're paying the float32 tax. Run the header check. If the scales are F16, import patches.py before you load the model, quit everything else, and expect about ten times the speed.
The model was fast all along. Every part of the stack was doing exactly what it was told, carefully, at four times the cost.
Runs: 6–10 November 2026. Mac Studio M3 Ultra, 256 GB. mlx 0.32.4, mlx-vlm 0.7.12 (git @7230940), transformers 5.22.1, Python 3.11. Model: mlx-community/GLM-5.3-Flash-4bit (204 GB, mixed 4/5/6-bit, groups of 64). Temperature 0.7, top_p 0.95, thinking at the chat template's default (maximum) effort, one request at a time. From 8 November: linear attention and output layer at 8-bit in memory, plus the model's MTP draft head.