The Lab

Glossary

Words from the research posts, each explained once. In a post, any word with a dotted underline links here: tap or hover for the short version.

Model (large language model)

The AI program behind chatbots like ChatGPT and Claude: trained on huge amounts of text to predict what comes next, one piece at a time.

A language model learns from billions of pages of writing until it is very good at continuing text. Ask it a question and it writes an answer the same way, piece by piece. The models we test are smaller ones that can run on a home computer, made by companies like Alibaba (Qwen) and Google (Gemma).

Token

The pieces a model reads and writes: whole words, parts of words, or punctuation. A thousand tokens is roughly 750 English words.

A model doesn't see letters or whole words. Text is chopped into tokens, and the model writes one token at a time, each time picking the next piece. That is also why limits are counted in tokens: a cap of 4,096 tokens is about 3,000 words.

Thinking

Working notes many current models write before they answer. Apps usually hide them, but in our tests we can read them and count them.

Before replying, a 'thinking' model writes out its reasoning: trying options, checking, changing its mind. The answer comes after. We measure things like how long the thinking is, and how often it stops to check itself (a 'wait' or 'actually').

Parameters (the '9B' in a model's name)

The settings a model learns during training, like billions of tiny dials. '9B' means 9 billion of them: a rough measure of the model's size.

Training turns those dials, a tiny bit at a time, until the model is good at predicting text; then they stay fixed. The count is often part of the name: Qwen 3.5 9B (from Alibaba) has about 9 billion, Gemma 4 31B (from Google) about 31 billion. Bigger is usually more capable, but needs a bigger computer to run.

Base model and chat model

A base model has only learned to continue text. A chat (or 'instruct') model has been trained further to follow instructions and act as an assistant.

Every chat assistant starts as a base model, trained on a huge amount of text to predict what comes next. Further training then teaches it to answer questions, follow instructions and refuse some requests. Comparing the two shows what that extra training added.

System prompt

Instructions given to a model before the conversation starts, by whoever runs it. The person chatting usually never sees them.

When you use a chatbot, the company has already told it who to be and what rules to follow, in text placed before your first message. Because the model treats those instructions differently from what a user says, a claim can land differently depending on whether it arrives in the system prompt or in the chat.

Activations (the model's inner activity)

The numbers inside a model at each step as it works. Looking at them shows what is going on inside, not just what it writes.

A computer can only work with numbers, so every token the model reads is turned into a long list of numbers that stands for what that piece of text means in its context. That list is passed up through the model's layers and changed at each one. These changing numbers are the activations. Unlike the parameters, which stay fixed, they're different for every sentence, and patterns in them line up with things like whether a task is finished or who the model thinks it's talking to.

Layer

Models work in stages stacked one on top of another. Each stage is a layer; a typical model has a few dozen.

Text passes up through the layers one after another. Roughly: early layers deal with the words themselves, middle layers with meaning and planning, and the last layers with picking the next token. Most of what we measure is taken from the middle layers.

Direction

A pattern in a model's inner activity that is stronger in one situation than another, found by comparing the two.

We collect the model's activations on two sets of examples (say, a crossword it has finished and one it hasn't), average each set, and take the difference. That difference is the direction. What it means comes from the two sets we chose, not from the name we give it.

Steering

Changing what a model does by adding a pattern directly to its inner activity while it works, instead of changing what you say to it.

We take a direction and add it to the model's activations at one layer, at a chosen strength, while it writes. If the behaviour changes and a random pattern of the same size doesn't change it, the direction is doing something real. Strength is measured against the normal size of the activity: 0.4 means 40% of it.

Removing a direction (ablation)

Taking one pattern out of a model's inner activity at every step, so it can't use it, then watching what still works.

The opposite of steering. At every step, the share of the model's activity that matches the pattern is subtracted, so that pattern is gone while everything else stays. If the model carries on as normal, that pattern wasn't needed for the behaviour. If it breaks, it was.

Lens and J-space

A lens is a translation key that turns a model's inner activity (just numbers) into the words that activity is leading it to say. J-space is the small part of the activity that carries those words.

Inside, a model's activity is only numbers. Researchers at Anthropic, the company that makes the Claude chatbot (Gurnee and colleagues, 2026), worked out a key, calculated from the model's own workings, that shows which words a given moment's activity would most likely lead it to say, now or a little later. They found those 'things it could say' are carried by a small part of the activity, which they call J-space (the J comes from the name of the maths used to build the key; you don't need it to use the idea). We use the key to see what a model is leaning towards before it writes.

Interval (range of uncertainty)

The range a result would probably land in if the test were run again with different examples. If two ranges don't overlap, the difference is probably real.

Any test uses a limited set of questions, so a number like '70%' could have come out a bit higher or lower with other questions. An interval shows how much. Wide intervals mean we can't be sure; two results whose intervals overlap may not really differ.

Log-odds

A scale for how strongly a model favours one of two answers. Zero means no preference; the further from zero, the stronger the lean.

Instead of 'it chose A', we can read how much more likely the model thought A was than B. On the log-odds scale, 0 means 50/50. Around +2 means it thinks A is about seven times as likely as B; around +5, about 150 times. Negative numbers are the same lean towards B. (The 'log' is short for logarithm, the maths behind the scale; you don't need it to read the numbers.)

AUC (area under the curve)

How well a measure tells two groups apart. 0.5 is no better than a coin toss; 1.0 is perfect.

Rank every example by a measure (say, how strongly a pattern shows up) and ask: if I pick one 'finished' example and one 'unfinished' one at random, how often does the finished one come out on top? That chance is the AUC. The name comes from a graph statisticians draw. Above 0.9 is a very clean separation.

Mixture of experts (MoE)

A way of building a big model so that each word only goes through a few of its many smaller parts, the 'experts'. The model can be huge while each word costs much less to work out.

In most layers of a mixture-of-experts model, one big block of calculations is replaced by many smaller ones, the experts, often over a hundred. For every token, a small part of the model called the router scores the experts and picks a handful, say 8, and only those run. Many MoE models also have a shared expert that every token uses. That's how a model can have 320 billion parameters but use only 18 billion per token: all the experts still have to sit in memory, but each token only does the work of a few. The experts aren't tidy specialists in human subjects like 'maths' or 'French'; what each one ends up handling is learned in training.

Quantisation (4-bit, 8-bit)

Shrinking a model by storing each of its numbers in fewer bits, so it takes less memory and usually runs faster, at a small cost in accuracy.

A model's parameters are usually stored at 16 bits each. Quantising them to 8 or 4 bits halves or quarters the size, which is often what makes a model fit on a home computer. A common method splits the numbers into small groups (say 64), storing each group as small whole numbers plus a scale and an offset that turn them back into real values when they're used. A '4-bit' model is often mixed, with the parts that matter most kept at 5, 6 or 16 bits. Done well, the answers barely change. The choices made when a model is converted can also affect how fast it runs, not just how big it is.

Checkpoint

The saved files that make up a trained model: its parameters, plus files describing how they fit together. Downloading a model means downloading a checkpoint.

A checkpoint is a snapshot of a model's parameters saved to disk. For open models it's usually a set of large .safetensors files holding the numbers, a config.json describing the model's shape and settings, and the tokenizer. One model can have many checkpoints: the original release, plus versions other people have converted for different software or quantised to smaller sizes. Those versions should give the same answers, but they don't always behave or run identically.

dtype and precision (float32, bf16, fp16)

The format a computer uses to store each number (its data type, or dtype). Bigger formats are more precise; smaller ones use less memory and run faster.

Numbers with a decimal point are stored in standard formats. float32 uses 4 bytes per number and is the safe default. bf16 and fp16 use 2 bytes each but split their bits differently: bf16 keeps float32's huge range with less precision, while fp16 is more precise over a smaller range. Most models today run in bf16. When two different formats meet in one calculation, software often converts both to a bigger format to be safe. That's correct, but it can make the work several times slower without any warning. 'Precision' is the general word for how many bits each number gets.

Kernel (GPU kernel)

A small program that runs on the graphics chip (GPU) and does one job, such as a big multiplication. Which kernels a model uses decides much of its speed.

When a model runs, each calculation is handed to the GPU as a kernel. Fast, purpose-built kernels can be many times quicker than general ones, but they often only accept certain inputs, such as one particular number format, and when they don't, software can quietly fall back to a slower path. Each kernel launch also has a small fixed cost, so lots of tiny operations are slower than a few big ones. On Apple computers the GPU programming language is called Metal, so you'll see 'Metal kernels'.

Unified memory

On Apple Silicon Macs, the main processor and the graphics chip share one pool of memory, so the graphics chip can use almost all of it. That's why a Mac with lots of memory can run very large models.

On a typical PC, the graphics card has its own separate memory, often 8 to 32 GB, and a model that doesn't fit there has to be split up or run slowly. Apple's M-series chips put the processor and graphics on one chip sharing the same memory, so a 256 GB Mac Studio can hold a model of around 200 GB. The trade-off is speed: that shared memory is fast, but it can usually be read more slowly than a top graphics card's own memory, and for big models, how fast memory can be read is often what limits tokens per second.

GB and GiB

Two ways of counting gigabytes. A GB is a billion bytes; a GiB is 1,073,741,824 bytes (2 to the power of 30), about 7% more.

Disk and download sizes are usually given in GB. Computer memory comes in powers of two, so it's really counted in GiB, but macOS's Activity Monitor labels it 'GB' anyway: a Mac sold with '256 GB' of memory has 256 GiB. So the same model can be 204 GB on disk and 190 GiB in memory without anything changing. The 7% gap rarely matters, until a model nearly fills the machine.

MLX (and mlx-lm, mlx-vlm)

Apple's free framework for running and training AI models on Apple Silicon Macs. mlx-lm runs text models on it; mlx-vlm runs models that can also take images.

MLX is made by Apple's machine-learning research team and is designed around the Mac's unified memory. Libraries for particular kinds of model sit on top of it: mlx-lm, from the same team, for text-only language models, and mlx-vlm, a community project, for vision-language models that can also read images or video. A model that can see images runs in mlx-vlm even if you only talk to it. Support for a brand-new model usually arrives days or weeks after its release, sometimes in the libraries' development versions first.

Attention

The part of each layer that lets a model look back over everything earlier in the conversation and pick out what matters for the next token.

For each new token, attention compares it with the tokens before it and pulls in information from the ones that seem relevant. Doing that in full gets more expensive the longer a conversation grows, so newer models mix in cheaper kinds. Linear attention keeps a running summary instead of looking at everything, and sparse attention looks at only a limited number of the most relevant earlier positions. Some models, such as GLM 5.3 Flash, combine both.

Residual stream

The running total of a model's working state, passed from layer to layer. Each layer reads it and adds its own contribution.

Rather than each layer replacing what came before, a model keeps one long list of numbers for each token, and every layer adds to it. That makes very deep models far easier to train. The activations that researchers study are usually read from this stream. Some newer designs, such as hyper-connections, keep several parallel streams for each token and mix them between layers.

Cosine similarity

A score for how closely two long lists of numbers point the same way. 1.0 means the same direction, 0 means unrelated, and −1 means opposite.

A model's activations are long lists of numbers, which mathematicians treat as arrows pointing somewhere in a space with thousands of directions. Cosine similarity compares only which way two arrows point and ignores how long they are. It's a quick check that two versions of a calculation still agree: 0.9999 means all but identical, 0.99 means close but drifting. It's also used to ask whether two patterns inside a model are related.

Speculative decoding (and MTP)

A speed trick: something cheap guesses the next few tokens, and the full model checks them all at once and keeps the ones it agrees with. You get what the model would have written anyway, just faster.

A big model writes one token at a time, and for each one it has to read most of the parameters it uses from memory. Checking several guessed tokens in one pass costs little more than writing one. So a small draft model, or a small extra layer built into the main model for the purpose (multi-token prediction, or MTP), proposes the next few tokens; the main model accepts those it agrees with and replaces the first one it doesn't. When the guesses are often right, as in predictable text like code, the speed-up can be large.

Chat template

The model's built-in recipe for laying out a conversation as one piece of text, with markers for who's speaking. It also sets defaults, such as whether the model thinks first.

A model doesn't see a chat as separate messages: it sees one long text with special markers between the turns. The chat template, which comes with the model, does that layout. Using the wrong one, or a slightly different one, can change how a model behaves, which is why careful tests send prompts exactly as the model was trained to see them. Templates often carry settings too, such as how hard a thinking model should think.

Open-weight model

A model whose trained parameters are published for anyone to download and run on their own computer, unlike ChatGPT or Claude, which you can only use through the company that makes them.

'Open-weight' means the checkpoint is public, usually under a licence such as MIT or Apache, so you can run it offline, test it and change it. It doesn't always mean the training data or training code are public as well; that's 'open source' in the stricter sense. Labs such as Alibaba (Qwen), Google (Gemma), Z.ai (GLM) and DeepSeek release open-weight models.

Memory bandwidth

How fast data can be read from memory, in GB per second. For big models it's often what sets the speed limit.

To write each token, a model has to read the parameters it uses from memory. If that's 18 GB per token and the memory can supply 800 GB a second, the best it can possibly do is about 44 tokens a second, before any other overheads. That's why quantisation speeds models up as well as shrinking them, and why mixture-of-experts models, which read only some of their parameters for each token, suit home machines. An M3 Ultra Mac Studio is rated at about 800 GB/s.

Swap and page-ins

When memory runs short, macOS moves data out to the disk (swap) and fetches it back when it's needed (a page-in). A little is normal; a lot makes everything crawl.

The disk is far slower than memory, so a model that doesn't fully fit, or a machine busy with other work, can slow right down as data shuttles back and forth. Activity Monitor's Memory tab shows how much swap is in use. If a model is running slowly but its process shows hardly any page-ins, memory isn't the problem and the cause is somewhere else.

Benchmark

A fixed set of test questions used to score and compare models.

Because every model gets the same questions under the same settings, a benchmark makes results comparable, and repeatable after a change. MMLU-Pro, for example, is a widely used set of hard multiple-choice questions across subjects like health, law and maths, with up to ten options each. A score tells you how a model does on that set of questions, not how good it is at everything.

Matrix multiply (matmul)

The basic calculation inside a model: multiplying a list of numbers by a grid of the model's parameters. Models spend most of their time doing it.

Every layer turns its input into its output mainly through matrix multiplies, each one combining a list of numbers with a grid of parameters. Because there are so many of them, how they're run decides most of a model's speed: in which number format, with which kernel, and whether the parameters are quantised.

Virtual environment (venv)

A self-contained folder holding Python and the add-ons for one project, so installing things for it can't break anything else on your computer.

AI tools often need particular versions of their libraries, and two projects may need different ones. A virtual environment keeps each project's set separate: you create it once, install into it, and run that project with its own copy of Python. If something goes wrong, you can delete the folder and start again without touching the rest of your system.