A Few Concepts on LLM
1. Token
A model doesn’t read whole words. It reads tokens, which are common words or pieces of words.
- “The screen isn’t huge” →
The·screen·isn·'t·huge
2. Parameter: a learned number
| Gemma 4 26B A4B | Qwen 3.6 35B A3B | |
|---|---|---|
| Layers | 30 | 40 |
| Experts per layer | 128 | 256 |
| Used per token | 8 + 1 shared | 8 + 1 shared |
| Total parameters | ~25.2B | ~35B |
| Active per token | ~3.8B | ~3B |
A parameter is one number inside the model, set during training. Think of each one as a tiny knob. Training adjusts billions of knobs until the model predicts text well.
- “26B” = about 26 billion parameters.
- The model file is mostly these numbers. At Q4_K_M each one takes about half a byte, which is why Gemma’s file is 16.8 GB.
- More parameters generally means more capacity to learn, but also more memory needed.
A small router picks a few experts for each word (token), and the rest sit idle for that token. That’s why a 26B model only does about 4B worth of work per token, which explains the “A4B” (active 4B) in Gemma’s name.
3. Layer: one processing step
The parameters are organized into layers stacked in order. Every token passes through all of them, bottom to top, like an assembly line with 30 stations (Gemma) or 40 (Qwen).
Each layer does two things:
- Attention: each token looks at the other tokens and takes in context. Here
hugeconnects toisn't, and both connect toscreen. - Feed-forward: each token is processed further using what the layer learned in training.
4. Experts: specialists inside each layer (MoE models only)
In an ordinary (“dense”) model, the feed-forward part of each layer is one big block, and every token uses all of it.
In a mixture-of-experts model, that block is split into many smaller experts, and a small router picks a few for each token:
Think of a hospital floor with 128 specialists, where a triage nurse sends each patient to 8 of them. The hospital employs everyone (total parameters, all in memory), but each patient sees only a few (active parameters, the work done per token).
5. Python Code Example
N_CPU_MOE = 0 if int(mem_mib) / 1024 >= 22 else 18
The “MoE” in --n-cpu-moe stands for mixture of experts, the design both of the Qwen and Gemma mentioned above use. The setting counts layers, not experts.

Discussion