A Few Concepts on LLM

1. Token

A model doesn’t read whole words. It reads tokens, which are common words or pieces of words.

  • “The screen isn’t huge” → The · screen · isn · 't · huge

2. Parameter: a learned number

Gemma 4 26B A4BQwen 3.6 35B A3B
Layers3040
Experts per layer128256
Used per token8 + 1 shared8 + 1 shared
Total parameters~25.2B~35B
Active per token~3.8B~3B

A parameter is one number inside the model, set during training. Think of each one as a tiny knob. Training adjusts billions of knobs until the model predicts text well.

  • “26B” = about 26 billion parameters.
  • The model file is mostly these numbers. At Q4_K_M each one takes about half a byte, which is why Gemma’s file is 16.8 GB.
  • More parameters generally means more capacity to learn, but also more memory needed.

A small router picks a few experts for each word (token), and the rest sit idle for that token. That’s why a 26B model only does about 4B worth of work per token, which explains the “A4B” (active 4B) in Gemma’s name.

3. Layer: one processing step

The parameters are organized into layers stacked in order. Every token passes through all of them, bottom to top, like an assembly line with 30 stations (Gemma) or 40 (Qwen).

Each layer does two things:

  1. Attention: each token looks at the other tokens and takes in context. Here huge connects to isn't, and both connect to screen.
  2. Feed-forward: each token is processed further using what the layer learned in training.

4. Experts: specialists inside each layer (MoE models only)

In an ordinary (“dense”) model, the feed-forward part of each layer is one big block, and every token uses all of it.

In a mixture-of-experts model, that block is split into many smaller experts, and a small router picks a few for each token:

Think of a hospital floor with 128 specialists, where a triage nurse sends each patient to 8 of them. The hospital employs everyone (total parameters, all in memory), but each patient sees only a few (active parameters, the work done per token).

5. Python Code Example

N_CPU_MOE = 0 if int(mem_mib) / 1024 >= 22 else 18

The “MoE” in --n-cpu-moe stands for mixture of experts, the design both of the Qwen and Gemma mentioned above use. The setting counts layers, not experts.

Leave a Comment

Your email address will not be published. Required fields are marked *