The 0.47 MB That Makes Your LLM 18% Faster

A 2.6B model has 30 transformer layers. Each layer has three weight groups: a layer norm (0.016 MB), an attention block (8 MB), and an FFN block (38 MB).

The standard GGUF layout interleaves them: norm, attn, FFN, norm, attn, FFN, repeated 30 times. Every time the model reads a layer norm on cold start, the preceding 38 MB FFN block has already evicted it from L3 cache. The norm has to reload from RAM.

All 30 layer norms together are 0.47 MB. My L3 cache is 3 MB.

I moved all 30 norms to sit together at the front of the file. Generation speed went up 18% on cold start.

The Idea

The previous Gattai-LM post was about splitting model layers across devices. This experiment is about a different axis: the order tensors are laid out on disk.

reorder.py rewrites a GGUF file so tensors appear in access-frequency order rather than the original per-layer interleaved order. The hypothesis: if small, frequently-accessed tensors live together on disk, the OS page cache loads them together, and the CPU L3 cache holds them across all 30 layers instead of constantly reloading from RAM.

The model: LFM2.5-2.6B-Q4_K_M (1,596.9 MB, 266 tensors, 30 layers). Hardware: Intel Core i3, 3 MB L3 cache, Windows. Benchmark: llama.cpp llama-cli build b10549.

What a GGUF File Actually Looks Like

GGUF is a binary format that stores model metadata followed by raw tensor data. By default, llama.cpp writes tensors in layer order:

[token_embd — 205 MB]
[layer 0]  0.5 MB norm | 8 MB attn | 38 MB FFN
[layer 1]  0.5 MB norm | 8 MB attn | 38 MB FFN
...×30
[output — same weights as token_embd, 205 MB]

During cold start (model not in OS page cache), the OS reads the file sequentially. By the time layer 1's norm is needed, layer 0's 38 MB FFN has already filled and blown the L3 cache. Layer 1's norm must be fetched from RAM. This repeats 30 times.

The Reordering Scheme

reorder.py classifies each tensor by access frequency and rewrites the file:

Reordered layout:

[HOT  — token_embd + output, 205 MB]
[WARM — all 30 norms, 0.47 MB]     ← fits in 3 MB L3
[MED  — all 30 attn blocks, 245 MB]
[COLD — all 30 FFN blocks, 1139 MB] ← blows cache last

The FFN blocks still blow the L3 cache — they always will at 38 MB each. The difference is that they come last, after norms and attention are already processed. They don't pollute anything that needs to be read again.

Results

Cold start (model not yet in OS page cache):

Warm cache (model already resident in OS page cache):

Prompt processing is essentially unchanged in both conditions. Generation speed improves 18% cold, then the effect disappears entirely once the OS page cache is warm.

Why the Cold-Start Gain Is Real

The mechanism is straightforward:

Original layout — what happens per-token during cold start:

Read layer 0 norm  (0.016 MB)  → loads into L3
Read layer 0 attn  (8 MB)      → L3 partially evicted
Read layer 0 FFN   (38 MB)     → L3 fully evicted
Read layer 1 norm  (0.016 MB)  → cache miss → RAM fetch
Read layer 1 attn  (8 MB)      → L3 partially evicted
Read layer 1 FFN   (38 MB)     → L3 fully evicted
... ×30

Every norm access after layer 0 is a RAM round trip at ~20 GB/s, competing with the 38 MB FFN that just ran.

Reordered layout — what happens:

Read all 30 norms (0.47 MB total) → loaded once, stay in 3 MB L3
Read layer 0 attn (8 MB)
Read layer 0 FFN  (38 MB)         → L3 evicted, but norms are done
Read layer 1 attn (8 MB)
Read layer 1 FFN  (38 MB)
... ×30

All 30 layer norms fit together in the 3 MB L3 cache. After the first pass, all subsequent norm accesses are L3 hits instead of RAM fetches. The FFN blocks still evict everything — but they arrive after the norms are no longer needed per-layer.

The confirmation: 0.47 MB WARM tier is less than 3 MB L3 cache. This is the only measurement that needs to hold for the mechanism to work, and it does.

Why Warm Cache Shows No Gain

Once Windows loads the model once, the entire file stays in the OS page cache — RAM, not disk. Subsequent loads read from RAM at roughly 20 GB/s regardless of tensor layout. Both models run at identical RAM speed, so there is no delta.

This also means the 18% figure applies specifically to first-run-after-boot (or after a cache flush). In steady-state usage where the model stays loaded, reordering has no effect.

One confound in this benchmark: the second baseline run started after the reordered model had already warmed the page cache. Its numbers are higher than a true cold baseline would show, which means the measured delta may actually understate the real cold-start gain.

When This Actually Helps

Reordering delivers the most benefit when:

• Model is loaded cold — first run after boot, or after swapping to a different model
• Model barely fits in available RAM — frequent paging back to disk makes layout order matter on every generation step, not just the first
• Multiple models swap frequently — each cold load benefits; a server that cycles through several models will see cumulative gains

For a single model that stays resident in RAM all day, the benefit is zero after the first load. For a developer who boots, runs inference, and shuts down — every session starts cold.

Benchmark Confounds

This test was not fully controlled. Being direct about what a clean benchmark needs:

Problems with this run:

• No OS cache flush between runs — Windows kept the first model in page cache, affecting the second run's baseline
• Reordered model ran multi-turn chat; original ran isolated single prompts — different usage patterns
• 40 tokens is too few for stable generation statistics (high variance per run)

Clean protocol for future runs:

# Flush OS page cache between every run
# (requires Sysinternals RAMMap → Empty → Empty Standby List, or restart)
 
.\llama-cli.exe --model <path> `
  --ctx-size 512 --threads 4 -p "Hello" -n 200
 
# Run 3x per model
# Discard run 1 (OS warmup artifact)
# Average runs 2-3

The +18% number is real — the mechanism is confirmed — but the exact figure needs a controlled rerun with cache flushes and more tokens. That's FINDINGS4.

Conclusion

Cache-aware weight reordering gives 18% faster cold-start generation on a 2.6B Q4 model running on an i3 with 3 MB L3 cache. The mechanism is small: 0.47 MB of layer norms fit entirely in L3 when laid out together, eliminating 30 RAM round trips per generation pass.

The effect is real and theoretically grounded. The measurement needs a cleaner protocol to pin down the exact number. Both of those things can be true.

reorder.py is in Jeff9497/Gattai-LM. Pass it a GGUF file and it outputs a reordered copy with the same weights, same quantization, different tensor layout.

Hardware: Intel Core i3-7020U, 3 MB L3 cache, Windows. Model: LFM2.5-2.6B-Q4_K_M (1,596.9 MB). Tool: llama.cpp build b10549.