I Made My LLM 20% Smaller. It Got 13% Slower.

The previous Gattai-LM post showed that reordering a model's tensors by access frequency gives 18% faster cold-start generation. The natural next experiment: what if we also requantize the cold FFN weights to a lower bit-width? Less data to read from disk should mean faster loading.

The model shrank 20%. Generation speed dropped 13%.

This is that story.

The Result

The tier-split model takes the reordered file from the previous experiment and requantizes the FFN blocks — 71% of the model by size — from Q4_K_M down to Q3_K_M. File size drops from 1,597 MB to 1,270 MB. Generation speed drops from 6.9 t/s to 6.0 t/s.

Negative results are real results. Here is what this one reveals.

Why Fewer Bits Should Be Faster

CPU inference for a single sequence is memory-bandwidth-bound. Each token requires reading the full model weights from RAM, doing a matrix-vector multiply against the activation, and producing the next token. The bottleneck is how fast you can stream weights into the CPU, not how fast the CPU can compute.

Under this assumption, fewer bits = smaller file = fewer bytes to read = faster generation. Q3 uses 3 bits per weight vs Q4's 4 bits — a 25% reduction in bytes. On a bandwidth-bound system, this should translate directly to a 25% speedup.

The assumption is wrong, or at least incomplete.

The Dequantization Cost

CPU inference has two costs per token, not one:

time_per_token = bytes_read / bandwidth  +  flops / throughput
                 ─────────────────────────   ─────────────────
                   bandwidth cost              compute cost

The bandwidth cost is what we optimized for. The compute cost is what bit us.

Q4_K_M dequantization — tight AVX2 kernel, roughly 2 operations per element:

val = (qs[i] >> shift) & 0xF;          // nibble extract
out = d * scale[sub] * val - min[sub]; // scale + subtract

Q3_K_M dequantization — more complex, roughly 5 operations per element:

hi  = (hmask[i/8] >> (i%8)) & 1;       // extract high bit
lo  = (qs[i/4] >> (2*(i%4))) & 3;      // extract low 2 bits
val = (hi << 2) | lo;                   // reconstruct 3-bit value
out = d * scale[sub] * (val - 4);      // scale + offset

On an i3-7020U without AVX-512, Q3_K dequantization costs roughly 1.4× more compute per element than Q4_K. The 20% bandwidth saving is entirely eaten by 40% more dequantization work. The net is a regression.

Q4_K_M sits at the bandwidth-compute optimum for this CPU. Moving to lower bits saves bytes but costs disproportionate compute. The crossover has already passed.

The Revised Bottleneck Picture

Running the numbers reveals how much headroom actually exists:

Theoretical DDR4 ceiling   :  34 GB/s ÷ 1.60 GB = 21.3 t/s
Effective bandwidth (~75%) :  25 GB/s ÷ 1.60 GB = 15.6 t/s  (hard ceiling)
Current actual             :                        6.9 t/s  (44% of ceiling)

We are running at 44% of the effective bandwidth ceiling. The gap breaks down roughly as:

• ~25% lost to dequantization compute overhead (Q4_K_M is already near-optimal here)
• ~15% lost to memory latency and TLB pressure (4KB pages for a 1.6GB model)
• ~10% lost to thread synchronisation, softmax, norm, and other small ops
• ~6% lost to KV cache access competing with weight reads

Getting to 10 t/s means closing roughly 30% more of the gap. Lower quantization doesn't help. The gains are in the latency and overhead terms.

Every Optimization Tried So Far

Full summary across all Gattai-LM experiments to date:

The reorder result from FINDINGS3 remains the only positive finding on warm-cache generation. Everything else is noise or regression.

Getting to 10 Tokens Per Second

The three viable paths forward, ranked by implementation effort.

IQ Quantization

IQ3_S (3.44 bpw) and IQ4_XS (4.25 bpw) use non-linear codebooks with SIMD-optimised lookup-table dequantization instead of arithmetic reconstruction. For IQ3_S specifically: same bit-width as Q3_K but dequant is a table lookup (1 instruction) vs arithmetic (5 instructions).

This directly attacks the problem that killed the Q3_K_M experiment — compute cost at lower bit-width.

# Step 1: compute importance matrix from calibration data
.\llama-imatrix.exe --model baseline.gguf --calibration-data wiki.txt
 
# Step 2: tier-split with IQ quantization on FFN
python quant_tier.py --ffn-quant iq3_s

Expected: 1.60 GB shrinks to ~1.15 GB, dequant speed roughly equal to Q4_K_M. Estimated result: ~9.4 t/s — the bandwidth saving without the compute penalty.

iGPU via Vulkan

The i3-7020U has Intel HD Graphics 620 — 24 execution units, Vulkan 1.0, shared DDR4 memory with the CPU. If the llama-bin build includes the Vulkan backend:

# Check if Vulkan backend is available
.\llama-cli.exe --list-gpu-backends
 
# Offload FFN layers to iGPU
.\llama-cli.exe --model model_reordered.gguf --n-gpu-layers 20

Offloading the 90 FFN layers to the iGPU means the shader array handles the heavy matrix-vector multiply while the CPU handles attention and norms. Since memory is shared (no discrete VRAM transfer), there is zero copy cost. Potential: 1.5-2x on the FFN path.

This is the highest-ceiling option — and the most uncertain, since Intel HD 620 Vulkan performance under llama.cpp is essentially unbenchmarked.

TLB Pressure

Default mmap loads the model using 4KB pages. A 1.6GB model requires roughly 400,000 TLB entries. The i3-7020U's TLB is far smaller. TLB misses stall the memory pipeline on nearly every weight access.

--no-mmap forces eager loading into process RAM with contiguous pages, dramatically reducing TLB pressure:

.\llama-cli.exe --model model_reordered.gguf --no-mmap --threads 4

Expected gain: 5-15%. Zero code changes required. This is the lowest-effort experiment remaining and should have been tested earlier.

Software Prefetch

A custom inference loop that issues prefetch instructions for layer N+1 weights while computing layer N. Overlapping memory load with compute hides roughly 30% of memory latency.

This requires modifying ggml or building a wrapper around the inference loop — the most invasive change in the list, and the most novel. If it works, it attacks the 15% latency loss identified in the bottleneck breakdown directly.

Conclusion

Negative results are real results.

Q4_K_M is the bandwidth-compute optimum for the i3-7020U. Going lower in bits saves bytes but costs disproportionate dequantization compute. The crossover has already happened — we were past the optimal point before the experiment started.

The path to 8-10 t/s is not through lower quantization. It is through:

• IQ3_S lookup-table dequant — same bits, faster compute
• Intel HD 620 Vulkan iGPU offload — parallel FFN compute at zero transfer cost
• --no-mmap — contiguous pages, lower TLB pressure
• Software prefetch — hide latency by overlapping load with compute

FINDINGS5 will run --no-mmap first — it is a one-flag change with meaningful expected gain and no implementation risk.

Hardware: Intel Core i3-7020U (Kaby Lake), 2 cores / 4 threads, Intel HD 620 iGPU. Model: LFM2.5-2.6B-Q4_K_M (1,596.9 MB, 30 layers). Tools: quant_tier.py + reorder.py in Jeff9497/Gattai-LM. Benchmark: llama.cpp llama-cli b10549.