Three Build Failures to Run the Newest Small Model on an i3

K2 Horizon released September 3rd, 2026. Two days later I had it running on an i3-7020U at 15.9 t/s — the fastest generation speed in the entire Gattai-LM series. Getting there required three separate build failures and a compiler switch.

This post covers the setup journey, the numbers, and a complete summary of every model tested across all five FINDINGS documents.

What K2 Horizon Is

K2 Horizon is a model family from MBZUAI's Institute of Foundation Models — six sizes: dense 0.9B, 3.7B, 7B, 32B, and sparse MoE 36B-A4B and 375B-A23B. All weights and code under Apache 2.0.

The 0.9B is the compact member, targeting watch and glasses-class hardware. It is a reasoning model — it produces a chain-of-thought block ([Start thinking] ... [End thinking]) before every answer, including simple factual queries.

The architectural novelty is Mixture of Values (MoVA) — described in the next section. It is different from every model previously tested in this series.

Three Blockers to First Token

Blocker 1: No Mainline llama.cpp Support

The MoVA architecture kernel had not merged into mainline llama.cpp as of release date. Standard llama-bin packages fail to load K2 Horizon — they lack the architecture code entirely.

Resolution: clone the MBZUAI-IFM fork on the model branch:

git clone --depth 1 -b model/K2Horizon \
  https://github.com/MBZUAI-IFM/llama.cpp ifm-llama-cpp

No pre-built Windows binaries existed as of 2026-09-05. Build from source required.

Blocker 2: MSVC Unicode Regex Failure

MSVC build completed — 543/543 targets — but inference failed at runtime:

Failed to process regex: '(?:'[sS]|'[tT]|...\p{L}...\p{N}...)'
regex_error(error_escape): The expression contained an invalid
escaped character, or a trailing escape.

Root cause: K2 Horizon's tokenizer uses Unicode property escapes (\p{L}, \p{N}, \p{M}). MSVC's std::regex does not support Unicode property escapes — a compiler limitation, not a model bug. The model loaded and architecture parsed correctly, but vocabulary initialisation failed before any inference ran.

Blocker 3: MSYS2 CLANG64

CLANG64 uses Clang/LLVM, which handles Unicode property escapes correctly.

pacman -S mingw-w64-clang-x86_64-cmake \
          mingw-w64-clang-x86_64-ninja \
          mingw-w64-clang-x86_64-clang
 
cd /d/ifm-llama-cpp
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
ninja -C build

543/543 targets. Tokenizer loaded. Inference ran.

Key implication for Windows users: K2 Horizon currently requires MSYS2 CLANG64 or WSL2. This resolves once the IFM PR merges into mainline llama.cpp and standard llama-bin packages ship. Track: github.com/ggml-org/llama.cpp/discussions/28308

The broader point: build toolchain is a research variable. A new architecture may require a custom fork even when llama.cpp is the target runtime. This friction is temporary but it is real.

BF16 vs Q4_K_M: The 3x Gap

The IFM/K2-Horizon-0.9B-GGUF repository contained only BF16 weights at download time. Quantization performed locally:

.\build\bin\llama-quantize.exe `
  /d/models/K2-Horizon-1B-BF16.gguf `
  /d/models/K2-Horizon-0.9B-Q4_K_M.gguf Q4_K_M

Quantize time: 128,430ms (~2 minutes 8 seconds on i3-7020U). Output: 2,056 MB reduced to 632 MB (4.92 BPW).

Prompt processing: 7.8x faster. Generation: 3.0x faster.

Why the Gap Is So Large

From FINDINGS4, CPU inference has two costs per token:

time_per_token = bytes_read / bandwidth  +  flops / throughput

BF16 is 16 bits per weight. Q4_K_M is 4.92 bits per weight. BF16 pushes 3.25x more bytes through the memory bus per token. On an i3-7020U with roughly 25 GB/s effective bandwidth:

BF16   2.16 GB → theoretical ceiling ~11.6 t/s
Q4_K_M 0.63 GB → theoretical ceiling ~39.6 t/s

Q4_K_M AVX2 dequantization is cheap (~2 ops per element — confirmed in FINDINGS4). The bandwidth saving dominates the dequant cost. Q4_K_M sits at the optimal tradeoff — which is the same conclusion reached for LFM 2.6B. Going lower regresses; BF16 saturates the memory bus.

15.9 t/s against a 39.6 t/s theoretical ceiling puts K2 Horizon at 40% of the bandwidth ceiling — similar efficiency to LFM 2.6B (6.9 t/s against a 15.6 t/s ceiling, also ~44%).

The MoVA Architecture

Standard attention applies a value matrix directly to the hidden state. Mixture of Values (MoVA) inserts a routing step: the hidden state goes through a value router first, then into the value matrix. Conceptually similar to MoE routing applied to the value computation in attention rather than to FFN expert selection.

On a 0.9B model running at CPU speeds, the MoVA router overhead is negligible — inference is bandwidth-bound, not compute-bound. The router adds arithmetic but bandwidth dominates. Whether MoVA overhead is measurable at 3.7B or 7B, where attention is a larger fraction of compute, is not yet tested.

Everything Tested: Cross-Model Comparison

All experiments on the same hardware: i3-7020U, same prompt, ctx 512.

K2 Horizon 0.9B Q4_K_M is the fastest CPU-local result in the series, beating LFM 1.2B (9.31 t/s) by 70% despite the similar parameter count. The difference is file size: 633 MB vs 695 MB, similar bandwidth, but K2 Horizon's architecture produces 3.25x fewer bytes per weight than BF16 after quantization.

Seven Findings Across All Gattai-LM Experiments

1. The network floor is a physical constant. Split inference consistently floors at ~1.5 t/s on Tailscale WiFi regardless of model size or split ratio. Local speed varies; split speed does not. This is Tailscale round-trip latency per token — not a tunable parameter.

2. GPU is a category difference, not an improvement. LFM 2.6B: 4.5 t/s CPU vs 87.8 t/s on a T4. 19.5x from memory bandwidth alone (25 GB/s DDR4 vs 320 GB/s GDDR6). No software optimization at the CPU level approaches this gap.

3. Q4_K_M is the optimal quantization point on i3-7020U. Going lower (Q3_K_M FFN tier-split) regresses 13% — dequant compute exceeds bandwidth savings. Going higher (BF16) saturates the memory bus. Q4_K_M sits at the crossover for this CPU. Confirmed independently for both LFM 2.6B and K2 Horizon 0.9B.

4. Cache-aware reordering works on cold start. 0.47 MB of layer norms fits in 3 MB L3. Reordering delivers +18% generation on cold start. Effect disappears once the OS page cache is warm — RAM speed equalises all file orderings.

5. Speculative decoding is a GPU tool on this hardware. DSpark: +17% on T4 (77% acceptance, idle CUDA cores absorb drafter for free). −32% on i3 (34% acceptance, drafter competes for same 2 cores and memory bus). The determining factor is idle compute, not acceptance rate alone.

6. MoE models are incompatible with phone RPC on this setup. MoE "8B" layers are 400–600 MB each — 2-3x heavier than dense 8B per layer. llama.cpp allocates based on total RAM (7,665 MB reported by Android) not available RAM (1,863 MB usable). Android OOM kills the RPC server mid-transfer.

7. Build toolchain is a research variable. K2 Horizon required MSYS2 CLANG64 due to MSVC's lack of Unicode property escape support. A novel architecture may require a custom fork before inference is possible at all. This is temporary but adds meaningful setup cost to evaluating new model releases.

Next on the Gattai-LM list: --no-mmap to reduce TLB pressure (identified in FINDINGS4 as the lowest-effort remaining path to higher throughput) and the Intel HD 620 Vulkan iGPU offload test.

Code, reorder.py, and quant_tier.py: Jeff9497/Gattai-LM.

Hardware: Intel Core i3-7020U (Kaby Lake), 2 cores / 4 threads, CPU-only. Tools: MBZUAI-IFM fork of llama.cpp (K2 Horizon) + llama.cpp b10549 (LFM series).