Research notes on mechanistic interpretability, LLM experiments, and building AI agent systems from scratch. Based in Nairobi.
Kroniqo's memory system uses exponential decay with λ=0.03. Why that number? What happens at 0.01 or 0.1? This is the reasoning behind a design decision that most agent frameworks skip entirely.
K2 Horizon 0.9B dropped September 3rd with a novel MoVA attention architecture. Getting it running on Windows took three separate build failures. The result — 15.9 t/s on an i3-7020U — makes it the fastest model tested in the Gattai-LM series.
Tier-splitting a 2.6B model's FFN weights to Q3_K_M shrank the file 20% but made generation 13% slower. The reason reveals something important about how CPU inference actually works — and why Q4_K_M is already near-optimal for this hardware.
Reordering a 2.6B model's weights by access frequency — putting all layer norms together before attention and FFN blocks — delivers 18% faster token generation on cold start. The mechanism fits in 3 lines of cache theory.
What happens when you split transformer layers across a Windows laptop and an Android phone over Tailscale? Actual benchmarks, a crash, and some findings that change how I think about distributed inference on consumer hardware.
The canonical GPT-2 IOI circuit — the mechanism that tracks indirect objects across names — exists in Llama-3.2-1B at the same relative depth. One head, L11·H31, fires universally across 8 experiments. It works for Kamau and Wanjiru as well as Mary and John.
A single jailbreak instruction flips France→Paris to Berlin at 83.3%. Attribution analysis reveals the model isn't updating its beliefs — it's copying the token from the instruction. And in one run, it explicitly said so.
OPT-1.3b stores the France→Paris fact in a single attention head — Layer 21, Head 0. Llama 3.2-1B distributes the same knowledge redundantly across dozens. The contrast reveals something fundamental about how different architectures store facts.
How I QLoRA-tuned LFM2.5-1.2B for skill trigger classification, hit near-perfect accuracy, then realized I had solved the wrong problem entirely.