Research notes on mechanistic interpretability, LLM experiments, and building AI agent systems from scratch. Based in Nairobi.
The canonical GPT-2 IOI circuit — the mechanism that tracks indirect objects across names — exists in Llama-3.2-1B at the same relative depth. One head, L11·H31, fires universally across 8 experiments. It works for Kamau and Wanjiru as well as Mary and John.
A single jailbreak instruction flips France→Paris to Berlin at 83.3%. Attribution analysis reveals the model isn't updating its beliefs — it's copying the token from the instruction. And in one run, it explicitly said so.
OPT-1.3b stores the France→Paris fact in a single attention head — Layer 21, Head 0. Llama 3.2-1B distributes the same knowledge redundantly across dozens. The contrast reveals something fundamental about how different architectures store facts.
How I QLoRA-tuned LFM2.5-1.2B for skill trigger classification, hit near-perfect accuracy, then realized I had solved the wrong problem entirely.