The Head That Tracks Names: IOI Circuit Universality from GPT-2 to Llama

I gave Llama-3.2-1B this sentence:

"When Kamau and Wanjiru went to the market, Wanjiru gave mangoes to ___"

The model had to predict Kamau — the indirect object. Not Wanjiru, who appears twice and is the repeated subject. Kamau, who is mentioned first, then disappears from the sentence structure.

The same attention head fired as it does for Mary and John. Layer 11, Head 31. Every time, across 8 experiments, regardless of whether the names were Anglo-Saxon, Kikuyu, Swahili, same-gender, or mixed cultural pairs.

This is what Session 6 of CircuitLens found — and it's the cleanest result across all six sessions so far.

Background: The IOI Task

The Indirect Object Identification task is the canonical mechanistic interpretability benchmark, introduced by Nanda et al. (2022) on GPT-2-small. The prompt structure is:

"When Mary and John went to the store, John gave a drink to ___"
                                        ↑
                              S appears twice → suppress
↑
IO appears once → predict this

The model must track which name is the indirect object (IO) — the entity that was mentioned first but not repeated — and predict it at the end. It's a test of structural role-tracking, not factual recall.

The original paper found a 5-component circuit in GPT-2-small:

• Duplicate Token Heads (~L0–L3) — detect that a name appears twice
• Induction Heads (~L5–L6) — pick up the repetition pattern
• S-Inhibition Heads (~L7–L8) — suppress the repeated subject name
• Name Mover Heads (~L9–L10) — copy the IO name to the output position
• Backup Name Movers — redundancy fallback

GPT-2-small has 12 layers. The name movers sit at 75–83% of model depth. The question entering this session: does Llama-3.2-1B implement the same circuit? And — the extension I cared about most — does it work for names outside the Anglo-Saxon distribution that dominates English pretraining data?

Setup

Same CircuitLens activation patching pipeline as the France→Paris experiments:

• Clean prompt — correct name order (IO named first)
• Corrupted prompt — names swapped (S named first, disrupting IO tracking)
• Target token — the indirect object name
• Output — patching grid across all 16×32 heads; the key head is the cell that most restores the target token when clean activations replace corrupted

clean     = "When Mary and John went to the store, John gave a drink to"
corrupted = "When John and Mary went to the store, John gave a drink to"
target    = model.to_single_token("Mary")
 
clean_tokens     = model.to_tokens(clean)
corrupted_tokens = model.to_tokens(corrupted)
 
_, clean_cache = model.run_with_cache(clean_tokens)
 
results = torch.zeros(model.cfg.n_layers, model.cfg.n_heads)
 
for layer in range(model.cfg.n_layers):
    for head in range(model.cfg.n_heads):
        hook_name = f"blocks.{layer}.attn.hook_z"
 
        def patch_hook(z, hook, l=layer, h=head):
            z[:, :, h, :] = clean_cache[hook_name][:, :, h, :]
            return z
 
        patched_logits = model.run_with_hooks(
            corrupted_tokens,
            fwd_hooks=[(hook_name, patch_hook)]
        )
        delta = (
            patched_logits[0, -1].softmax(-1)[target]
            - corrupted_logits[0, -1].softmax(-1)[target]
        )
        results[layer, head] = delta
 
best_layer = results.max(dim=1).values.argmax().item()
best_head  = results[best_layer].argmax().item()
print(f"Key Head: L{best_layer}·H{best_head}")
# Key Head: L11·H31

The Baseline: L11·H31

The canonical Mary/John prompt from the original paper:

Clean:     "When Mary and John went to the store, John gave a drink to"
Corrupted: "When John and Mary went to the store, John gave a drink to"
Target:    "Mary"
Key Head:  L11·H31

Layer 11, Head 31 is Llama-3.2-1B's IOI name mover head. One cell in the 16×32 patching grid lights up, sitting in the bottom-right quadrant — late layer, high head index.

Before doing anything else, I want to note the circuit universality observation:

Two completely different model families. Different tokenizers. Different positional encodings. Different training procedures. Different parameter counts. The name mover head sits at roughly the same relative depth in both — around 70–80% through the forward pass.

This is consistent with the depth-scaling hypothesis: attention circuits that perform the same computational role tend to emerge at the same relative position in the forward pass, regardless of model family. The IOI circuit is the first place I've seen it hold cleanly across GPT-2 and Llama.

Name-Agnostic: Order Flip and New Names

First check: does the head track structural position or memorized name associations?

Test 2A — Flipped assignment:
Clean:     "When John and Mary went to the store, Mary gave a drink to"
Target:    "John"     Key Head: L11·H31

Test 2B — Different names entirely:
Clean:     "When Alice and Bob went to the office, Bob sent the memo to"
Target:    "Alice"    Key Head: L11·H31 (H30 visible as backup)

Same head. When Alice and Bob are used, L11·H30 shows as a secondary lit cell alongside H31 — this is the backup name mover component from the original paper, present in Llama as an adjacent head. The model has a redundancy mechanism.

The circuit is tracking structural position (first-mentioned, non-repeated entity) — not which specific name is being processed. Swap names entirely and the head doesn't change.

No Gender Dependency

Both names female — gender can't disambiguate:

Clean:     "When Sarah and Emma went to the cafe, Emma passed the note to"
Corrupted: "When Emma and Sarah went to the cafe, Emma passed the note to"
Target:    "Sarah"
Key Head:  L11·H31

The IOI circuit does not use gender features. It fires identically for same-gender pairs. The disambiguation is purely syntactic — which slot in the sentence structure the entity occupies — not semantic.

This rules out the explanation that the circuit is leveraging gendered pronoun statistics from training data. The structural mechanism is operating independently of any gender or name semantics.

Kenyan Names: The Cultural Generalization

This is the test I came into the session most interested in.

Anglo-Saxon names dominate English pretraining corpora. The concern was that the IOI circuit might be brittle to names with low training frequency — that it might fail on names underrepresented in English text.

Kikuyu pair:

Clean:     "When Kamau and Wanjiru went to the market, Wanjiru gave mangoes to"
Corrupted: "When Wanjiru and Kamau went to the market, Wanjiru gave mangoes to"
Target:    "Kamau"
Key Head:  L11·H31

Swahili pair:

Clean:     "When Amani and Baraka went to the office, Baraka sent the report to"
Corrupted: "When Baraka and Amani went to the office, Baraka sent the report to"
Target:    "Amani"
Key Head:  L11·H31

Mixed (Kenyan + Western):

Clean:     "When Kamau and John went to the store, John gave a drink to"
Corrupted: "When John and Kamau went to the store, John gave a drink to"
Target:    "Kamau"
Key Head:  L11·H31

L11·H31 fires identically for Kamau, Wanjiru, Amani, and Baraka.

The circuit doesn't require name familiarity from training data. It's tracking sentence structure — which entity occupies the first-mention, non-repeated slot — regardless of whether those entities are common in English text or not.

The circuit is culturally agnostic because it is syntactically driven.

This has a meaningful implication for where to look for cultural bias in language models. If you're observing that Llama performs differently on Kenyan vs Western names in some task, the IOI attention circuit is probably not the culprit. The attention mechanism for structural role-tracking generalizes cleanly. The more likely source of cultural bias is the MLP layers — parametric knowledge encoded during pretraining, where frequency biases from training data would actually live.

Interpretability gives you a scalpel. This finding tells you where not to cut.

Three-Name Extension

The Nanda et al. paper only tested two-name sentences. This is a novel extension.

Clean:     "When Mary, John and Kate went to the store, John gave a drink to"
Corrupted: "When Kate, John and Mary went to the store, John gave a drink to"
Target:    "Mary"
Key Head:  L11·H31

Three candidates. John is the repeated subject — suppress him. Mary and Kate are both non-repeated. Which one does the circuit select?

Mary — the entity in the first structural position among the non-suppressed candidates.

The circuit doesn't split probability between Mary and Kate. It picks the first-position entity. This tells us something about the S-Inhibition → Name Mover handoff: after the repeated name is suppressed, the remaining selection strategy is positional — first-mention wins.

The original paper couldn't describe this because it only tested two names. With two candidates, first-position and only-remaining-candidate are the same thing. Three names separates these two possible strategies and shows the circuit is using position, not uniqueness.

Circuit Dissociation: Two Tasks, Two Heads

The clearest single result of this session is the separation between structural role-tracking and factual retrieval.

I ran the France→Paris patching experiment from the previous session as a control:

Factual retrieval:  "The capital of France is" vs "The capital of Germany is" → "Paris"
Key Head:           L9·H23

IOI name moving:    "When Mary and John..." → "Mary"
Key Head:           L11·H31

Two different tasks. Two different heads. Two different layers.

L9·H23  — appeared in 0/8 IOI experiments, 1/1 factual experiment
L11·H31 — appeared in 8/8 IOI experiments, 0/1 factual experiment

The model has dedicated computational machinery for each type of inference. Structural role-tracking and parametric fact-retrieval do not share circuitry.

This makes a testable prediction: interventions that suppress factual retrieval — like the adversarial repetition attacks from Session 2 — should have no effect on IOI performance. The circuits are independent. You could theoretically damage France→Paris recall completely while leaving the Mary/John tracking mechanism intact.

Running that cross-circuit intervention test is on the roadmap for Session 7.

A Note on the 0.0% Display Values

Every patching experiment showed +0.0% in the CircuitLens Max Δ Prob header. This is a known limitation worth being transparent about.

Llama-3.2-1B-Instruct is RLHF-trained. When completing "...John gave a drink to", it doesn't want to predict a bare name token — it wants to produce conversational completions like "I think the answer is Mary..." The absolute probability assigned to a single name token like " Mary" is below 0.1%, which rounds to 0.0% in the display.

The head ranking — which cell in the grid wins — is computed before rounding and is the valid signal. L11·H31 genuinely wins the argmax across all 8 experiments. The ranking is consistent and reproducible.

The fix is straightforward: use Llama-3.2-1B (base model, no instruct fine-tuning) where bare token probabilities are higher and the display values are meaningful. I'll run Session 7 on the base model to get clean absolute numbers. The relative finding — L11·H31 as the dominant IOI head — should hold.

Full Results Table

What's Next

Session 7 has two targets:

Switch to base model — run the full IOI suite on Llama-3.2-1B without instruct fine-tuning. Get clean absolute Δ values and confirm L11·H31 holds with displayable numbers.

Cross-circuit intervention — ablate L9·H23 (the factual retrieval head) and check IOI performance, then ablate L11·H31 and check factual retrieval. If the circuits are truly independent, each ablation should only affect its own task.

If that experiment confirms independence, we have a clean anatomical map of two distinct computational systems inside the same 1.2B parameter model. That's the Session 7 goal.

Experiments run on Llama-3.2-1B-Instruct using TransformerLens 2.11.0 on Colab T4, with CircuitLens frontend for patching grid visualization. Code and frontend: Jeff9497/Circuit6. Reference: Nanda et al., "In-context Learning Heads" (2022).