Gattai-LM: I Split an LLM Across My Phone and Laptop

Gattai (合体) is Japanese for fusion or combined. The idea is simple: a transformer model is just a stack of matrix multiplications. Normally one device runs the entire stack. Gattai-LM splits that stack across devices.

Input tokens
     ↓
[Laptop — layers 0 to N]
     ↓  activation tensor sent over Tailscale (~8KB/token)
[Phone — layers N+1 to end]
     ↓
Output tokens

The object passed between devices is the activation tensor — the model's internal state mid-computation. If you've read the CircuitLens posts, this is the residual stream that mechanistic interpretability researchers study, just serialized over TCP and shipped across a Tailscale tunnel.

I ran three experiments across three models on the same hardware. Here is what actually happened.

Hardware

Network: Tailscale encrypted tunnel, ~50ms latency over WiFi.

The tool is llama.cpp's RPC backend (build b10549). One device runs llama-server --rpc, the other runs llama-cli with --rpc pointing at the remote address. llama.cpp handles layer assignment and activation serialization automatically.

Experiment 1 — LFM 1.2B

Model: LiquidAI LFM 1.2B, Q4_0, 695MB, 17 transformer layers.

Finding 1: llama.cpp auto-assigned all 17 layers to the phone.

When the RPC server starts, llama.cpp queries the remote device for available RAM. The phone reported 7,665MB — more than the laptop's 2,874MB. So llama.cpp pushed the entire model to the phone. The phone's ARM CPU is significantly slower than the i3-7020U, resulting in 4× slower generation.

Finding 2: The true 8/8 split was the slowest configuration.

Each token now requires two sequential network round trips — laptop to phone and back. Both devices compute, but never simultaneously. The overhead compounds.

Experiment 2 — Granite 3B

Model: IBM Granite 3B, Q4_K_M, 2.1GB, 40 layers. Required --ctx-size 512 — the default context tried to allocate 6,380MB KV cache, exceeding laptop free RAM.

Network overhead: 44% slower generation. 1.2 t/s lost to Tailscale latency.

This is the more honest result. A model that comfortably fits on the laptop runs slower when split. The network cost is real and measurable.

However — the multi-turn stability test held up. I ran 6 consecutive conversation turns at 1.5–1.8 t/s with no connection drops and under 0.15 t/s standard deviation across turns. Tailscale over WiFi is stable enough for sustained inference sessions.

Experiment 3 — LFM 8B MoE

Model: LiquidAI LFM 8B Mixture of Experts, Q4_K_M, 5.2GB.

The local result is HDD-bound — the model exceeds free RAM, so the OS swaps weights from disk on every token.

The split attempt crashed with an Android OOM kill mid-transfer. Root cause: MoE architecture. Layer names contain _exps suffix. Each expert layer is 400–600MB vs ~200MB for a standard dense layer. llama.cpp allocated 4,908MB to the phone. Android's available RAM was 1,863MB. The OOM killer terminated the RPC server.

The deception: "8B MoE" and "8B dense" are not equivalent for RAM planning. The parameter count is the same. The memory layout is not.

What This Reveals

Pipeline parallelism is sequential for autoregressive generation

This is the central insight and it is not obvious before you measure it.

Token N generation:
  Laptop computes layers 0–8          (sequential)
  ↓ sends activation over network     (sequential)
  Phone computes layers 9–17          (sequential)
  ↓ sends result back                 (sequential)
  Laptop decodes output token         (sequential)
  → repeat for token N+1

Both devices are never computing simultaneously. Each token requires a full round trip. More split points means more round trips means slower generation. There is no parallelism here — only pipelining of sequential work across a network boundary.

Batch inference is different. If you're processing multiple independent sequences, different sequences can be in-flight on each device at the same time. For single-sequence token generation — the common interactive use case — splitting is always a latency cost, never a speedup.

The real value is RAM pooling, not compute speed

Split inference is slower when any single device can hold the model. The only case where splitting wins is when the model is too large for any single device:

Laptop free:   2.8GB   →  ~2.5B dense model maximum
Phone free:    1.8GB   →  ~1.5B dense model maximum
Combined:      4.6GB   →  ~4B dense model feasible

For anything above ~2.5B on this hardware, splitting is the only option to run at all. Slower is better than impossible.

llama.cpp reads total RAM, not available RAM on Android

This one matters practically.

# What llama.cpp sees from the RPC server:
Phone RAM: 7665MB   ← total physical RAM
 
# What's actually usable:
Phone RAM: 1863MB   ← after Android OS + running apps
 
# What llama.cpp allocated:
To phone:  4908MB   ← OOM kill

Android reports total physical RAM to the system API, not available RAM. llama.cpp queries total and allocates accordingly. The result is catastrophic over-allocation and an OOM kill.

Manual control is essential. The safe formula for layer allocation:

layers = floor( (phone_available_MB / model_size_MB) × total_layers × 0.8 )

For Granite 3B on 1,863MB available:

floor( (1863 / 2100) × 40 × 0.8 ) = 28 layers max

I used 8 — well within the safe ceiling, which is why Granite ran stably.

The phone becomes a real compute node

During split inference, the phone's CPU cores spin up and do genuine matrix multiplication. Android power management normally keeps those cores parked when the device is idle. Under RPC load, all cores engage.

The practical consequence: the laptop offloads heat and CPU pressure to a device sitting on the desk doing nothing. This matters less for 1B–3B models but becomes meaningful for the 4B–7B range where the laptop would otherwise be thermal-throttling.

What Makes This Setup Unusual

Most distributed inference research uses GPU clusters. Consumer-hardware + Android + Windows + Tailscale split has essentially no published benchmarks. The specific contributions from these experiments:

• Quantified 44% Tailscale WiFi overhead on a 3B model split
• Documented the Android RAM reporting mismatch with llama.cpp RPC
• Confirmed multi-turn stability over 6 turns on consumer WiFi
• Demonstrated that pipeline parallelism is sequential per token (not concurrent)
• Measured the MoE memory deception at the layer level

The repo is Jeff9497/Gattai-LM. The main script Gattai.py handles the split inference pipeline, targeting LFM 2.6B after validating with 1B.

Hardware: i3-7020U laptop (Windows) + Android phone (ARM64) over Tailscale. Tool: llama.cpp RPC backend build b10549. All experiments run at fixed seed, 40-token output.