I Fine-Tuned a 1.2B Model to 98.7% Accuracy, Then Scrapped It

The model worked. That was the problem.

I spent three days generating training data, running QLoRA on a Colab T4, watching the loss curve drop cleanly, and hitting 98.7% eval accuracy. Then I sat with it for an hour and realized I had solved the wrong problem entirely.

This is that story.

Background: Kroniqo and the Routing Problem

Kroniqo is an AI assistant I've been building — multi-backend LLM routing across Groq, Mistral, Gemini, Cerebras, Claude, SambaNova, and a few others. One of the harder design problems is skill routing: when a user sends a message, which capability should activate?

The skills available at any point include:

• code_exec — user wants to run or debug code
• web_search — user wants current information
• memory_recall — user wants context from past sessions
• default — just respond

Early versions used keyword matching. import in the message → probably code. today or latest → probably search. Brittle. Kept misfiring on edge cases. The obvious next step felt like a classifier.

The Fine-Tuning Plan

My reasoning: train a small, fast model to route. Something light enough to run on a free tier, fast enough to add no perceptible latency.

LFM2.5-1.2B from LiquidAI fit the profile. Small, recent architecture, instruction-tuned, fast inference on Groq.

QLoRA because I'm not paying for GPU credits. A T4 on Colab free can barely hold a 1.2B model at full precision. With 4-bit quantization and LoRA adapters, it sits comfortably.

Dataset Construction

I generated ~600 labeled examples using Claude and manual labeling, covering each skill class with realistic message variation:

examples = [
    {"input": "run this and tell me what it outputs",    "label": "code_exec"},
    {"input": "what's the latest paper on SSMs",         "label": "web_search"},
    {"input": "what did we discuss about Kroniqo yesterday", "label": "memory_recall"},
    {"input": "explain the attention mechanism",         "label": "default"},
    {"input": "fix this KeyError",                       "label": "code_exec"},
    {"input": "is Mistral still free on OpenRouter",     "label": "web_search"},
]

Split: 80/10/10 train/val/test.

Training Config

from peft import LoraConfig, TaskType, get_peft_model
from transformers import TrainingArguments, Trainer
 
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.SEQ_CLS,
)
 
training_args = TrainingArguments(
    output_dir="./lfm25-skill-router",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    warmup_ratio=0.1,
    fp16=True,
    evaluation_strategy="epoch",
    logging_steps=20,
    save_strategy="epoch",
    load_best_model_at_end=True,
)

Training ran for roughly 40 minutes on a T4. Loss curve was smooth — no spikes, no overfitting signal.

Eval accuracy: 98.7%.

On the held-out test set. For a 1.2B model on a 4-class problem, that's essentially solved.

Where It Went Wrong

I pushed the adapters, wired Kroniqo to call the classifier before each routing decision, and started testing.

Latency was acceptable — classification added ~90ms before the main LLM call. Outputs were correct. But something felt wrong. Not the numbers. Something structural.

The model was solving a proxy task.

The actual routing problem isn't "what capability does this message invoke?" — it's "should I call a tool at all, which one, and with what arguments?"

Those are fundamentally different:

A user saying "run this" is easy to classify as code_exec. But which code? The one from 3 messages ago? A new snippet they're about to paste? Should it even run, or ask for clarification first? The classifier answers none of that.

Meanwhile — LFM2.5-1.2B-Instruct natively supports function calling. I had never tested that.

What I Should Have Done

Zero-shot tool calling test, before generating a single training example:

tools = [
    {
        "type": "function",
        "function": {
            "name": "execute_code",
            "description": "Execute Python code in a sandboxed environment",
            "parameters": {
                "type": "object",
                "properties": {
                    "code": {"type": "string", "description": "The Python code to execute"},
                    "language": {"type": "string", "enum": ["python"]}
                },
                "required": ["code"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "web_search",
            "description": "Search the web for current information",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string"}
                },
                "required": ["query"]
            }
        }
    }
]
 
# Just pass the tools — let the model decide
response = client.chat.completions.create(
    model="liquid/lfm-2.5-1.2b-instruct",
    messages=[{"role": "user", "content": user_message}],
    tools=tools,
    tool_choice="auto"
)

If native tool calling is reliable, the classifier is entirely redundant. The model already knows which tool to call based on its instruction tuning — no additional training required.

Test zero-shot before collecting data. Always.

What That 98.7% Actually Measured

High eval accuracy on a classification benchmark is easy to get on synthetic data. What I actually validated was:

1. Whether the model learned my label conventions
2. Whether my synthetic data distribution matched real user behavior
3. Whether a 4-class intent taxonomy was even the right framing

None of those are "does this make Kroniqo route correctly in production?"

The gap between benchmark accuracy and real-world utility is where most fine-tuning efforts die. The benchmark is easy to optimize. The real task is harder to even define.

The Actual Lessons

Validate the task formulation before collecting data. I should have spent one hour stress-testing the routing problem before spending three days on the solution. The question isn't "can I train a classifier?" — it's "is classification the right primitive here?"

Check what the base model already does. Instruction-tuned models often handle structured tasks zero-shot better than expected. Fine-tuning earns its cost when there's a clear, measured gap between zero-shot and your performance target. Not before.

Small models can surprise you. LFM2.5-1.2B-Instruct's zero-shot tool calling is more capable than I assumed. I'm testing it properly now.

Fine-tuning is expensive cognition, not just compute. Data collection, labeling, evaluation, iteration — the real cost isn't GPU hours. It's attention. That cost needs clear justification.

The adapters are archived. The Colab notebook is saved. And Kroniqo now routes via native tool calling, zero extra model, zero extra latency.

If the zero-shot results hold across enough production traffic, this post becomes the story of the fastest I've ever over-engineered a problem.

Part of an ongoing series on building Kroniqo — an AI assistant with multi-backend routing, persistent memory, and a sub-agent system.