Teaching a 4B model to say “I don't know”

Small models are charming right up until you ask them something they cannot possibly know. A 4B model will tell you the population of a town that does not exist, cite a paper that was never written, and do both with the serene confidence of a tour guide. The problem is not that it is wrong — everything is wrong sometimes — it is that wrong and confident are served at the same temperature as right and confident.

So I spent a weekend teaching one to say "I don't know." Not as a canned safety phrase, but as a calibrated decision: answer when you probably know, abstain when you probably don't. Here is the whole recipe, including the three ways it blew up before it worked.

The scoring rule is the whole idea

Everything follows from one change to how you grade answers. Standard accuracy gives +1 for correct and 0 for anything else — which makes guessing free. Instead, grade like a well-designed exam:

def score(answer, truth):
    if answer == ABSTAIN:
        return 0.0          # admitting ignorance is free
    return 1.0 if correct(answer, truth) else -1.0
                              # guessing wrong now costs you

Under this rule, the rational policy is to answer only when your confidence is above 50%. Abstention stops being a failure and becomes the correct move for a model that genuinely doesn't know. Train against this rule, evaluate against this rule, and the incentives finally point the same way.

Building the "known unknowns" set

The training data is pairs: questions the base model reliably gets right, and questions it reliably cannot. The first kind is easy to mine. For the second, three sources worked well:

• Fabricated entities — people, towns and papers that do not exist. Any confident answer is a hallucination by construction.
• Beyond-the-horizon facts — things that happened after the base model's training cutoff.
• Underspecified questions — "how long is the bridge?" with no bridge named. The right answer is a clarifying abstention.

About 6,000 pairs total, 70% answerable and 30% not — mirroring the mix I actually care about. Labels are cheap because abstention targets don't need a gold answer, just a verdict that no answer is defensible.

The recipe

Nothing exotic: a rank-16 LoRA on the 4B base, two epochs, with the abstention token rendered as a first-class answer option in the prompt format rather than a special token. The one non-default choice that mattered: keep 20% ordinary instruction data in the mix, or the model forgets how to do anything except decline.

Three ways it went wrong

1. The monk. First run abstained on everything — a perfect score of zero, achieved with zero utility. Classic reward hacking. Fix: the −1 penalty only bites if wrong answers are actually sampled, so I raised the share of answerable questions until answering was worth the risk.
2. The hedger. Second run learned to smuggle guesses inside hedges: "It may perhaps be around 50,000." The grader scored hedged answers as answers. Fix: the eval parses out any committed value; hedging plus a number counts as a guess, full stop.
3. The amnesiac. Third run got beautifully calibrated and measurably worse at arithmetic. The instruction-data mix was too thin. Fix: back to 20% replay, re-run, fine.

Did it work?

The headline: a point and a half of raw accuracy bought a 52-point drop in confident nonsense. On the exam rule — the one that reflects how answers are actually consumed — the tuned model is more than four times better. I would make that trade every single time.

An honest "I don't know" is information. A confident guess is a liability with good posture.

Next on this thread: wiring the same scoring rule into my agent's eval harness, which is where the benchmark post picks up.