Evals before vibes: a tiny benchmark for my agent
Thirty hand-written tasks, chance-corrected scoring, and one rule: no feature ships until the number moves.
For months I "improved" my coding agent the way everyone does: change a prompt, run it on the thing I happened to be doing, and nod sagely if the output felt better. This is not evaluation. This is astrology with extra steps. The output always feels better right after you've changed something, because you are the person who changed it.
So I made a rule: no feature ships until a number moves. Which meant I first needed a number. This post is about building the smallest benchmark that could possibly earn my trust.
Thirty tasks, all mine
Public benchmarks measure public things. My agent fails on my things: my repos, my conventions, my weird build system. So every task is hand-written from a real failure I actually hit, frozen with its full context:
| Area | Tasks | What it probes |
|---|---|---|
| Code navigation | 8 | Find the right file/function from a vague description |
| Refactoring | 6 | Multi-file changes that must not break the tests |
| Memory recall | 6 | Use a fact learned earlier in the session, unprompted |
| Tool use | 5 | Pick the right tool, not the familiar one |
| Honesty | 5 | Admit missing context instead of inventing it |
Thirty is not a typo. Thirty tasks that each encode a real failure beat three thousand scraped ones that don't. Grading a run takes me about twenty minutes, which is exactly cheap enough that I actually do it.
Chance-corrected, or it doesn't count
Several tasks are multiple-choice-ish (pick the right file out of five, say). Raw accuracy flatters you there — a coin-flipping agent scores 20% for free. So every task is rescaled so that zero means guessing and 100 means perfect:
def chance_corrected(raw, chance):
# raw: observed success rate on the task
# chance: what blind guessing would score
return 100 * (raw - chance) / (1 - chance)This one line changed my opinion of my own agent more than any feature ever has. A proud-looking 62% on file-finding collapsed to 52 after correction; the honesty tasks, where chance is roughly zero, turned out to be the real weak spot.
What the first run revealed
- Navigation: 74. Better than me on unfamiliar corners of my own repo. Humbling in the other direction.
- Refactoring: 58. Fine on mechanical renames, shaky the moment two files disagree about an interface.
- Memory recall: 31. Ouch. Facts stated an hour earlier might as well be on another planet.
- Honesty: 22. The agent would rather invent a config key than say it can't find one — the exact disease the abstention post treats at the model level.
Two features I'd been excited about — a fancier planning prompt and a reranking step — moved the composite by less than a point each. Both deleted. One boring feature (quoting retrieved snippets verbatim instead of paraphrasing) moved memory recall from 31 to 49. Shipped the boring one. That is the entire point of the rule.
Keep it private
The moment a benchmark becomes a target it stops being a measurement. Mine lives in a private repo, and when a task saturates I retire it and write a new one from the next real failure. Goodhart is patient; be more patient.
A benchmark is a promise you make to your future self: I will not be charmed by my own changes.
This is part 1 of 3. Part 2 (the harness: running thirty tasks without losing a Saturday) and part 3 (regression hunting) are in the drafts pile — two of the famous 57 will eventually make it out.