RAG is a filing cabinet, not a brain
Retrieval gives your model a library card, not a memory. On the difference between looking things up and knowing them.
Somewhere along the way, "add RAG" became the reflexive answer to every model limitation, the way "add blockchain" once was. Model doesn't know your codebase? RAG. Forgets what you told it? RAG. Hallucinates? Believe it or not, RAG. And retrieval genuinely is wonderful — but it is wonderful the way a filing cabinet is wonderful. Nobody confuses a filing cabinet with an employee who knows things.
I build memory systems for coding agents for a living, so this distinction is not pedantry to me; it's the difference between features that work and features that demo well. Here is the cleanest version of it I can manage.
Lookup versus knowing
When you look something up, the information arrives at the moment of use, shaped like the document it came from, and leaves when the context ends. When you know something, it has been digested: connected to other things, compressed into a judgment, available even when no one asks the question in the document's words. Retrieval does the first. Only something that changes state between sessions does the second.
| Situation | Filing cabinet (RAG) | Memory |
|---|---|---|
| "What does section 4.2 say?" | perfect | overkill |
| "Which approach did we abandon, and why?" | struggles | the whole point |
| Synthesis across twenty documents | top-k misses the shape | digested once, reused |
| Noticing what's absent | can't retrieve a gap | knows what should exist |
| Fresh facts that change daily | exactly right | goes stale |
Where the cabinet fails
Three failures show up constantly, and all three are structural, not fixable with a better embedding model:
- Synthesis. The answer isn't in any chunk; it's the shape of thirty chunks together. Top-k retrieval hands the model five tiles of a mosaic and asks it to describe the picture.
- Negative knowledge. "We tried this in March and it broke" is the most valuable sentence in any project — and it's precisely the thing nobody wrote a document about. You cannot retrieve what was never filed.
- The question's vocabulary. Retrieval finds documents that sound like the question. Hard questions rarely sound like their answers.
A library card is not an education. It's a very good thing to give someone who already has one.
The practical heuristic
When I'm deciding which side of the line a feature belongs on, I ask one question: should the system behave differently next week because of what happened this week? If no — the knowledge is external, stable, and bigger than the system — use the filing cabinet, and use it proudly. If yes — the knowledge is about the work itself, accumulating, often never written down anywhere retrievable — you need memory: something that writes, revises and occasionally deletes its own state.
The strongest setups I've built use both, with a narrow pipe between them: retrieval for the world, memory for the relationship, and a nightly pass that promotes hard-won retrieval conclusions into remembered judgments. The cabinet stores the documents. The brain remembers which drawer always lies.
How I check whether the memory half actually works, instead of just feeling clever? That's what the benchmark post is for — memory recall scored 31 out of 100 the first time I measured it, which is exactly why I stopped trusting vibes.