Every knowledge-system pitch starts the same way: documents go in, a chat box comes out, and the first three questions get impressive answers. Six months later the same system is quietly ignored, because the fourth hundred questions didn't go as well and nobody can say why.
Retrieval-augmented generation is a technique. It is not a strategy, and it is definitely not a finished system. The distance between the demo and production is where the actual engineering lives.
Why the demo always works
Demos are run on curated documents, by the person who built the system, asking questions they already know the corpus can answer. Production is the opposite on every axis: messy documents, users who phrase things sideways, and questions the corpus half-answers or answers three contradictory ways.
The failure isn't the model. It's everything around it.
The four problems the demo skips
Permissions. Your document store has access controls for a reason. A knowledge system that indexes everything and answers anyone has just flattened them. Matter walls, HR records, deal rooms — the retrieval layer must enforce the same boundaries the source systems do, per user, at query time. Retrofitting this is a rebuild, which is why we design the permission model before the index.
Freshness. The policy changed Tuesday. The system indexed it last month. Every stale answer costs trust, and trust is the whole product. Production systems need a defined sync cadence per source, staleness flags on results, and a stated maximum age for answer-bearing documents.
Evaluation. "It seems good" is not a metric. Before we build, we collect a golden set — 100 to 300 real questions from the people who'll use the system, with reference answers reviewed by the client's own experts. Every change to the system runs against that set. Without one, you cannot tell whether last week's tweak made things better or worse, and neither can your vendor.
Citation discipline. An answer without a source is a liability in any workflow that matters. Production systems cite the specific passage, link to the live document, and — this is the part demos skip — decline to answer when retrieval confidence is low. A system that says "I don't have a grounded answer for that" earns more trust than one that improvises fluently.
What "correct" means operationally
Correct is not a feeling; it's a set of numbers agreed before the build:
- Grounded-answer rate — the share of answers fully supported by the cited passages, judged against the golden set.
- Coverage — the share of real user questions the system answers rather than declines.
- Permission integrity — zero answers drawing on documents the asking user cannot open. This one is pass/fail.
- Freshness — maximum age between a source change and its appearance in answers.
Notice the tension: pushing coverage up tends to push groundedness down. Good systems pick the operating point deliberately — in regulated workflows we'd rather decline more and hallucinate never. That trade-off should be a written decision, not an accident of default settings.
The acceptance criteria we write
A typical knowledge-system contract from us includes criteria shaped like this: grounded-answer rate at or above an agreed threshold on the golden set; coverage measured on live query logs after week four; permission integrity verified by adversarial testing before launch and quarterly after; source-to-answer freshness within the agreed window per source system. Under managed operations, we report against all four monthly — the same Scope → Build → Operate structure we use everywhere else.
When a vendor won't put numbers like these in writing, the demo is the product. When they will, you're buying a system.
If your team is evaluating a knowledge system — or living with one that demos well and answers badly — book a consult. We'll tell you whether the fix is engineering or expectations.