Most document systems answer plausibly and wrongly. The answer reads well, cites nothing, and is confidently built from a policy that was replaced last year. A user has no way to tell it apart from a correct one, so they act on it.
A wrong answer that sounds right costs more than no answer, because nobody checks it. The work of a retrieval system is mostly about making answers checkable, and most of that work happens before the model sees a question.
Why plausible answers go wrong
When a retrieval answer is wrong, the model is usually not the cause. The model did a good job with what it was given. What it was given was wrong, and it came from 4 kinds of failure:
- Chunks cut through the middle of a table, so a figure arrives without its row and column
- A superseded document ranked above the current one, because it was longer and matched more words
- An answer stitched together from 2 documents that disagree, with the disagreement smoothed over
- A confident reply when the index had nothing relevant, assembled from whatever came closest
Each of these happens upstream of generation. Changing the model will not fix them. A better model writes a more convincing version of the same wrong answer.
Indexing by structure
Documents have structure, and the structure carries meaning. A clause belongs to a section. A figure belongs to a row and a column. A definition at the top of a contract governs every use of the term below it. Splitting a document into fixed-size chunks throws that away.
Indexing by structure keeps it. Documents are split on headings, clauses and table boundaries, and each piece carries its place in the document: which section, which version, which table. A retrieved passage then arrives with enough context to mean what it meant in the original.
A passage cut from its heading is a sentence without a subject. The model will supply one.
Version and scope
Two filters run before ranking, and both are simple.
The version filter keeps superseded documents out of the answer. Each document carries the dates it was in force. A question about current policy only sees current policy. A question about what the policy was in a given year can still find the old version, deliberately.
The scope filter keeps users inside what they are allowed to read. A user who cannot open a document should not receive an answer built from it. The check uses the same permissions as the document store, at query time, so a change in someone's access takes effect on their next question.
Citations that open the source
Each answer points to the passages it came from, and the citation opens the source at the clause. A reader who doubts the answer can check it in seconds. A reader who does not doubt it still sees where it came from, which changes how much they trust it.
Citations also make errors visible to the team. When a user reports a wrong answer, the citation shows which passage misled the model. The fix is usually in the index: a chunk boundary, a missing version date, a document that should have been retired.
A refusal path
The hardest case is the question the index cannot answer. The pressure in a generated system is to produce something. Without a defined behaviour for "not found", the model produces its closest guess, phrased as confidently as a real answer.
So "not found" gets a defined answer. When retrieval returns nothing above a relevance threshold, the system says so plainly and routes the question to a person who can answer it. The person's answer can then become a document, or a case in the evaluation suite, so the next user gets a real reply.
This is the piece most teams leave out, because it feels like admitting the system is incomplete. It is the piece users trust most, because it is the one that tells them the truth when the truth is "we do not know yet".
What to ask of a retrieval system
A short list of questions separates a system that can be trusted from one that only reads well. They work for a system you are building and for one you are buying.
- Does each answer cite the passage it came from, and does the citation open the source at the right place?
- What happens when the index has nothing? Is there a defined answer, and who receives the question?
- How are superseded documents kept out of current answers?
- Does a user ever receive an answer built from a document they cannot open?
- Is there an evaluation suite built from your documents, and does it block a change that makes answers worse?
A vendor who answers the second question with a description of the model has not understood it. The question is about behaviour when the model has nothing true to say.
Measuring it
A retrieval system needs its own evaluation suite, built from the documents it serves and the questions users ask. Each case checks 3 things: that the answer is right, that it cites the passage it should, and that the refusal path fires when it should. The third check is the one generic benchmarks skip.
The suite runs on each change to the index, the chunking, the ranking or the model. A drop in any of the 3 scores blocks the change and names the questions that failed.
Three wrong answers
Take 3 questions your system gets wrong today. For each one, trace the answer back: which passages were retrieved, which were ranked first, and what was missing. In most systems the 3 traces point to 2 or 3 fixes in the index, and none of them involve the model.
Then add citations, so the next wrong answer traces itself. And decide, in writing, what the system says when it has nothing. That one decision usually does more for trust than any change to how the answers are generated.