When an AI feature returns a wrong answer, the first instinct is to blame the model. The bug report says "the model hallucinated." The Slack thread says "the model is dumb." The proposed fix is almost always the same: swap to a bigger model, or rewrite the prompt for the fourth time.
Most of the time this is a misdiagnosis. The model is a component in a larger system, and that system has more surface area for failure than the model itself. When you trace a bad answer back to its origin, you usually land somewhere upstream: a retrieval step that returned the wrong documents, a data pipeline that silently dropped a field, a context window packed with irrelevant text, or workflow logic that called the model with the wrong inputs in the first place.
This matters because the fix you reach for depends on the diagnosis you make. If you decide the model is the problem, you spend two weeks evaluating a more expensive model and change nothing about the actual failure. The retrieval step is still returning the wrong documents. You have just made it more expensive to be wrong.
The failure is almost always upstream
Consider a retrieval-augmented generation system answering questions over an internal knowledge base. A user asks about the refund policy for enterprise contracts. The answer comes back confidently wrong, citing the consumer refund policy instead.
There are at least five places this could have gone wrong before the model ever ran:
- The document about enterprise refunds was never ingested, because the pipeline skips PDFs over a certain size.
- The document was ingested but chunked badly, splitting the relevant clause across two chunks so neither one is retrievable as a coherent unit.
- The embedding model scored the consumer policy higher than the enterprise policy for this query, so retrieval never surfaced the right chunk.
- The right chunk was retrieved but ranked sixth, and the context assembly step only passes the top three.
- Everything worked, but the prompt template puts the retrieved context after 4,000 tokens of boilerplate instructions, and the relevant clause landed in the middle of the context where models attend to it least.
In four of these five cases, a better model changes nothing. The model never saw the enterprise refund clause. It answered the question it was actually asked, which was a question containing the wrong source material. The failure was upstream, and no amount of prompt engineering reaches upstream.
A diagnostic ordering
The reason model-blame persists is that people debug in the wrong order. They start with the most visible component, the prompt, because it is the thing they wrote most recently and the thing they can change fastest. Start instead with the cheapest, most measurable failure modes and work toward the model. The model should be the last thing you touch, not the first.
Here is an ordering that holds up in practice.
1. Check retrieval hit-rate before touching the prompt. For a sample of real queries with known correct answers, measure whether the document containing the answer appears in the retrieved set at all. Not whether the final answer was right - whether the source material was even present. If your retrieval hit-rate is 70 percent, then 30 percent of your answers are guaranteed to be wrong or fabricated no matter what model you use, because the model is working from material that does not contain the answer. This single number tells you whether you have a retrieval problem or a generation problem, and it takes an afternoon to compute.
def retrieval_hit_rate(eval_set, retriever, k=5):
hits = 0
for example in eval_set:
retrieved = retriever.search(example["query"], k=k)
retrieved_ids = {doc.id for doc in retrieved}
# example["gold_doc_ids"] are the docs known to contain the answer
if retrieved_ids & set(example["gold_doc_ids"]):
hits += 1
return hits / len(eval_set)If this number is low, stop. Do not touch the prompt. You have found your problem, and it lives in chunking, embedding choice, or ranking.
2. Check the data before you check retrieval. If retrieval is missing documents, verify the documents exist in the index in a usable form. Pull the raw ingested text for a chunk and read it. You will frequently find OCR garbage, tables flattened into unparseable strings, boilerplate headers repeated on every chunk, or content that was truncated during ingestion. Retrieval cannot surface what was never stored correctly. Data quality problems masquerade as retrieval problems, which masquerade as model problems.
3. Check context assembly. Once the right documents are retrieved, look at what actually reaches the model. Log the full, final, rendered prompt for a failing case and read every token of it. Count how much of the context window is boilerplate versus relevant material. Check ordering. Check for truncation - if your assembled context exceeds the window, something got silently cut, and it is often the part you needed. This step catches an enormous class of bugs precisely because almost nobody looks at the real assembled prompt. They look at the template, not the rendered output.
4. Check workflow logic. In multi-step agent systems, a wrong answer often comes from an earlier step calling the model with the wrong inputs. A router misclassified the query and sent it to the wrong tool. A prior step returned an error that got stringified and passed forward as if it were data. The model faithfully processed garbage it was handed by your own orchestration code.
5. Now, and only now, consider the model and the prompt. If retrieval hit-rate is high, the data is clean, the assembled context is correct and well-ordered, and the workflow handed the model the right inputs, and the answer is still wrong - now you have a genuine generation problem. This is where prompt changes and model upgrades are the right tool. In my experience this is the minority of cases, maybe one in four.
A worked example
Take a support-triage assistant that classifies incoming tickets and drafts a first response. The team reports it is "getting worse" and wants to upgrade the model. Here is what a disciplined pass looks like, with illustrative numbers to show the shape of the reasoning. These are examples, not measured client results.
Baseline: 100 tickets, human-labeled with the correct category and a reference response.
- Retrieval hit-rate for the relevant knowledge-base article: 62 percent. Immediately, 38 percent of drafts are working from missing or wrong source material. Root cause found in step one.
- Inspecting the misses: 22 of the 38 involve tickets that reference a product renamed three months ago. The knowledge base uses the new name; customers use the old one. Embedding similarity between "LegacyName crashes on export" and an article titled "NewName export troubleshooting" is low. This is a data and retrieval problem, not a model problem. A synonym map or a query-rewriting step fixes most of it.
- Of the remaining tickets where retrieval succeeded, context assembly logs show the article is appended after a 1,800-token instruction block, and in 9 cases the article was truncated because the ticket thread itself was long. Trim the instruction block, and put retrieved context near the end of the prompt.
- After fixing retrieval synonyms and context assembly, re-run the eval. Hit-rate rises to 89 percent, and answer quality follows, with the same model.
The team came in wanting to spend money on a bigger model. The actual fixes were a synonym map, a query rewrite, and moving text around in a prompt template. Total model cost change: zero. The model was never the problem.
What to instrument now
You cannot debug in this order if you are not logging the right things. The minimum viable instrumentation for any RAG or agent system:
- The exact query sent to retrieval, after any rewriting.
- The IDs and scores of every retrieved document, not just the ones you kept.
- The fully rendered prompt, exactly as the model received it, including token count.
- The raw model output before any post-processing.
- For agent systems, the input and output of every intermediate step.
With those five logs, most "the model is wrong" tickets resolve in an hour, and most of them resolve somewhere other than the model.
The uncomfortable truth is that reaching for a bigger model is often the path of least resistance for the team, not the path to the fix. It is a purchase order instead of an investigation. The investigation is cheaper, and it is the only thing that tells you what is actually broken.
If your AI system is failing in production and you are not sure whether the model, the data, or the plumbing is at fault, that is exactly the diagnosis we do. See how we approach it at /services/ai-reliability.