What to Log So You Can Debug an AI System Later

A customer forwards a screenshot of an answer your assistant gave three weeks ago, citing a policy that was withdrawn in March. You open the logs. You find the request id, the timestamp, the latency, and the answer text. You do not find the chunks that were retrieved, the model version serving traffic that afternoon, or the system prompt as it existed before Tuesday's deploy. The failure is not reproducible, so it is not fixable. You add a guardrail against the symptom and move on, and the same class of bug returns in a different shape next quarter.

That is the whole problem, and it is not a volume problem. In deterministic software, the code plus the input reproduces the bug, and the code is in version control. In an AI system, the input to the model is assembled at runtime out of a retriever, a history summarizer, a tool registry, and a prompt template that each move independently. None of that is written down anywhere. You logged the output of a function whose input you never recorded.

The reproducible unit is the assembled context, not the user's message

The user's message is the smallest and least interesting part of what the model saw. Everything that decided the answer sat around it: the system prompt, the tool schemas that happened to be in scope, the summarized history, and the retrieved passages. Log the user's turn alone and you have logged the one input you could already get from the support ticket.

So the record is per model call, not per request, and it captures the call as it was actually made. Fourteen fields is the minimum that gets you there:

• request_id and call_id - one request, one row per model call inside it, so agent loops are readable in order.
• timestamp and latency_ms.
• model and model_version - the exact served version, not the alias you asked for. Aliases move under you.
• prompt_template_version - a git sha or a semver, not a name.
• retrieved_ids - the chunk or document ids and their scores. Ids, not the text; the text is in your store and it is expensive to duplicate.
• context_hash and context_bytes - a hash of the fully assembled prompt, plus its size. The hash is what lets you group thousands of failures into three causes.
• input_tokens, output_tokens, cost_usd.
• outcome - one of ok, refusal, tool_error, timeout, guardrail_block.

Add raw_prompt on the sampled subset only. The hash tells you which calls are identical; the raw text is what you read when you need one of them.

Write it at the call site, in one place

The failure mode of observability projects is that the fields get scattered: token counts in the gateway, retrieval scores in the RAG service, model version nowhere. Wrap the provider client once, and have every model call in the system pass through it. That wrapper is the only code that writes a trace record, and it writes all fourteen fields or none.

{
  "request_id": "req_8f21",
  "call_id": 2,
  "timestamp": "2026-07-14T09:41:02Z",
  "model": "provider-model-a",
  "model_version": "2026-07-11",
  "prompt_template_version": "a41c9e2",
  "retrieved_ids": [
    { "id": "doc_512#c3", "score": 0.81 },
    { "id": "doc_098#c1", "score": 0.77 }
  ],
  "context_hash": "sha256:7c1e...",
  "context_bytes": 18432,
  "input_tokens": 4820,
  "output_tokens": 311,
  "cost_usd": 0.0193,
  "latency_ms": 2140,
  "outcome": "ok"
}

Surface the request_id in the interface, next to the answer, the way a support tool shows a case number. It costs nothing and it converts "the bot said something wrong last month" into a row you can pull.

Sample the successes, never the failures

The objection is always storage, so size it. At 50,000 calls a day and about 1.4 KB a record with the raw prompt truncated out, that is 70 MB a day and roughly 2.1 GB a month. That is arithmetic you should redo with your own volume and record size, but the shape holds: the structured record is small, and the thing that is genuinely large is raw prompt text.

So split the policy. Keep 100 percent of records where outcome is anything but ok, of anything a user thumbed down, and of anything a human corrected. Sample clean successes at 10 percent, which is enough to see distribution shifts in context_bytes and retrieval scores. Store the raw prompt only for the kept set, with a shorter retention than the record itself.

The human correction is a log field, not a support ticket

The most valuable line in an AI system's logs is the one where a person disagreed with it, and it is almost always the one that lives somewhere else: a Zendesk macro, a spreadsheet, an escalation channel. Write it back against the request_id as a field on the trace - what was wrong, what the right answer was, who said so.

That single join is what turns observability into a feedback loop. Corrections attached to traces become eval cases with their exact failing context already attached. Corrections in a ticket queue become nothing.