What to Log So You Can Debug an AI System Later

A customer forwards a screenshot of an answer your assistant gave three weeks ago, citing a policy that was withdrawn in March. You open the logs. You find the request id, the timestamp, the latency, and the answer text. You do not find the chunks that were retrieved, the model version serving traffic that afternoon, or the system prompt as it existed before Tuesday's deploy. Without the inputs and versions, it is hard to reproduce the failure. The team can patch the symptom while missing the cause.

That is the whole problem, and it is not a volume problem. In deterministic software, the code plus the input reproduces the bug, and the code is in version control. In an AI system, the input to the model is assembled at runtime out of a retriever, a history summarizer, a tool registry, and a prompt template that each move independently. If that context is missing from the trace, you have the output without enough of the input to explain it.

The reproducible unit is the assembled context, not the user's message

The user's message is the smallest and least interesting part of what the model saw. Everything that decided the answer sat around it: the system prompt, the tool schemas that happened to be in scope, the summarized history, and the retrieved passages. Log the user's turn alone and you have logged the one input you could already get from the support ticket.

So the record is per model call, not per request, and it captures the call as it was actually made. These fourteen fields are a useful starting point:

• request_id and call_id - one request, one row per model call inside it, so agent loops are readable in order.
• timestamp and latency_ms.
• model and model_version - the exact served version, not the alias you asked for. Aliases move under you.
• prompt_template_version - a git sha or a semver, not a name.
• retrieved_ids - the chunk or document ids and their scores. Ids, not the text; the text is in your store and it is expensive to duplicate.
• context_hash and context_bytes - a hash of the fully assembled prompt, plus its size. The hash is what lets you group thousands of failures into three causes.
• input_tokens, output_tokens, cost_usd.
• outcome - one of ok, refusal, tool_error, timeout, guardrail_block.

Add raw_prompt on the sampled subset only. The hash tells you which calls are identical; the raw text is what you read when you need one of them.

Write it at the call site, in one place

The failure mode of observability projects is that the fields get scattered: token counts in the gateway, retrieval scores in the RAG service, model version nowhere. Wrapping the provider client gives every model call a consistent trace record, with checks for missing fields.

{
  "request_id": "req_8f21",
  "call_id": 2,
  "timestamp": "2026-07-14T09:41:02Z",
  "model": "provider-model-a",
  "model_version": "2026-07-11",
  "prompt_template_version": "a41c9e2",
  "retrieved_ids": [
    { "id": "doc_512#c3", "score": 0.81 },
    { "id": "doc_098#c1", "score": 0.77 }
  ],
  "context_hash": "sha256:7c1e...",
  "context_bytes": 18432,
  "input_tokens": 4820,
  "output_tokens": 311,
  "cost_usd": 0.0193,
  "latency_ms": 2140,
  "outcome": "ok"
}

Surface the request_id in the interface, next to the answer, the way a support tool shows a case number. It gives a support report a concrete record to investigate.

Keep enough detail to investigate failures

The objection is always storage, so size it. At 50,000 calls a day and about 1.4 KB a record with the raw prompt truncated out, that is 70 MB a day and roughly 2.1 GB a month. That is arithmetic you should redo with your own volume and record size, but the shape holds: the structured record is small, and the thing that is genuinely large is raw prompt text.

So split the policy. Where your data policy allows, keep records for errors, user reports, and human corrections. Sample routine successes at a rate that fits your traffic and monitoring needs. Redact sensitive data and apply a shorter retention period to any stored prompt text.

The human correction is a log field, not a support ticket

One of the most useful signals is a human correction, but it often lives outside the system trace: a Zendesk macro, a spreadsheet, an escalation channel. Write it back against the request_id as a field on the trace - what was wrong, what the right answer was, who said so.

That single join is what turns observability into a feedback loop. Corrections attached to traces become eval cases with their exact failing context already attached. Corrections in a ticket queue become nothing.