A bad answer has to be reconstructable
A user complains about a wrong answer. The chat is gone. The prompt is unversioned. Nobody knows the chunks. Without observability that answer cannot be reconstructed.
What has to be reconstructed
Not the whole model. That one run:
Without that a complaint cannot be handled. In a regulated shop that is an operating risk, not a logging nice-to-have.
What I do not mean
A dashboard of token totals. That is FinOps. Useful. It does not answer: why did sentence three invent that clause?
Langfuse and similar tools help. The product name is not the goal. The goal is the path from question to claim. See source grounding.
A typical finding
A team logs HTTP status and latency. The answer body is missing. Two weeks later an incident cannot be reconstructed. Cost per request is also unknown because tokens are not tied to the request.
MCP tool calls without metadata belong in the same drawer. The MCP article covers logging. Here it matters whether a reviewer can rebuild the run without the chat.
What the check does with it
In the LLM readiness check this is day 3 next to evaluation. If the path is missing, the report says not production-ready, even if the demo shines.
Related: evaluation. Service: AI engineering. Practice: healthcare chatbot.
Related articles
LLM evaluation in production
A prompt or model change makes quality worse. Nobody notices if evaluation is missing. What I measure in regulated LLM projects.
Read articleLLM hallucinations are a compliance problem
When a model invents citations, legal stops the project. LLM traceability needs source grounding, not a better prompt.
Read articleA call
30 minutes. If the use case does not belong in production, I'll say so.