A bad answer has to be reconstructable
A user complains about a wrong answer. The chat is gone. The prompt is unversioned. Nobody knows the chunks. Without observability that answer cannot be reconstructed.
What has to be reconstructed
Not the whole model. That one run:
Without that a complaint cannot be handled. In a regulated shop that is an operating risk, not a logging nice-to-have.
What I do not mean
A dashboard of token totals. That is FinOps. Useful. It does not answer: why did sentence three invent that clause?
Langfuse and similar tools help. The product name is not the goal. The goal is the path from question to claim. See source grounding.
A typical finding
A team logs HTTP status and latency. The answer body is missing. Two weeks later an incident cannot be reconstructed. Cost per request is also unknown because tokens are not tied to the request.
MCP tool calls without metadata belong in the same drawer. The MCP article covers logging. Here it matters whether a reviewer can rebuild the run without the chat.
What the check does with it
In the LLM readiness check this is day 3 next to evaluation. If the path is missing, the report says not production-ready, even if the demo shines.
Related: evaluation. Service: AI engineering. Practice: healthcare chatbot.
Related articles
The EU AI Act from the seat of the person building the system
Traceability, human oversight, documentation: what an engineer can contribute to the AI Act, and where the legal classification begins.
Read articleLLM operations: from demo to a run someone owns
A prototype needs an API key. Operations need versioned prompts, a pipeline, and someone who owns the run when it breaks.
Read articleA call
30 minutes. If the use case does not belong in production, I'll say so.