The EU AI Act from the seat of the person building the system

Whether a system counts as high-risk under the AI Act is a legal classification, made in the end by someone else, not me. What I can contribute: the technical groundwork that makes such a classification checkable in the first place. Without it, any legal assessment is a puzzle with missing pieces.

What I notice across projects: the questions the AI Act raises are almost the same ones I already know from compliance reviews on RAG systems. Traceability, human oversight, documentation. The AI Act turns these into a formal duty instead of a good idea.

What belongs here from an engineering seat

Traceability comes first. A decision or claim from the system has to trace back to a source or a process step. That is the same point as source grounding: a model that invents citations cannot document its own decision, because there was none, only a plausible sentence.

Then human oversight, and not as a line in the terms of use, but as a point in the flow where a human can intervene before a claim goes out the door. For the audit assistant that means: a finding from the AI is a draft tied to a clause, not a finished statement. The auditor reviews it and signs off.

Documentation of operational data belongs here too. For a RAG system this is more concrete than for a trained model: which documents sit in the index, which version, when last updated. A system that cannot say which content is in its index, or from which version, already has an operational problem before the legal question even comes up.

Just as important is behavior on expected failure. What happens when retrieval finds nothing? Does the system say "no answer," or does it fill the gap? That is evaluation and source grounding under a new heading.

And finally, transparency toward the user. Does the user know they are talking to an AI system, not a person? That sounds obvious, but it is not a given in chat interfaces deliberately designed to sound human. A disclosure is a UI detail, not an architecture problem, and it still gets forgotten regularly because it never shows up as a task in any sprint.

The category decides the depth, not the tech stack

Not every LLM project needs the full program. An internal assistant that summarizes meeting notes sits in a different place than a system that pre-screens applications or produces findings for a certification audit. Where a specific system lands is, again, a legal question. What I can contribute technically is a clean enough answer to "what does this system actually do, whose interests does the output affect" that a classification can build on it. A system nobody can explain in three sentences cannot be classified seriously either.

The technical file

Other EU product regulations use the idea of a technical file that proves how a product works and why it is safe. For an AI system, the equivalent is a description that captures: what data goes in, what decision or output comes out, where a human can step in, how the system was tested. I write this description if it does not exist yet, because it is the foundation for onboarding, maintenance and the next architecture review anyway. That it is also useful for a legal review is a side effect of good engineering, not the reason to write it.

Where I stay careful

I am not a lawyer, and I will not claim in a review that a system is "AI Act compliant." What I deliver is the technical foundation: a list of what the system does, with which data, with what oversight, with what traceability. The classification into a risk category, and the duties that follow from it, belong with someone trained for that.

Something I see more often than I would like: a team hopes a good model automatically eases the classification. That is not how it works. The classification depends on the purpose and context of the system, not on which model you picked. A small, cleanly documented system with clear oversight is often in a better position from a compliance standpoint than a large model with neither.

What this means for rollout

The points a review for the AI Act needs are almost identical to the points I check before any rollout anyway: source grounding, evaluation, observability, who holds oversight. Whoever has already done this homework before a legal review walks into that conversation with answers instead of a blank page.

The practical side effect: the same documentation that makes a system provable for the AI Act also makes it maintainable for your own operations. A team that knows which index version is live and who signed off on a finding needs that information not only for an auditor, but for the next incident inside the building.

What the readiness check does with it

The LLM readiness check checks technical maturity, not legal compliance. But the findings overlap almost completely: a gap in source grounding or traceability that shows up in the check is exactly the gap a later legal review will find too.

The service behind this: AI engineering. Related: hallucinations as a compliance problem, evaluation in production.

Anyone starting to build these points into an existing system is not doing it for the law alone. A system nobody can trace is an operating risk regardless of which regulation happens to apply.