Lire en français

We deploy agents. We count tokens. We track the model’s accuracy rate. We celebrate generation time.

And we call that measuring AI success.

These indicators tell us what the agent does. They say almost nothing about what the organisation gains or loses.

What we can measure today: actual cycle time on critical processes, before and after. And the gap between deployment rate and real usage. An agent can be live without being used, and the gap is often considerable.

The first missing indicator is what supervision costs in attention: how much time does a senior expert really spend validating, correcting and arbitrating what an agent produces — and is that load falling over time, or rising?

Without that figure, a reported production gain is only half a result.
The second is the residual error rate of the system, not the agent. What matters is not the accuracy of the model in isolation, but the error rate of the final deliverable after human review.

And that distinction has a consequence rarely measured: a better-performing agent does not necessarily improve the system. The more plausible an output looks, the less it gets scrutinised.

With one qualification that matters, and that works in our favour. A marketing authorisation dossier is not produced in one block: it is assembled section by section, then module by module, each verified at its own level, before a consistency review at dossier level. Where that layered architecture survives, a lapse in vigilance on one link remains recoverable. Where AI produces a complete deliverable reviewed once, it no longer is.

In other words, what looks like process heaviness may be our best protection against our own biases. It would be the worst place to go looking for efficiency.

The two missing indicators — supervision load and system error rate — are far harder to obtain than a model’s accuracy. That is probably why accuracy is preferred.

And what we measure always ends up defining what we optimise.

Yet, in our field, there are indicators nobody needs to invent: the number of questions from authorities, rework after submission, deviations linked to inconsistent documents. They are already tracked, they are audited, and they will tell the truth about our deployments before any dashboard does.

Which indicators do you use, or wish you could use?