How I Evaluate AI Systems

The benchmark is not the system

When I had to choose an extraction provider for a production document pipeline, the headline accuracy number turned out to be the least useful figure I had. A model can score well on a public benchmark and still be the wrong choice, because a benchmark measures the model in isolation and a production system never runs it in isolation. What matters is whether the output survives contact with everything downstream: the schema it has to fit, the budget it runs against, and the manual cleanup a wrong answer forces a human to do later.

So I do not evaluate a model. I evaluate the model in the seam where it sits.

What I actually compare

Three things, in this order.

Structure. Does the output fit a strict schema every time? In the pipeline, every extraction has to land in a typed Pydantic schema. A provider that returns fluent, correct-looking prose but will not reliably produce the schema has failed the only test that matters for an automated system. The next step cannot consume it.

Accuracy per cleanup, not headline accuracy. The real cost of a wrong extraction is the human time it takes to find and fix it downstream. A cheaper provider that needs an afternoon of data-cleaning is not cheaper.

When I compared a dedicated OCR service, a general-purpose LLM (gpt-4o) and a document-native model, the decision came down to exactly that. The OCR service returned raw labelled boxes that fit no schema without a custom normalization layer; the general LLM produced clean JSON but fumbled low-quality scans; the document-native model was accurate enough to need no formatting cleanup at all. Price per page was the tie-breaker, not the deciding factor.

Failure behaviour. Does it fail loudly or quietly? A model that throws an error is safe: the system stops and falls back. The dangerous one returns a confident, well-formed and wrong answer, because nothing downstream knows to distrust it.

Validate before you trust

That last point drives a concrete design rule: fall back on validation failure, not just on API errors. Every result is validated against its schema before it is accepted, and a validation failure takes the same path an outright error would, through to another provider. The cost is a second call's latency, and what it buys is that a plausible-but-wrong extraction does not silently enter the books.

A confident answer that fails the schema is more dangerous than an error. An error stops; a plausible-but-wrong value flows downstream and surfaces weeks later.

The honest gap, and what closing it showed

An earlier version of this piece ended here with an admission: accuracy in that pipeline was high in practice but unmeasured. That gap became the work. The harness now exists, and building it taught me more than the results did.

It is the four pieces above, built roughly in that order.

A small hand-labelled golden set beats a large unlabelled one. Ten documents scored field by field, plus a 60-document agreement run against the incumbent's stored extractions and a 35-document classification sweep across everyday mail. Ten sounds thin. It was enough to decide a provider migration, because each field is scored independently and a per-field miss is unambiguous.

Holding one layer constant located the real value. Running a cheap general LLM and the dedicated service over identical Markdown showed the extraction step is close to a commodity: the cheap extractor matched the dedicated one at roughly 1/200 the cost. What the incumbent was actually worth was its parsing. No headline benchmark would have told me that. Only fixing one layer and swapping the other did.

End to end, the document-native model matched or beat the incumbent on every data-bearing field at a fraction of the cost. I am deliberately not quoting per-field accuracies here, and the reason belongs to the topic: the corpus they rest on is no longer reproducible, because the hand labels lived outside version control. A number you cannot recompute is not a measurement, it is a claim. The new golden set is being rebuilt, versioned this time.

The disagreement log was the artifact, not the score. Every divergence was inspected by hand instead of averaged away, and a meaningful share of them were the reference being wrong: cash-register reports the incumbent had filed as invoices in its own store. Raw agreement was penalising the challenger for being right. A benchmark that trusts its own ground truth is measuring something else.

The results were not uniformly good either, which is the point of scoring per field rather than over one headline rate. On mixed everyday mail, document-type classification came out markedly weaker than on clean supplier invoices. Both trace to causes I can name: a taxonomy too coarse for the everyday German document types, and a recipient question that is a domain decision rather than an extraction error, since supplier invoices are addressed to the proprietor and not the trading entity.

The fourth piece is still missing. Corrections are made in a review queue, but the correction rate per field is not yet tracked as a metric feeding back into the comparison. That is the next gap, and the one that would close the loop to accuracy-per-cleanup.

Evaluation is the part most AI demos skip. It is also the part that decides whether a system can be trusted to run unattended.