Document Pipeline

The business problem

A small business, a couple of rental properties, and the household behind them generate two streams of paper. The first is incoming post — invoices, official letters, dunning notices, tax and property paperwork — that has to be opened, understood, classified, filed, and acted on before a deadline passes. By hand that ran 5–10 hours a month, and the financial documents were the unforgiving part: a missed invoice is a missed due date.

The second stream is heavier and operational: the daily business documents — point-of-sale daily-close reports, article-level sales reports, card-settlement summaries, supplier invoices. About fifteen of these land every day. Each one used to be typed into spreadsheets by hand, line by line, and that cost was high enough that the numbers were only worked through every six months. New articles went in on instinct — two quarters could pass before the figures showed whether one was a top seller or dead stock. For a small business that lag is expensive: it is capital sitting in the wrong inventory, found out months too late.

A figure you mistype is an error you find months later. A sales report you never type is a decision you make blind.

The reading is fuzzy and human; the classifying, filing, and routing are rules you can state exactly. The system is built on that seam — a wide triage front door for the mail, and a high-precision extraction engine for the daily business documents behind it.

The architecture

The pipeline runs in two stages over one shared filesystem. Stage 1 is the letter-opener: it opens, reads, and triages incoming post. Stage 2 — Landing AI, the pre-stage to the books — turns the daily business documents into accounting-grade, reconciled data.

Stage 1 — the letter-opener. A watched folder; each PDF is keyed by content hash and read into a ~24-field Pydantic schema behind a multi-backend provider abstraction that runs one primary with one automatic fallback. Deterministic Python — never the model — decides which context a document belongs to (the business, a rental property, the household) and whether it is business-relevant. Household and other paperwork is filed under a recipient/year/category/ hierarchy and synced to Notion. Stage 1 is mostly mail, but scanning is human: a business document occasionally lands in the wrong intake folder. The same content classification catches it and reroutes it to Stage 2 rather than mis-filing it, so a misplaced scan does not become a missing entry in the books.

The reading layer inside Stage 1 is on its second architecture. It began as local Tesseract OCR feeding a general LLM; today a document-native parse-extract model (Mistral OCR) takes the PDF directly and returns layout-aware Markdown plus the typed fields in one call, with uncached volume bundled into asynchronous batch jobs at half the per-page price. The provider abstraction is what made that swap a configuration change rather than a rewrite.

The operations layer. What the 2026 iteration added around the model is the part that makes it a system rather than a script. The extraction cache is versioned by schema and prompt hash, so a prompt change invalidates exactly the extractions it affects — re-run at batch price. A re-scanned letter is caught by a content fingerprint (sender, date, amount, reference number) rather than byte identity and diverted to a duplicates folder instead of being filed twice. Every document carries a UUID embedded in the PDF's own metadata, which follows it across the filesystem, the Notion index, and a self-hosted Paperless-ngx full-text archive. Degraded extractions are never silently filed: they carry a review flag, get corrected by a human in the archive's UI, and the correction syncs back into cache and index; hard failures go to quarantine. The taxonomy grew from nine document types to fifteen — tax assessments, payslips, bank statements, certificates — each extraction tagged with tax relevance and context, because in a German household, findable-at-tax-time is half the point of filing.

Stage 2 — the accounting pre-stage. This is where the roughly fifteen business documents a day become bookkeeping data. Each is first sorted by a keyword-scoring classifier into one of four document types — invoice, daily-close, article report, card-settlement — each with its own typed schema. A cost-aware three-tier router then decides how to read it, and results are stored zlib-compressed in SQLite, archived by year/month/type.

Route each document to the cheapest tool that can actually read it — and pay for the expensive one only when it earns its keep.

Reconciliation. Extracted line items are matched against a supplier-article catalog (in Supabase), so the same product reconciles across vendors whatever a given invoice calls it — turning raw extractions into comparable, queryable data rather than disconnected rows. That reconciliation is what makes article-level performance answerable on demand instead of once every six months.

Decisions & trade-offs

LLMs extract facts; deterministic code makes decisions. The model reads; rule-based Python classifies by business context and routes. Asking the model which entity an invoice belongs to would be asking it to guess, so it never does.

Fallback on validation failure, not just on API errors. A provider can return a confident, well-formed answer that still doesn't match the schema — the dangerous case, because a plausible-but-wrong extraction sails into the books. So a result is validated before it's accepted, and a validation failure falls through to another provider. The trade-off is a second call's latency and cost, bought for guaranteed structure.

The provider migration was a decision, not a default. What decided it wasn't headline price but accuracy-per-cleanup: a cheaper extraction that needs an afternoon of data-cleaning isn't cheaper.

The fourth migration was earned by a harness, not a hunch. An earlier version of this study admitted that accuracy was high in practice but unmeasured. That gap became the work: a hand-labelled ground-truth corpus and per-field scoring — recipient, type, date, amount, reference number — benchmarked the candidates layer by layer. Holding the parse constant and swapping only the extractor showed that the extraction step is a commodity: a cheap general LLM matched the dedicated service on identical Markdown at roughly 1/200 of the cost, which located the incumbent's real value in its parsing. The document-native model then matched or beat it end to end, and a corpus run across all four business document types settled it with near-perfect agreement on dates and amounts.

Disagreement with the reference is evidence, not error. Every divergence between challenger and incumbent was inspected by hand rather than averaged away — and a number of them turned out to be the incumbent's fault, including cash-register reports its own store had filed as invoices. The migration was decided on the disagreement log, not on the headline agreement rate.

Batch over sync, and a local path held in reserve. Mail is not time-critical, so uncached PDFs are bundled into asynchronous batch jobs at half the per-page price — minutes of latency, occasionally an hour — dropping automatically to synchronous calls when batching is unavailable. For documents that arguably shouldn't leave the house at all, a fully local two-stage path was benchmarked and shelved as a ready option: Docling parses acceptably in half a minute to a few minutes per document on CPU, while the GPU-bound alternatives (Marker, MinerU) took over half an hour for a single PDF.

What broke

The migration benchmark was meant to produce a clean accuracy table; it produced the answer by failing instead. The dedicated OCR service returned raw labelled boxes that didn't map to the target schema at all — every line item came back unmatched without a custom normalization layer on top. In production that meant a backfill re-extracting 10,842 line items from ~1,800 documents the old service had captured as text but never structured, plus ~70 merchant-name spelling variants to reconcile. The general LLM was usable but not flawless on bad scans — a date read a year wrong, a vendor mangled into OCR noise. And the intake schema guarantee is honest-but-imperfect: validation drives the fallback at the read boundary, but the dict finally written passes through a lighter coercion layer, so the guarantee is weaker at the write boundary.

The 2026 intake benchmark broke in more instructive ways. The document-native model returned an empty {} on about half the corpus; the root cause was a schema in which every field was Optional, which the model's structured-output mode reads as permission to return nothing. Making the core fields required fixed it — a contract detail no documentation states. The prompt then hardened one observed misread at a time: on a cash receipt the model returned the cash tendered instead of the amount actually to pay; on B2B invoices it put the buyer in the sender field instead of the issuer on the letterhead. Every rule in the prompt traces back to a specific line in the disagreement log.

The reference data broke too. The archive used as ground truth contained misnamed files — a supermarket receipt filed as an article report — and the benchmark CSV split on an unquoted comma in a filename. Even the incumbent's own store held misclassifications, so raw agreement penalized the challenger for being right. The operational footnotes: batch jobs answered HTTP 402 until billing was enabled in the provider console, and the batch result arrives as a streaming response that has to be read before the payload exists.

The outcome

In use since 2023 across versions, with the move onto Landing AI made in late 2025 and the intake's reading layer moved onto Mistral OCR in mid-2026. The extraction engine runs across ~3,500 invoices and ~15,700 line items, most parsed free locally and only the hard few hundred sent to the paid model at roughly €0.14 each; the intake now reads at $0.002 a page in batch — for a month of mail, cents rather than euros.

Two burdens lifted. The mail intake replaced 5–10 hours a month of opening, sorting, and filing with structured, deadline-tracked records. The heavier one is the daily business documents: about fifteen a day that previously had to be typed into spreadsheets by hand. Checked against real examples — a daily-close report, an article-sales report, a multi-line wholesale invoice — three to five minutes of careful entry each is if anything conservative, since a long invoice whose positions have to be matched against the article catalog runs longer. Fifteen a day is on the order of an hour daily, five to six hours a week. Valued at Germany's statutory minimum wage plus the roughly 30% employer on-costs — about €18 an hour — that is on the order of €400–470 a month, near €5,000 a year, of pure data entry. Offshoring it is cheaper per hour, but the documents are German and every extracted figure has to be re-validated by a German speaker, so the validation overhead eats most of the saving. The pipeline removes the task rather than relocating it.

The larger win is not the hours, though — it is the decision they were blocking. Sales that were worked through every six months because the manual entry was prohibitive are now reconciled as the documents process, so whether a newly introduced article is a top seller or dead stock is visible within days instead of two quarters later. For a small business deciding what to stock, that is the difference between steering and guessing.

The honest gaps named in the previous version of this study — extraction accuracy high in practice but unmeasured, and the whole thing running by hand on one machine — did not survive the year. Both became work packages: a benchmark harness with a hand-labelled ground-truth corpus closed the first, and a watch mode with autostart, file logging, and a cost-accounting CLI closed the second. The gaps that replaced them are just as honest: sensitive documents still go to a cloud model (the local path is benchmarked, not wired), and the intake's new model has demonstrated on the corpus that it could read the accounting engine's document types too — that consolidation is measured and designed, but not yet built.