Document Pipeline

The architecture

Two stages over one shared filesystem, split along the work rather than the technology. Stage 1 is the letter-opener: it opens, reads and triages incoming post, so nothing carrying a deadline goes unseen. Stage 2 is the pre-stage to the books: it turns the daily business documents into accounting-grade, reconciled data a person can actually query.

One seam carries the whole design. Reading a document is fuzzy and human, so a model does it. Deciding what a document is and where it belongs is a rule you can state exactly, so code does it, and the model is never asked to guess.

Stage 1, the letter-opener

A watched folder. Each PDF is keyed by content hash and read into a ~24-field Pydantic schema, behind a multi-backend provider abstraction that runs one primary with one automatic fallback.

Deterministic Python decides which context a document belongs to, meaning the business, a rental property or the household, and whether it is business-relevant. Never the model. Household and other paperwork is filed under a recipient/year/category/ hierarchy and synced to Notion.

Scanning stays human, so a business document occasionally lands in the wrong intake folder. The same content classification catches it and reroutes it to Stage 2 rather than mis-filing it, so a misplaced scan does not become a missing entry in the books.

The reading layer is on its second architecture. It began as local Tesseract OCR feeding a general LLM. Today a document-native parse-extract model (Mistral OCR) takes the PDF directly and returns layout-aware Markdown plus the typed fields in one call, with uncached volume bundled into asynchronous batch jobs at half the per-page price. The provider abstraction is what made that swap a configuration change rather than a rewrite.

The operations layer

What makes this a system rather than a script is not the model. It is what sits around it.

The extraction cache is versioned by schema and prompt hash, so a prompt change invalidates exactly the extractions it affects, and those are re-run at batch price.

A re-scanned letter is caught by a content fingerprint, meaning sender, date, amount and reference number, rather than byte identity. It is diverted to a duplicates folder instead of being filed twice.

Every document carries a UUID embedded in the PDF's own metadata, which follows it across the filesystem, the Notion index and a self-hosted Paperless-ngx full-text archive.

Degraded extractions are never silently filed. They carry a review flag, get corrected by a human in the archive's UI, and the correction syncs back into cache and index. Hard failures go to quarantine.

The taxonomy grew from nine document types to fifteen, among them tax assessments, payslips, bank statements and certificates. Each extraction is tagged with tax relevance and context, because in a German household, findable-at-tax-time is half the point of filing.

Stage 2, the accounting pre-stage

This is where the roughly fifteen business documents a day become bookkeeping data. A keyword-scoring classifier first sorts each into one of four document types, invoice, daily-close, article report or card-settlement, each with its own typed schema.

A cost-aware three-tier router then decides how to read it. Results are stored zlib-compressed in SQLite, archived by year/month/type.

Route each document to the cheapest tool that can actually read it, and pay for the expensive one only when it earns its keep.

Reconciliation. Extracted line items are matched against a supplier-article catalog in Supabase, so the same product reconciles across vendors whatever a given invoice calls it. That turns raw extractions into comparable, queryable data rather than disconnected rows, and it is what makes article-level performance answerable on demand instead of once every six months.

Decisions & trade-offs

LLMs extract facts; deterministic code makes decisions. The model reads, rule-based Python classifies by business context and routes. Asking the model which entity an invoice belongs to would be asking it to guess, and a guess there is not a typo but a wrong tax treatment, so it never does.

Fallback on validation failure, not just on API errors. A provider can return a confident, well-formed answer that still does not match the schema. That is the dangerous case, because a plausible-but-wrong extraction sails into the books. So a result is validated before it is accepted, and a validation failure falls through to another provider. The trade-off is a second call's latency and cost, bought for guaranteed structure.

The provider migration was a decision, not a default. What decided it was not headline price but accuracy-per-cleanup. A cheaper extraction that needs an afternoon of data-cleaning is not cheaper.

The fourth migration was earned by a harness, not a hunch. An earlier version of this study admitted that accuracy was high in practice but unmeasured. That gap became the work: a hand-labelled ground-truth corpus and per-field scoring across recipient, type, date, amount and reference number, benchmarking the candidates layer by layer.

Holding the parse constant and swapping only the extractor showed that the extraction step is a commodity. A cheap general LLM matched the dedicated service on identical Markdown at roughly 1/200 of the cost, which located the incumbent's real value in its parsing. The document-native model then matched or beat it end to end, and a corpus run across all four business document types settled it with near-perfect agreement on dates and amounts.

Disagreement with the reference is evidence, not error. Every divergence between challenger and incumbent was inspected by hand rather than averaged away, and a number of them turned out to be the incumbent's fault, including cash-register reports its own store had filed as invoices. The migration was decided on the disagreement log, not on the headline agreement rate.

Batch over sync, and a local path held in reserve. Mail is not time-critical, so uncached PDFs are bundled into asynchronous batch jobs at half the per-page price, at minutes of latency and occasionally an hour, dropping automatically to synchronous calls when batching is unavailable.

For documents that arguably should not leave the house at all, a fully local two-stage path was benchmarked and shelved as a ready option: Docling parses acceptably in half a minute to a few minutes per document on CPU, while the GPU-bound alternatives Marker and MinerU took over half an hour for a single PDF.

What broke

The migration benchmark was meant to produce a clean accuracy table. It produced the answer by failing instead.

The dedicated OCR service returned raw labelled boxes that did not map to the target schema at all, and without a custom normalization layer on top every line item came back unmatched. In production that meant a backfill re-extracting 10,842 line items from ~1,800 documents the old service had captured as text but never structured, plus ~70 merchant-name spelling variants to reconcile.

The general LLM was usable but not flawless on bad scans: a date read a year wrong, a vendor mangled into OCR noise. And the intake schema guarantee is honest but imperfect. Validation drives the fallback at the read boundary, while the dict finally written passes through a lighter coercion layer, so the guarantee is weaker at the write boundary.

The 2026 intake benchmark broke in more instructive ways. The document-native model returned an empty {} on about half the corpus. The root cause was a schema in which every field was Optional, which the model's structured-output mode reads as permission to return nothing. Making the core fields required fixed it, and no documentation states that contract detail.

The prompt then hardened one observed misread at a time. On a cash receipt the model returned the cash tendered instead of the amount actually to pay. On B2B invoices it put the buyer in the sender field instead of the issuer on the letterhead. Every rule in the prompt traces back to a specific line in the disagreement log.

The reference data broke too. The archive used as ground truth contained misnamed files, such as a supermarket receipt filed as an article report, and the benchmark CSV split on an unquoted comma in a filename. Even the incumbent's own store held misclassifications, so raw agreement penalized the challenger for being right.

Two operational footnotes: batch jobs answered HTTP 402 until billing was enabled in the provider console, and the batch result arrives as a streaming response that has to be read before the payload exists.

The last thing to break was not technical. Routing household paperwork into Notion as deadline-tracked tasks worked exactly as designed. By August 2026 that task database held 48 open items, every one of them past its due date, and not one of them had been filed there wrongly.

The diagnosis is pull instead of push. A deadline in a database exists only for someone who opens the database, and opening it was the one step the system could not take on anybody's behalf. The correspondence database had also quietly become a second archive beside the full-text one, which is duplication rather than redundancy.

So the task layer is being retired rather than repaired: deadlines get pushed to where attention already is, because a database somebody has to go and visit is precisely what does not work here. The archive keeps one home instead of two. That is the more useful finding in this study. A system nobody visits ends up indistinguishable from a system nobody built, and no accuracy figure catches that.