Safety-Critical Rigor in Production AI
The discipline that keeps a regulated system safe — deterministic decisions, validation at every boundary, fail-safe defaults, a human in the loop for irreversible actions — is exactly what production LLM systems need.
The mindset transfers; the certification does not
Fourteen years in regulated, safety-critical software train a specific reflex: assume the component will fail, and design so that its failure is safe. In automotive systems engineering under ISO 26262 and ASPICE, a defect is a recall, not a bug ticket, so you do not get to hope a part behaves — you design for the case where it does not.
An LLM is a probabilistic component. The reflex applies directly. To be clear about what I am and am not claiming: this is borrowing the mindset, not certifying an LLM to a functional-safety standard. There is no safety case for a language model. But the engineering discipline — deterministic decisions, validation at every boundary, fail-safe defaults, and a human in the loop for anything irreversible — is exactly what production LLM systems are usually missing.
Deterministic decisions, probabilistic reading
The clearest rule I apply: the model reads, deterministic code decides. In a document pipeline, the LLM extracts facts from an invoice, but it never decides which entity the invoice belongs to or where it should be routed — that is rule-based Python. Asking a probabilistic component to make a decision you need to be correct is asking it to guess. Use the model for the fuzzy, human part (reading); keep the consequential part (deciding) in code you can read and test.
Validate at the boundary
Probabilistic components fail quietly. So a result is validated against a strict schema before it is accepted, and a validation failure — not just an API error — triggers a fallback. The dangerous failure is not the call that errors out; it is the confident, well-formed answer that is wrong. A boundary that only checks "did it respond?" misses exactly the failure that matters.
A human airlock for irreversible actions
In a brand-facing content pipeline, the generator never holds the publish button. Everything it produces lands in an approval queue, and a separate process publishes only what a human has approved. Publishing to a brand's public channels is irreversible — so that is precisely where a human belongs.
A generator that can publish is a generator that can publish a mistake. Separate the role that creates from the role that releases.
This is the same principle as a release gate in safety-critical work: the system that produces a change is not the system that authorises it. The cost is one human touchpoint; the insurance is that a single bad generation cannot reach the public on its own.
The cost of rigor, stated honestly
None of this is free. A human approval step adds a daily touchpoint. Validating before accepting adds a second call's latency. Separating decision from reading adds moving parts. Rigor is a cost you pay deliberately, in proportion to what a failure actually costs — a wrong invoice in the books and a wrong post on a public feed both justify it; a low-stakes internal draft might not. The judgement of how much rigor a given system warrants is itself the experience I bring from regulated work.