Safety-Critical Rigor in Production AI
What keeps a regulated system safe: code makes the decisions, every boundary validates, the system stops when in doubt, and anything irreversible needs a human to approve it. That is exactly what production LLM systems are missing.
The mindset transfers; the certification does not
Fifteen years in regulated, safety-critical software train a specific reflex: assume the component will fail, and design so that its failure is safe. In automotive systems engineering under ISO 26262 and ASPICE, a defect is a recall, not a bug ticket, so you do not get to hope a part behaves. You design for the case where it does not.
An LLM is a probabilistic component. The reflex applies directly. To be clear about what I am and am not claiming: this is borrowing the mindset, not certifying an LLM to a functional-safety standard. There is no safety case for a language model. The engineering discipline does transfer, and it is what production LLM systems are usually missing: code makes the decisions, every boundary validates, the system stops when in doubt, and anything irreversible needs a human to approve it.
Deterministic decisions, probabilistic reading
The clearest rule I apply: the model reads, deterministic code decides. In a document pipeline, the LLM extracts facts from an invoice, but it never decides which entity the invoice belongs to or where it should be routed. That is rule-based Python.
Asking a probabilistic component to make a decision you need to be correct is asking it to guess. So the model gets the fuzzy, human part, which is reading. The consequential part, deciding, stays in code you can read and test.
Validate at the boundary
Probabilistic components fail quietly. So a result is validated against a strict schema before it is accepted, and a validation failure triggers the same fallback an API error does. The dangerous failure is not the call that errors out. It is the confident, well-formed answer that is wrong. A boundary that only checks "did it respond?" misses exactly the failure that matters.
A human gate for irreversible actions
In a brand-facing content pipeline, the generator never holds the publish button. Everything it produces lands in an approval queue, and a separate process publishes only what a human has approved. Publishing to a brand's public channels is irreversible, so that is precisely where a human belongs.
A generator that can publish is a generator that can publish a mistake. Separate the role that creates from the role that releases.
This is the same principle as a release gate in safety-critical work: the system that produces a change is not the system that authorises it. It costs one human touchpoint. In exchange, a single bad generation cannot reach the public on its own.
What this rigor costs
None of this is free. A human approval step adds a daily touchpoint. Validating before accepting adds a second call's latency. Separating decision from reading adds moving parts. Rigor is a cost you pay deliberately, in proportion to what a failure actually costs. A wrong invoice in the books and a wrong post on a public feed both justify it; a low-stakes internal draft might not. The judgement of how much rigor a given system warrants is itself the experience I bring from regulated work.