What EOD School Taught Me About Building AI Systems That Don't Kill You

In EOD school, one of the first things they teach you about explosives is that a predictable failure is safer than an unpredictable one.

A device that always fails in the same way can be managed. You know the failure mode. You design around it. You build in the safeguard.

A device that fails randomly is the dangerous one. Not because any individual failure is worse, but because you can't anticipate it, test for it, or protect against it.

I've been thinking about this a lot as organizations rush to deploy agentic AI systems.

The Parallel

Most organizations test AI for success: does it produce the right output when everything works correctly?

Almost nobody tests AI for failure: how does it fail? In what circumstances? Is the failure mode predictable? What triggers the edge cases?

In EOD, the testing protocol goes the other way. You don't just test whether the procedure works correctly. You systematically probe for failure modes, what conditions cause this to go wrong, and can we design a system where the failure is predictable and contained?

The same discipline applies to AI systems.

Testing for Failure, Not Just Success

Here's the protocol I apply when evaluating any AI deployment:

1. Test the edge cases before production. Don't just test inputs that should work. Test inputs that are ambiguous, incomplete, contradictory, or adversarial. How does the system respond when the input is outside the training distribution? If you don't know, you haven't finished testing.

2. Map the failure modes. For every agent action that has real-world consequences, list the ways it can go wrong. Categorize them: predictable (designed around), unpredictable (requires more testing), and catastrophic (requires human-in-the-loop regardless of confidence).

3. Design for graceful degradation. When the system is uncertain, what does it do? The right answer is "it tells you it's uncertain and asks for input." The wrong answer is "it makes its best guess and proceeds without flagging the uncertainty."

4. Build the kill switch first. Before you give an AI agent the ability to send emails, update records, or execute financial transactions, design the override. How do you stop it? How fast? Who has the authority? If you can't answer these before deployment, the system isn't ready.

The Human Oversight Question

There's a debate in AI circles about how much human oversight slows things down. The argument goes: if you require humans to review AI actions, you lose the speed advantage.

In EOD, we had a version of this debate. Standing procedures could be executed autonomously by the team without calling up the chain, that's what standing procedures are for. But there was always a class of decisions that required a senior human in the loop, regardless of how confident the junior operator was.

The question wasn't efficiency. It was consequence.

For AI systems, the same logic applies: the requirement for human oversight should scale with the consequence of a wrong decision, not with the confidence level the model reports.

High confidence + low consequence = appropriate for full automation.
Low confidence + high consequence = always requires human oversight.
High consequence regardless of confidence = human in the loop, period.

The Insight

AI systems don't need to be perfect. They need to fail predictably.

A system with a 5% error rate that always fails in the same recognizable way is safer to operate than a system with a 1% error rate that fails in unpredictable ways across a wide variety of conditions.

Predictable failures can be designed around. Unpredictable failures compound.

The Takeaway

Before your next AI deployment, run this check: do you know how this system fails? Not just whether it can succeed, how it fails, when it fails, and whether the failure mode is predictable.

If the answer is no, you're not done testing.