The Paradox of Automation

On June 1, 2009, Air France Flight 447 fell out of the sky because its pilots didn't know how to fly their own plane. That's the brutal summary. The more interesting question is why they didn't know — and the answer implicates the very system that had kept the Airbus A330 safe for fifteen years.

The fly-by-wire system on the A330 was, by any reasonable measure, a masterpiece of engineering. It smoothed clumsy inputs into graceful maneuvers. It prevented pilots from doing anything aerodynamically stupid. It was so good that in the plane's first fifteen years of commercial service, there had been zero crashes. But when ice crystals blocked an airspeed sensor and the system downgraded itself to a mode with less hand-holding, the co-pilots were so unused to actually flying the aircraft that they stalled it and didn't even understand what was happening. Captain Dubois, arriving from a nap, stood behind them for 23 silent seconds — long enough to fall 4,000 feet — unable to diagnose the situation. The plane hit the Atlantic four minutes later. Everyone died.¹

Tim Harford identifies three strands to this. First, automation accommodates incompetence — the system is so easy to operate that an unskilled person can function for years before their lack of skill becomes visible. Second, even skilled operators atrophy — you can't maintain a skill you never practice. Third, automated systems tend to fail in unusual situations, which are precisely the ones that require the most skill to handle. A more reliable system makes all three problems worse, not better.¹

Earl Wiener, the cult figure of aviation safety, put it more pithily: "Digital devices tune out small errors while creating opportunities for large errors."

The process that stifles creativity (on purpose)

There's a counterexample, and it's instructive. The on-board shuttle software group at NASA's Johnson Space Center wrote code with an error rate that made commercial software look like cave paintings. The last three versions of their 420,000-line program had one bug each. The group's secret wasn't genius — it was a fanatical process that deliberately suppressed individual brilliance. No hotshot coders. No creative flourishes. Every line annotated, every change traced to a 2,500-page specification, every error logged in a database going back twenty years. The culture was "aggressively intolerant of ego-driven hotshots."²

The shuttle group solved the automation paradox by never automating the humans away. The software was automated; the process of writing it was relentlessly manual. They had verifiers who tried to break the code and developers who tried to write it perfectly, in formal rivalry. They didn't just fix bugs — they fixed whatever permitted the bug. Their error database was so comprehensive they could write models that predicted how many errors should appear in each new version, and if reality came in low, they assumed they'd missed something.

But here's the thing I find most interesting: this process worked because the stakes were unambiguous. Ted Keller flew to Florida before every launch to personally sign a document certifying the software wouldn't kill anyone. Most software has no equivalent of that moment. Most software exists in the vast middle ground where it's not critical enough to justify the shuttle's $35 million per year budget for 260 people writing one program — but it's not unimportant either. And in that middle ground, the paradox of automation runs wild.

The squareabout

The Dutch traffic engineer Hans Monderman found an elegant solution to a version of this problem that had nothing to do with software. Two children had been killed by cars in Oudehaske, and Monderman's radar gun confirmed drivers were going too fast. The standard engineering response would be traffic lights, speed bumps, warning signs. Instead, Monderman removed the signs, replaced the asphalt with red brick, and eliminated the distinction between road and sidewalk.¹

Drivers were so confused they slowed to a crawl. Monderman couldn't even capture their speed on his radar gun. His party trick when showing journalists his later "squareabout" in Drachten — a junction redesigned with the same philosophy — was to close his eyes and walk backwards into traffic. Cars would flow around him.

The principle is identical to the automation paradox, just inverted: by forcing people to confront the possibility of small errors constantly, the chance of catastrophic ones dropped. Accidents at the Drachten squareabout halved. Traffic flow actually improved. It worked because it kept humans engaged rather than letting them glaze over.

Programming sucks (and that might be load-bearing)

There's a third voice here that complicates things beautifully. Peter Welch's "Programming Sucks" is a rant, not an analysis, but it captures something the other two sources dance around: the lived experience of software as perpetual near-catastrophe.³

Every program you've ever used, Welch argues, was built by a process that resembles building a suspension bridge halfway, realizing nobody actually knew how suspension bridges work, then adding columns and leaving the cables attached because "nobody knows which parts they're holding up, but everybody's pretty sure they're important parts." Standards are unicorns. Half the internet runs on code with comments like "TODO: FIX THIS IT'S A REALLY DANGEROUS HACK" written ten years ago. The only reason it works at all is that thousands of people are maintaining it around the clock, and if they all took a lunch break at the same time, civilization would collapse.

This is the paradox of automation from the inside. The shuttle group proves you can write perfect software if you're willing to pay for it. Air France 447 proves what happens when you automate humans out of the loop. But most software lives in Welch's world, where the automation is partial, the process is chaotic, the humans are overstretched, and the whole thing holds together through a combination of duct tape and prayer. That's not a failure state — it might actually be the Monderman squareabout of software. The constant small fires keep people engaged in a way that a perfectly automated system wouldn't.

I'm not sure I believe that fully. But it's worth sitting with the discomfort of it: the shuttle group's perfection required total process control over a single program. The internet has no such luxury. And maybe the messy, chaotic, permanently-on-fire state of real software is what keeps it from failing in the catastrophic way that perfectly smooth systems eventually do. The mess is the immune system.

Or maybe we're all just lucky. The paradox of automation doesn't resolve neatly. It just asks you to notice how much of your safety depends on someone staying alert — and how hard every well-designed system makes that to do.