5 major network outages in three days: It’s not a conspiracy; it’s change
Like millions of other Americans, I was stuck at an airport on the infamous Friday the 13th in November when air traffic in the U.S. came to a halt. From where I sat at Fort Lauderdale airport, the panic was obvious. Crowds overflowed at gates. Hotel rooms disappeared quickly. I got one of the last ones available in the city. It was chaos.
That’s what happens when big networks go down: chaos. And it’s only just beginning unless organizations fix how they change their network infrastructures.
Of course, the ground stop would have made news by itself. But what really got people talking was that it was one of a series of major outages that occurred over just a few days’ time. Shortly before the ground stop, a Microsoft Outlook outage left businesses unable to send email. Our company had contracts held up by it. And on Monday, November 16, customers at several major banks found they couldn’t reach their accounts.
Email, air travel, banking—three essential elements of the American economy, not to mention people’s daily lives—crashed in the span of just three business days. Suddenly, people who don’t know a router from a switch were talking about network infrastructure. Conspiracy theories multiplied on social media and in the dark corners of the internet.
I spend my working life close to the networks behind failures like these, and I can say with complete confidence that there is no conspiracy here. The root cause of all the outages might sound less satisfying than a complex conspiracy theory, but it is daunting nonetheless: The infrastructure that runs the world is changing faster than ever, and change always brings risk. When organizations fail to manage changes in their network infrastructure, the result, as we all saw last week, can be chaos.
It almost always starts with a change
The issue at the root of network outages is almost always change. Based on what we know at this point, change was at the root of the air traffic control failure. The network that carries communications between the major air traffic control centers is decades old and in the middle of a long modernization, which involves moving from legacy protocols and transmission to modern fiber and new equipment.
At one step of that transition, when new network devices were introduced, something went wrong. A misconfiguration or a misbehaving device sent traffic into the network that other devices couldn’t handle. Those devices began to fail, and the failure cascaded. Shortly after, the rush to find a hotel room began.
The Outlook outage had the same origin: configuration changes, unintended consequences, knock-on failures. Some of the triggering configurations were already months old and sat quietly until a condition in the network finally exposed them.
Why can’t networks just heal themselves?
So, why do these outages occur? Networks are supposed to route around damage. In theory, they should. But in practice, change can shake up a network infrastructure in unpredictable ways.
A network is, by definition, a set of interacting systems. Change one element and you change how its neighbors behave, and how their neighbors behave after that. Change is necessary. Unfortunately, it’s not always good, and when it’s bad it just gets worse as it travels through the network infrastructure.
That’s why the teams who run critical networks practice serious discipline before a change: pre-change testing, a mapped-out plan for every step, a rollback plan. I’m confident the people involved last week did all of that. But the interconnections inside a large network are so vast that no team can test for every eventuality. That’s not carelessness. It’s the nature of complex, interconnected systems. It could happen to even the most prepared team.
Redundancy helps, and these networks have layers of it. But redundancy does not remove complexity. In some ways, it adds to it.
Why recovery takes so long
So if a change caused the problem, the simple solution is to undo the change. But that’s not easy to do because the same interconnectedness that caused the failure makes it hard to reverse. A problem becomes exponentially more difficult to fix as it cascades through the network infrastructure and tangles itself into a complex web of systems. Here’s what has to happen for a team to fix an issue related to change:
- The team has to diagnose it. When effects cascade across the network, often it’s not obvious which change caused which symptom. This is where visibility becomes critical. You can’t fix what you can’t see. An accurate picture of the network, what’s connected to what, how each element should be configured, and what is happening right now can make the difference between a fast fix and one that takes many times longer while the team works in the dark.
- The team has to remediate, and rollback is just the start. By the time the team finds a cause, the problem has usually triggered other failures. Elements of the network infrastructure have to talk to one another again before the network works as it should. That process often has to happen in a careful order so the applications on top return cleanly. Microsoft’s team reportedly spent days painstakingly rebuilding its environment to avoid further knock-on effects.
- The team has to regain confidence that the problem won’t come straight back. The initial fix doesn’t always work. And when lives depend on the answer, caution is always the right call.
I compare this process to heart surgery on a global stage. Teams get it right 99.99% of the time, but almost nobody notices. Their effort is monumental and continuous but largely anonymous. But when there is an outage, they are doing delicate, high-stakes work while the public watches the departure boards.
Get used to network outages
However, we’re about to hear a lot more about these teams because the rate of change in our infrastructure is rapidly accelerating. There are three reasons why:
- Decades-old systems are hitting their limits and must be modernized, as is the case with air traffic control. Modernization requires change, and change always involves risk.
- AI is putting new demands on networks. Agents and AI-powered applications are doing real work, and that work runs over infrastructure that can’t adequately carry it without growth and adaptation.
- AI brings a whole new set of security implications. This reason is the most urgent of the three. In my industry, we talk about the “patchpocalypse.” AI is helping uncover vulnerabilities faster than ever, some of which have been hidden for years or even decades. The time between discovery and exploitation has shrunk from a quarter or a year to mere hours. Every patch is a change to the network, and every change carries risk.
Put all of those factors together, and you get infrastructure that’s 30 or 40 years out of date and needs to change almost immediately. The networks themselves aren’t necessarily more fragile, but their criticality, complexity, and necessity for change are all growing faster than teams and funding can keep up. Everything is going to feel shaky for a while.
What IT leaders should do now
Many of the world’s most important networks are still managed largely by hand, with limited data, limited visibility, and limited tooling for governing change. That approach isn’t sustainable anymore. IT leaders need to make two critical investments:
- Give your teams real data about their infrastructure. They should know what the network is, how it’s interconnected, how it’s meant to be configured, and what’s happening on it right now. In a crisis, they should see at a glance the impact, the affected workloads, the change that likely caused it, and the downstream effects. When speed is essential, visibility is critical. Finding a problem is a major step toward solving it.
- Govern change before it reaches the network. Change drove all the high-profile outages in mid-November 2026. Teams need to understand the risk and behavior of a change before they make it. Modern patch windows are measured in hours rather than months. Teams need to build discipline into tools and processes.
The people running these networks are doing extraordinary work under extraordinary pressure with very little credit or recognition. The November outages demonstrate how much of the world’s economy and people’s daily lives depend on them. They deserve the data and the tools they need to keep networks running and avoid chaos.
内容来自 Fast Company,版权归原作者所有。
阅读原文 ↗