Crowd Struck!

In the 1950s as the volume of airline travel rose in America, a new problem emerged for airline operators.

Tickets for their flights could be booked at any airport and the sales were maintained manually on registers. However, there was a need to update every other airport to avoid overbooking. This used to be done by phone and registers had to be updated across the country.

This forced the airline industry to look at computers. American Airlines developed an application called Sabre to make computerised booking possible. There is an argument to be made that they developed the Internet before the Internet since all these airports had to be connected.

By the 1980s computers found their way into every corporation for a variety of applications. Today it would be impossible to think of running a company without computers.

In the 1990s Microsoft became dominant and when it comes to corporate IT purchases, there is a certain lock-in involved. By the late 90s as Apple was staring death in the face Microsoft had a 95% market share. This attracted a lot of unwanted attention which is exactly why Bill Gates loaned Apple money when Steve Jobs returned to rejuvenate the company.

Apple became cool. Today, for most of the people that work in tech it would be normal to see a sea of Macs when they enter their offices. At the same time, we forget the places that continue to use computers, especially ones that we do not see, such as when you visit the check-in counter at the airport or the teller window at the bank.

As I said, IT purchases have a lock-in due to systemic penetration, training, processes, etc. Hence a lot of old-world businesses that started using computers in the 1990s, are all of Windows. Most ATMs still use Windows 98!

Last week people realised who those businesses were rudely.

The absolute monopoly of Windows attracted another kind of attention - viruses, spyware and malware. A Virus is a program which moves through networks and infects computers. Now, if 1 in 100 computers is a Mac, going viral chasing Macs would be impossible. Therefore, most of the viruses targeted the Windows ecosystem. On the back of that threat, many anti-virus companies and anti-malware companies emerged.

As the 2010s swung around, spyware and malware became a bigger threat as the value of data became increasingly obvious. The objective was not to freeze the system but to extract data that could be leveraged for extortion, espionage or other such ends.

It was at the same time that SaaS was becoming a big thing and companies like Microsoft and Adobe who were used to selling software packages with one-time payment were beginning to move their products into the SaaS fold. Till 2010, you would have bought a one-time license for Office 2010 and used it as long as you pleased and security updates and patches would keep coming.

After Office 2013, this changed and getting a subscription became a necessity.

In this environment, a company called Crowdstrike was founded in 2011. The company promised to provide cloud workload protection, endpoint security, threat intelligence and cyberattack response services.

Under the hood, Microsoft began using Crowdstrike to protect itself from cyberattacks. To enable that, Microsoft granted the company unprecedented access to its operating system at the Kernel level.

In an operating system, the Kernel is the lowest abstraction to interact with the hardware. It goes down to the level of machine language which most programmers would not have seen.

Crowdstrike had kernel-level access and since their service was considered fundamental to the software system, they were allowed to push updates without informing the user; automatically.

Security and IT analysts searching for the root cause of the gargantuan outage say that it appears to be related to a “kernel driver” update to CrowdStrike’s Falcon software. Kernel drivers are the software components that allow applications to interact with Windows at its deepest level, the core of the operating system known as its kernel. That highly sensitive level of access is necessary for security software, so that it can run prior to any malicious software installed on the system and access any part of the system where hackers might seek to plant their code. As malware has improved and evolved, it has pushed defense software to require constant connection and more extensive control.

That deeper access also introduces a far higher possibility that security software—and updates to that software—will crash the whole system, says Matthieu Suiche, head of detection engineering at the security firm Magnet Forensics. He compares running malicious code detection software at the kernel level of an operating system to “open-heart surgery.”

Source: Wired

On the 19th of July 2024, an update was pushed that had an error.

The problem originated with an Austin, Texas-based cybersecurity firm called CrowdStrike, relied upon by most of the global technology industry, including Microsoft, for its Falcon program, which blocks the execution of malware and cyber-attacks. Falcon protects devices by securing access to a wide range of internal systems and automatically updating its defenses – a level of integration that means if Falcon falters, the computer is close behind. After CrowdStrike updated Falcon on Thursday night, Microsoft systems and Windows PCs were hit with a “blue screen of death” and rendered unusable as they were trapped in a recovery boot loop.

Source: The Guardian

And like that - a company that was tasked with protecting systems from being threatened with a cyberattack, inadvertently perpetrated the world’s greatest cyber attack!

10,000 flights had to be cancelled in the US alone. A mid-sized plane carries about 180 people; at an average ticket price of $100, you bill about $18,000 per flight; altogether that is $180,000,000 worth of cancellations. That number is extremely conversation - assumes all flights were domestic and flying a 1000 km route and does not account for payments for legroom, in-flight food, excess baggage, etc.

Across industries, the odds are that billions of dollars of losses would have been caused by systems that were not functioning.

This is a reflection of the power that individual companies hold on not just lives but entire systems that we take for granted every single day.

Who would have thought that one typo, on one line of code written in Texas would mean someone at the Hyderabad airport would have to carry a boarding pass written by hand?

The bigger question is one of transparency.

With the increasing penetration of technology into every aspect of our lives - some people have smart locks that are connected to the internet - there is a need to know what the hell we are depending on! What can give that could lock me inside my own house?

In a world filled with trade secrets, we have no idea what or who we are banking on to keep things functioning the way we are used to. I had not even heard of Crowdstrike before last week and I suppose that might be true for many across the world.

Is this right? Is this tenable?