Perfectly timed for the antipodean weekend during school holidays, a security vendor issues a full bodge update that crashes Microsoft Windows computers worldwide, including New Zealand. If you didn't know you're using Crowdstrike software, you do now. That is if you have a working device with which to read this.
For the record, Crowdstrike is a well-regarded information systems security company in Texas, United States, that has been involved in several high-profile cases investigating state sponsored hackers such as North Korea attacking Sony Pictures.
It's not the "Internet going down" but Microsoft Windows systems that are used in more places than you think failing. Interest.co.nz was able to confirm that users are not able to log in to ASB's banking app as of writing.
Users are reporting that just about everything imaginable has been hit, including supermarket checkouts, online banking, airlines, public transport, payments in stores including Paywave and EFTPOS (we're seeking confirmation on this), and more.
BBC reports there's worldwide travel chaos currently.
For the technically inclined, the issue appears to be a faulty update to security vendor Crowdstrike's Falcon Sensor software. This is code used to detect malware at a very low level in Windows.
It runs at very high privileges on Windows systems as a driver for the operating system kernel, the piece of code that keeps users' computers going. The reason for that is for security code to be effective, it needs to have full overview of everything on computers to catch threats.
If the Windows kernel crashes, PCs will usually display the "Blue Screen of Death" or BSoD. Windows users will know that recovering from a BSoD fault can be very difficult, and often involves manual tweaking of operating system files. Manual as in having to physically be present at the computer that needs to be restored.
Crowdstrike has acknowledged the problem in oblique language, saying its team is "fully mobilised to ensure the security and stability" of its customers.
A Crowdstrike director, Brody Nisbet, said the issue is not quite an update but a faulty channel file. The person suggested the following manual fix for users whose systems will not start up without a BSoD:
1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Go to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot normally.
We do not guarantee that the above will work. If the computer is using Bitlocker drive encryption, this could complicate access to the file system.
Next, we await the explanation from Crowdstrike as to what went wrong and also from Microsoft, which might need to rethink its policies that currently allow for global IT disasters like the above to happen.
Update Microsoft said in a support note that it has received reports from customers running Azure virtual machines that rebooting them, up to 15 times, can fix the Crowdstrike faulty update.
Update 20/07/2024 The National Cyber Security Centre (NCSC), part of New Zealand's Government Communications Security Bureau (GCSB), has issued a statement, linking to Crowdstrike's guidance page with advice on how to recover from the faulty channel update. Also, to the surprise of nobody, scammers are trying to take advantage of what might just be the world's largest IT outage (so far).
An IT outage following an update made by CrowdStrike software has caused significant disruption globally.
This update resulted in outages in windows systems.
The issue has been identified, isolated and the vendor has released remediation guidance for customers, available via their CrowdStrike Customer Portal which will be updated as the situation evolves.
We encourage New Zealand organisations that have been impacted by this disruption to review the guidance issued by the vendor and act immediately.
The NCSC has no information to indicate these issues are related to malicious cyber security activity.
However, there has been an observed increase in phishing referencing this outage as opportunistic malicious cyber actors seek to take advantage of the situation.
We encourage organisations and individuals to be alert to this increased activity. Helpful resources to protect against phishing are available below.
Know the Risks - Own Your Online
Update 21/07/2024 Microsoft said in a blog post that its estimate of the number of "Crowdstruck" Windows machine is 8.5 million. This, Microsoft said, is less than one per cent of all Windows machines.
Those Windows boxes are found at non C-list Microsoft customers:
"While the percentage was small, the broad economic and societal impacts reflects the use of Crowdstrike by enterprises that run many critical services," Microsoft vice president of enterprise and OS security, David Weston said.
41 Comments
Wow, what a catastrophe. I've just spent the last 8 hours running fixes and restores. The "fix" is not as easy as it looks on some systems, especially in Microsoft's Azure. Almost 2am and will start again at 08:00. Some heroic efforts being made by IT staff all over the world tonight. Haven't had a drink in over ten weeks, having a whisky now before I retire for the night.
There will be some very worried people this weekend who haven't managed to get their systems running.
Never deploy straight to production on Thursday or Friday! Also those two words "manual fix" on every device & terminal including those running BitLocker is truly evil, especially when banking systems, medical computers and machines, terminals and suppliers can be down so the journey will be that extra bit of interesting.
Those who had puts on Crowdstrike will feel like they have won the lotto esp the almost 93 THOUSAND percent gain on $295 strike 0DTE puts. Watching the fallout and falling knife of the stock side of things will be interesting.
The human cost could be more horrific though since medical & financial systems were down quite severely.
The irony to this, is that NZ is often regarded as a test country for updates such as this, since we are a smallish population to be put at risk, and at the start of the international timezones for a time based rollout.... so either way we can be the meat in the grinder.
That I would argue that this comment shows failure to understand that even Linux has security issues. If you want to know what happens weekly read the Hacker News - it is scary what happens and what is happening. Many issues affect obscure systems that most people don't know about but are silently making the world go round.
I would imagine if counterstrike had a version for linux or mac, and you installed it, then a faulty update could equally take down a linux or mac box.
But i feel it's far less common for linux admins would install third party commercial software for this sort of thing.
Linux is not immune though, a year or two back ubuntu put out a faulty kernel update in the main channel for the GCP kernel that wouldn't boot.
What it has made clear is that we can be thankful for a diversity of systems, so that when one 'breed' goes down the other 'breed' can stay accessible for communication and for diseminating an understanding and quick resolution for essential systems.
Sure, it's a pain to admin a diverse range of systems, but it's also times like that that the diversity pays off.
I have been running it for nearly twenty years now and it just steadily gets better and better. No surprises,no drama, a bit clunky at the beginning but still far more reliable than Windows. It is now becoming quite polished but still reliable. The same cannot be said for the Windows universe that seems to be plagued by a series of endless dramas and poor renditions interspersed with a few half reasonable renditions. They seem to have reached the limit of what they can achieve in a basic reliable system and keep running ahead with what they think is the latest gimmicks in an almost desperate effort to maintain market dominance.
I note that SAP now have a version formulated to run on Linux and the German government have ditched Windows in favor of Linux. There is a worldwide movement building in switching to Linux :-
https://en.wikipedia.org/wiki/List_of_Linux_adopters
I think that in order to survive and start offering something that is half useful, Microsoft will be forced to migrate Windows to a flavor of Linux (along with the multitude of other flavors) and start selling their software range in Linux compatible form.
The problem is the software concerned is installed in multiple computers around the world. Making CrowdStrike liable may result in CrowdStrike failing. Then the issue becomes is there any continuing support for the software - opportunities for malicious actors increase. While CrowdStrike's competitors may win, those using CrowdStrike then have to transition to another product and this is not easy or quick.
There is a can of worms that opens if CrowdStrike fails that may compound the issue of the failed update.
EDIT spelling
Let's take everything at face value from billion dollar corporates and condemn those crazy for questioning their pr narrative...
Even this article shows there's probably more to it than 'an update'...
A Crowdstrike director, Brody Nisbet, said the issue is not quite an update but a faulty channel file.
At the bottom of the issue is the software monoculture landscape of the business IT world that most managements have accepted as "business standard". It's essentially all Microsoft Windows based and Microsoft and its allies like CrowdStrike do what they can to convince businesses that this is the only way.
As a business IT manager you should ask your main application provider whether they can install on Linux (or anything else than Windows) and you will hear a long list of reasons why not. Mainly, often not mentioned will be that it is uneconomical (for them) because "everyone else uses it on Windows".
Monocultures come at a price and are bad on the long run.
Finally standing down now. Around about ten server restores plus a lot of phaffing about with other servers. In some cases it was easier to restore than try and work out how to get them to boot. Would restore as a test and then restore the real server just to be sure the restores were good. Took a cautious and slow approach.
We welcome your comments below. If you are not already registered, please register to comment.
Remember we welcome robust, respectful and insightful debate. We don't welcome abusive or defamatory comments and will de-register those repeatedly making such comments. Our current comment policy is here.