It’s incredible that a single company is responsible for so much IT infrastructure. It’s also very wrong.
It’s incredible that such a company would roll out an update after essentially no testing. It’s bad enough when joints like tumblr make site changes that break things, b/c insufficient (if any!) proper testing. I mean, “Boo hoo! I can’t post this news story!” is just an annoyance. Thousands being turned away from pharmacies, doctor, and surgery appointments; arriving 18 hrs late after delayed flights; banks and shops being unable to open; stock mkts being on the fritz - that is maaaaaajor fucking negligence.
crowdstrike and its still-fucking-up ceo need to be sued into oblivion and/or be taken over by the gov’t and dissolved. More companies need to be involved in (and taking proper care of!!!) IT infrastructure to avoid this shit ever happening again.
There have been issues with Crowdstrike and Linux before, and there probably will be again.
The “Mac and Linux hosts are not affected” meant “by this issue, this time.”
The last time I had to deal with Crowdstrike wanting to take out our Linux hosts, it didn’t crash them immediately, it just used up all the memory and crashed them that way. That gave us enough time to use our configuration management to disable Crowdstrike on our server fleet until a fix was ready to release.
Linux admins are not crowing over Windows admins right now: those of us with experience in this know full well that it could easily have been us. Next time it might be.
A hearty “fuck you” to Delta Airlines, who jerked us around for hours over a flight that was supposed to depart at 9:41 pm, but instead crept up and up with delays until it was finally cancelled at 3:05 am, stranding everyone on the flight in Atlanta and forcing me to sleep on the floor of the terminal.
Also, fuck you Delta for issuing me a hotel voucher that was only good for the day before, because you fucking jackasses didn’t account for the nearly six hour delay that pushed everyone into the next day.
Also, fuck you Delta for adding insult to injury with your absurd $12 food voucher, which even if one can actually use after all the fine print is read and conditions are met, won’t even cover a refrigerated sandwich from one the newsstands in the terminal.
Also, fuck you Delta for rebooking me for two fucking days from now, instead of on one of the closer flights that still had one-third of the seats empty.
Also, fuck you to the employees who protest that “it’s not our fault” and “there’s nothing we can do.” You voluntarily work for an organization that prioritizes fragility over resilience, shareholder value over customer service, and you are representatives of that organization. I am therefore entirely justified in targeting you with my ire.
The head of every Delta executive should be on the block right now. Instead they’ll get bonuses and continue to fail up.
I’m still at the airport, and will be until at least 11:00 pm tonight (it’s nearly 10 am now) when I’ve bought an expensive ticket on Southwest so that can actually (God willing) get home.
The first thing I saw on my new boarding pass was the word “delayed.”
I just sat in on a meeting of Elder Gods of the Internet and some takeaways were
there is an inherent tension between testing and timeliness when you’re patching for security,
liability transfer (versus actual cyber-security) as the motivation for using CrowdStrike or other vendors can subtly skew results by generating perverse incentives,
outsourcing cybersecurity (given that it is required) is legitimate. A stationery company isn’t in the computer business, and ought not to have to hire individual security people to do business.
doesn’t MicroSoft still check Windows kernel drivers before they are signed?
null pointer? really?
one company was saved when it randomly patched its DNS server first, which broke, and broke the ability of all other machines to retrieve the patch.
a design point to consider is allowing only one external-facing machine to download patches and check them, leaving you a smaller attack surface and,
if you can be bothered, hard-wire IP addresses for patches, not DNS names.
a shockingly long list of CrowdStrike adjacent domains were registered in the past couple of days.
where is the ability for roll-back? That used to be a thing in operating systems. The mismatch between the fix for CrowdStrike’s oopsie and the skills of your median computer user is a big problem.
I didn’t realize that CrowdStrike was patching as often as every 4 hours. If they are doing that then, and this is purely from a statistical standpoint, they are missing something fundamental in their model. I’ve read their fuzzy “AI” docs, and I get why there’s no full disclosure on techniques, but a frequency of update like that absent an immediate, new, 0-day type threat is a sign that something isn’t right.
I understand the need for an ire target, but I never go after service employees in situations like that. Their pay is shit, and it’s not necessarily easy to just quit and find another job, and as you imply, problems are rarely their fault.
And hey, being a friendly dinosaur instead just works better.
Yeah, the ones who are sincerely trying to help I’m very diplomatic with. The others who seem to consider this abrogation of responsibility to the passengers to be business as usual get both barrels.
Microsoft blames it on having to obey the law which tries to make competition a thing in their monopoly.
It’s bullshit though. They are outsourcing security, and it seems very likely that they are outsourcing validating the kernel (because technically this had to be approved by Redmond), because it’s cheaper than doing it themselves and it spreads liability, and also, I don’t know, they sacked the people who know how to do this to show Wall Street that they are Very Grown Up and Responsible Shareholder Value Makers. Oh and let’s piss away tens of billions on buying GPUs and server farms and licenses so we can pretend that “AI” is actually a viable business proposition because our shareholders have heard about it.
As Cory describes MS “too big to fail, too big to jail, too big to care.”
Their corporate incompetence is so longstanding, it rivals their aggression.
El Reg wrote this article with both eyes on Redmond’s attack lawyers. And don’t forget, lawyering is actually the institutional DNA expertise of MS. You don’t think they actually bothered writing DOS? But they did lawyer up a super deal.
Update: My Southwest flight left over an hour late (looks like the whole airport was tied up, so that at least isn’t on Southwest), but I finally walked in my door at 3:00 am this morning, after 26 hours of chaos at the Atlanta airport.
And now I get to do it all over again this Saturday.
It must be a weird tension to navigate the profitability of that product versus the security of Windows itself. “Hey, Bob, Alice over in the Defender group wants you to leave that DLL-loading bug unpatched for at least one more release, there’s been a slight uptick in exploits and that’s been good for her group.”
I’m not aware of any specific cases of something being sandbagged on the windows side for the sake of the security side(though there’s probably some argument to be made that some of the optional ASR rules should probably be default behaviors in windows or office); but precisely that conflict of interest(in relation to the “mailitemsaccessed” audit log item for Exchange Online being exclusive to Audit(Premium) rather than Audit(Standard), get your E5 now!) was one of the areas of acute displeasure in CISA’s scathing review of the mysterious token forgery incident last year.
They have since changed the licensing requirements to see “mailitemsaccessed” in the audit log to Audit(Standard) under heavy pressure.
It’s on the cloud side where they seem to be doing the most intricate slicing and dicing of security products; and the most carefully calibrated mixture of bundles and upsells. The influence of ‘cloud’ certainly hangs over the security guidance they provide for their on-prem products(documentation on things like red forest AD configs is typically still written like it’s 2012R2, if it hasn’t link rotted, and is usually prefaced with a “just use AAD unless you hate security” message); but the client side seems to get more neglect than active sabotage.
Is anyone else a bit surprised that coordinated attacks on cables at strategic points in the rail network have not led to reports of other outages? My layman’s understanding was that the convenience of big, continuous, strips of right-of-way between relevant places was so great that you’d expect to find a great deal of fiber for various purposes, not just the railway-specific requirements, running in the vicinity of a rail line; and fire is not a precise tool for snipping specific cables.
