Sysadmin

9183 readers

209 users here now

A community dedicated to the profession of IT Systems Administration

No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world

founded 2 years ago

MODERATORS

DarraignTheSane@lemmy.world

00Lemming@lemmy.world

L3s@lemmy.world

140

Delta can sue CrowdStrike over computer outage that caused 7,000 canceled flights (www.reuters.com)

submitted 1 day ago by cm0002@lemmy.world to c/sysadmin@lemmy.world

28 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] ricecake@sh.itjust.works 6 points 22 hours ago (1 children)

It's absolutely not required behavior! Software for servers has very different requirements from software for end users, and if you have a lot of them you also want to manage your end user machines differently.

Updates can go wrong, and if you roll out a bad update to everything at once you can crash everything and lose a lot of money. As aptly demonstrated by cloudstrike.

That's why Delta and many other companies disabled the auto update functions: so they could control the rollout cadence.
They reasonably believed that disabling autoupdates disabled them. They didn't expect a second autoupdate system that wasn't documented, wasn't controlled by the autoupdate system settings and couldn't be disabled.

[–] SupraMario@lemmy.world 1 points 20 hours ago (1 children)

It's not a second auto update. It's %100 documented in the software and you can %100 throttle it. Channel files are heavily discussed when you roll out CS.

[–] ricecake@sh.itjust.works 1 points 18 hours ago* (last edited 18 hours ago) (1 children)

https://www.crowdstrike.com/en-us/blog/falcon-content-update-preliminary-post-incident-report/

Might want to let crowdstrike know.

Rapid Response Content Deployment

Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment.

Improve monitoring for both sensor and system performance, collecting feedback during Rapid Response Content deployment to guide a phased rollout.

Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed.

Provide content update details via release notes, which customers can subscribe to.

https://www.theregister.com/2024/07/23/crowdstrike_lessons_to_learn/

Maybe you're thinking of changes that they made as a result of the incident?

[–] SupraMario@lemmy.world 1 points 18 hours ago

No channel files where %100 there. It's in the general GUI settings. You could throttle channel files. Now after this your able to do General availability, Early availability or pausing them.