Is it OK to run patches on 500+ VMs?

Just saw this message from ME:

I haven‘t approved updates since 12/2017 for our infrastructure with 500+ VMs.

I‘m not new to that topic but your team recently wrote that it is not wise to approve updates when your on patch level 12/2017. I think it was in march. Since then i didn‘t found a topic if to update or not. All thoughts was about if and how to update one single machine. Is there anything related to my problems to read from you?

Susan Bradley does a great Job but it would be interesting to have a algorithm how to patch when you’re on 12/2017 or similar. Its not something i ask you to do but in those times Microsoft does a horrible job which leads to spectacular ransom attacks in the future. I patch servers for 3 years now – i‘m definitely not a pro but why do i feel like Microsoft always tries to shoot our infrastructure into pieces. :/

Best regards, and thank you and your team for the great work.

Since Susan Bradley joined AskWoody several months ago, we have something of a dichotomy. On the one hand, we have people who just want to know when it’s safe to patch their individual (home or business) PCs. On the other hand, we have a widening group of admins who are in charge of hundreds — thousands — of machines.

As you’ve seen, the expectations and needs of those two groups is related, but still quite different in many respects. More than that, there’s a spectrum of needs — from folks who’d rather be playing mahjong, to folks who have to be concerned about protecting key corporate data.

One size doesn’t fit all. What’s evolved is kind of a dual system that’s grown out of my background helping individuals and Susan’s long background working with organizations.

The MS-DEFCON system is geared for people who really just want to get the furshlugginer thing working. I don’t even try to differentiate between a Win7 system running Office 2010

and a Win10 1803 system running Office 365. There are just too many variables. What I give with MS-DEFCON is a red light/green light system, with warnings about particularly irksome problems.

The Patch Lady recommendations (and her unique, lengthy Master Patch List) are designed for people who want — or need — to take a closer look at the patches.

The Patch Lady approach is a scalpel. The MS-DEFCON approach is a sledge hammer.

That doesn’t answer your question. But it should help you put into perspective the comments that are bound to come from people who have experienced your exact situation.

Should I patch now or wait?

Just got a good question from reader IB:

Woody, I know you’re a busy guy. Thanks for all you do…
I have a basic question. I have ONE home computer, win7 pro, 64 bit (whatever that means)
I use GWX control panel, and have ZERO desire to update to win 10. Heck, I just got rid of XP less than a year ago! My windows updates are set to “let me choose” etc.

As per your advise, I always install DEFENDER updates immediately.

But my confusion is about “important” updates: I think you have told us that anything with “security” in the title needs to be installed. (I currently have a couple security for “.NET FRAMEWORK” and about 10 for WIN7 64bit…”). But in response to a comment within the last week (I forget which thread) about security updates, you said, “wait”.

So my question…when I see ‘security’ updates…should I install them immediately, or wait till DEFCON says it’s time?

Thank you.

I say, always wait for the MS-DEFCON rating to come down, then follow the specific instructions I give when the number goes down.

There’s a reason for waiting. Many problems with Microsoft’s patches don’t appear in the first few days, or even the first few weeks, in some cases. For Win7 and 8.1, Microsoft pulls and re-issues the really bad ones. For Win10, well, we’re still not sure exactly what Microsoft will do. In all cases, within a few weeks we have a pretty good idea of what’s going to clobber systems, and what’s benign.

Not infallible, mind you, but reasonably accurate.

You also need to keep in mind that very, very few patches cover holes that are being exploited. There are fixes for zero-day problems, but most zero-days these days are directed at very specific targets – government installations, military, financial institutions and the like. For you and I, zero-days are rarely a concern.

Take a look at the latest SANS Internet Storm Center list. See the column marked “Known Exploits”? There aren’t any known exploits for any of the patches (although there’s one that had a published exploit later). Even when an exploit “Proof of Concept” is published, it takes weeks or months or years for the exploit to become a problem for you and me.

If you’re carrying a hundred thousand Social Security numbers, or storing nuclear launch codes, it’s a different problem, of course. But for the vast number of people, the vast majority of the time, waiting for patches to show their fangs is a very good idea.

You need to patch sooner or later. Yes. Definitely. But you don’t need to dance to Microsoft’s tune.

What does MS-DEFCON cover?

Another good one, this time from reader MA:

Greetings Woody,

I am in IT, and we got a single Windows update pushed to our machines today: KB 3095649.

One of our laptops is getting stuck on boot up after the encryption log in screen. The user said they rebooted after applying Windows updates.

They said they also shut down their machine every day, meaning it should have already had all of these latest updates.

My question is: Your defcon system, how updated do you keep it? I see it’s at level 2 currently. Does that have anything to do with this latest update? Or is it more about all of the October updates?

Anyway, thanks for your site and your time. I really appreciate it.

Thanks much, MA

I started the MS-DEFCON system a decade ago as a short, easy way to give a “go/no-go” decision on Windows and Office patches. Times have really changed since then, but the goal’s the same. I don’t claim to give advice to admins who are in charge of large numbers of machines — Susan Bradley’s patchmanagement.org does a far better job, and her constantly-updated list of available patches and their problems takes each KB to task.

I still try, very hard, to give a simple “go” signal when one is warranted. Occasionally (and, recently, more frequently) I have to list exceptions – specific KB’s that are spoiling the lot. But the basic idea is to wait until the patches have settled down, then give a green light.

A large majority of the people who follow the MS-DEFCON system are still using Win7, but many switched to Win8.1, and an increasing number use Win10. Each has different patching needs and screw-ups.

In the particular case that you mention – four patches released earlier this week – I was at MS-DEFCON 2 when they hit, and I stayed at MS-DEFCON 2. That’s not so much a reflection of known problems with the patches. It’s more of a wait-and-see attitude, in case one of the patches has problems, or someone discovers that one of the patches includes junk that nobody wants, like GWX updates, snooping platform patches, and so on.

It’s very rare that a really, really important security patch comes down the chute and needs to be installed immediately. People working in sensitive industries need to take those zero-days seriously because they’re often used in various kinds of attacks. But for normal, everyday people (and companies!), widespread attacks aren’t common in the days and weeks following a patch.

I’ll continue to aim the MS-DEFCON rating at people who don’t want to be dealing with every-blooody-patch as it rolls out. I’ll continue to cover Win7 and 8.1 in depth, Win10 as best I can (Microsoft’s updating policy makes that hard), and Vista a little bit.

I hope that answers your question! Comments always most welcome…

MS-DEFCON as red light/green light

I just got a very good message from SW that says:

Hi Woody,

I really appreciate your guidance on Windows updates and share it with many friends. I do have one suggestion that would be of value to me and likely others. The current status is Defcon 2. I know it refers to October because I have addressed, with your guidance, previous months. Someone visiting your sight for the first time or one who has not visited for a while might be confused. If you could state which updates are covered by the current Defcon it would be helpful.

Thank you very much for your guidance through the Microsoft minefields.

Here’s my reply:

I try to keep the MS-DEFCON level very, very, very simple. It’s basically a red light/green light (with occasional yellow) sort of warning system: When the light turns green, you should install all outstanding patches, no matter when they were first offered.

I’ve long thought that the best way to do it is to list each patch, one by one, and make recommendations – but that’s considerably more complicated than most users would like. (Susan Bradley does an excellent job of patch-by-patch analysis.)

Your approach would group patches by month — and a lot of people would find that too complicated.

So I’ll stick with red light/green light – and continue to recommend Susan’s analysis for people who want to look at specific patches, or groups of patches…

People who manage patches for a living — admins are the salt of the earth! — generally can’t afford to run their companies on a red light/green light system like MS-DEFCON. But most home users, and even small businesses, can benefit from some very generalized advice, methinks.

Other opinions most welcome, of course, in the comments.