Woody Leonhard's no-bull news, tips and help for Windows, Office and more… Please disable your ad blocker – our (polite!) ads help keep AskWoody going!
Home icon Home icon Home icon Email icon RSS icon
  • Is it OK to run patches on 500+ VMs?

    Posted on June 11th, 2018 at 13:40 woody Comment on the AskWoody Lounge

    Just saw this message from ME:

    I haven‘t approved updates since 12/2017 for our infrastructure with 500+ VMs.

    I‘m not new to that topic but your team recently wrote that it is not wise to approve updates when your on patch level 12/2017. I think it was in march. Since then i didn‘t found a topic if to update or not. All thoughts was about if and how to update one single machine. Is there anything related to my problems to read from you?

    Susan Bradley does a great Job but it would be interesting to have a algorithm how to patch when you’re on 12/2017 or similar. Its not something i ask you to do but in those times Microsoft does a horrible job which leads to spectacular ransom attacks in the future. I patch servers for 3 years now – i‘m definitely not a pro but why do i feel like Microsoft always tries to shoot our infrastructure into pieces. :/

    Best regards, and thank you and your team for the great work.

    Since Susan Bradley joined AskWoody several months ago, we have something of a dichotomy. On the one hand, we have people who just want to know when it’s safe to patch their individual (home or business) PCs. On the other hand, we have a widening group of admins who are in charge of hundreds — thousands — of machines.

    As you’ve seen, the expectations and needs of those two groups is related, but still quite different in many respects. More than that, there’s a spectrum of needs — from folks who’d rather be playing mahjong, to folks who have to be concerned about protecting key corporate data.

    One size doesn’t fit all. What’s evolved is kind of a dual system that’s grown out of my background helping individuals and Susan’s long background working with organizations.

    The MS-DEFCON system is geared for people who really just want to get the furshlugginer thing working. I don’t even try to differentiate between a Win7 system running Office 2010

    and a Win10 1803 system running Office 365. There are just too many variables. What I give with MS-DEFCON is a red light/green light system, with warnings about particularly irksome problems.

    The Patch Lady recommendations (and her unique, lengthy Master Patch List) are designed for people who want — or need — to take a closer look at the patches.

    The Patch Lady approach is a scalpel. The MS-DEFCON approach is a sledge hammer.

    That doesn’t answer your question. But it should help you put into perspective the comments that are bound to come from people who have experienced your exact situation.

    If that helped, take a second to support AskWoody on Patreon

    Home Forums Is it OK to run patches on 500+ VMs?

    This topic contains 18 replies, has 12 voices, and was last updated by  anonymous 1 month, 3 weeks ago.

    • Author
      Posts
    • #197257 Reply

      woody
      Da Boss

      Just saw this message from ME: I haven‘t approved updates since 12/2017 for our infrastructure with 500+ VMs. I‘m not new to that topic but your team
      [See the full post at: Is it OK to run patches on 500+ VMs?]

      5 users thanked author for this post.
    • #197259 Reply

      ltorres
      AskWoody Lounger

      What I have done in this occasions is to approve them in batches.

      I like to go by year, in that way you can see what you are approving (don’t want to deploy .net 4.7 to web servers if that is not your intention)

      Also you can monitor what systems are running low in space in that way you don’t stop exchange from working due to lack of space.

      take this time to go as far back as you can to see if patches from 2015 are applicable (u might have 2 systems)

      When I do catch-up on clients I do it one batch a week (200 servers and 3000 WKS on average) aprox… that way is easier to troubleshoot if things happen.

       

    • #197260 Reply

      Susan Bradley
      AskWoody MVP

      What operating systems?

      1. Have canaries.  Machines in your office where the users are savvy and use sample representations of your software.  They get it first.  They report back to you if anything broke.

      2.  Have a rule that all patches get installed by X date.  Honey if you haven’t installed anything since December, we need to talk.  I go no more than 30 days.  I do not install on day one.  Or even day five.  Or day 10.  At the earliest it’s day eleven (on the second weekend after patch Tuesday).

      3.  Standardize on machines/hardware.  One thing the 10 era has documented – one thing the Spectre/Meltdown has showcased is that we need to include bios and drivers in our patching sequences.  A lot of issues are cleared up with newer drivers.

      4.  I’m on the prior months updates by the time we enter the new month’s patching sequence.  So right now May updates are installed.  If you can’t install the prior months updates right now — what specific patching side effect are you worried about?

      Email me/private message me and let’s discuss as to why you are not comfortable and what options we can do to get you comfortable.  As 500 machines with no updates since December scares me more than updating through May.  And that’s the key …when I get to the point that I’m more scared about being unpatched than I am about patching that’s the point where I’m ready to patch.

      Susan Bradley Patch Lady

      6 users thanked author for this post.
      • #197261 Reply

        anonymous

        I think there’s something to be said for caution.  May is looking good and is what I’ll be moving my servers up to (my desktops are all on May already) from December 2017.  Before April, December 2017 was the last time patching was stable enough to be considered good.  January’s a good example with its TotalMeltdown exploit that took until March for it to be addressed, only to be released in a slew of patches that also broke virtual and Intel NICs.

        It might be a taste of things to come when outdated machines are more secure than bleeding-edge systems, especially as Win10/Server2016 becomes a service.

        3 users thanked author for this post.
      • #197297 Reply

        Bill C.
        AskWoody Lounger

        To Patchlady Susan.

        This is a great reply, and actually quite applicable even though I am not an admin or corporate IT person overseeing servers and dozens of machines, to the individual user.  As a home user with a few machines (plus some friends seeking advice) I found your first and second points are very applicable to the home user, especially if using a desktop.

        On Point #1.  My canary is my main machine, but that is in reality a second line canary.  The first line is actually the MVPs and participants in AskWoody.com and your patchlist, which I have used on and off for a number of years, before you were here.  My main machine must remain operational, so I use the month after patch Tuesday to monitor what might be potential pain points.  Since a lot of participants here are now including hardware, OS and Office info I can get a feel towards what is happening.  If similar hardware configurations or hardware generations are not having issues, I am more confident.  The MVPs and others also know many more sources of information and share knowledge far more than my 2 eyes and time are able to gather.  I pay special attention to reports of desktop woes as laptops are much more complex and many times use proprietary hardware or OEM manufacturer software tools that it is much harder to discern trends.

        On Point #2:  I will move forward.  I do not want to play the catchup game over months.  I prefer to age my patching over a few days and watch for the reported issues (as well as other anomalies), so I am not guessing which one may have been problematic.  Fortunately, I have yet to encounter a month where the various data points leave me without a path forward.  Even at the worst, of January-May 2018, I would do a backup, create a restore point, do an image, and then install the last of the potentially problematic patchs.  I was determined not to roll back in the April mess.  I prefer to proceed and possible fix rather than stall.  I have been lucky (knocking on wood).

        I also prioritize the patches based upon the potential vulnerability, and whether the exploit is in the wild.  As a result, as a Group B person, the IE Rollup is first even though I do not use IE, followed by Office.

        Point #3 is not a challenge for me as an individual, but it is a valid concern when dealing with both desktops and laptops.  At least I have standardized on Windows and Office versions (by choice and not design).

        With all this though, I do recognize I have to move forward as some of my hardware is going to remain vulnerable (Spectre/Meltdown) and Windows 10 is not going to allow user control or privacy settings that matter.  That challenges for me is between Fruit or Birds, but until then, thanks for all you do, and it is helpful to individuals also.

      • #197341 Reply

        Noel Carboni
        AskWoody MVP

        …when I get to the point that I’m more scared about being unpatched than I am about patching that’s the point where I’m ready to patch.

        That’s a great summary. Bravo, Susan.

        And we know that knowledge eases fear of the unknown.

        -Noel

        2 users thanked author for this post.
      • #197512 Reply

        mbhelwig
        AskWoody Lounger

        What do I do.

        I do not have the luxury of a canary.

        I have two computers at home –
        My own and
        My wife’s Dell Inspiron 15R 5520 laptop.– January and February updates caused this computer to take 8 times longer to boot up than without the updates.

        I addition to these I look after six computers where I do two days a week voluntry work. All computers have i5 processors (Haswell, Ivy bridge, or Sandy Bridge processors).
        I built them, softwared them up and maintain them.
        We also have a server running MS homesrver 2011 (now past EOL and does not get any updates from MS)

        All updates done up to end of December 2017.

        Windows Updates turned off —
        Image of computers done with Macrium Reflect as of end of December.2017
        Allow Reg key has also been installed by Avast antivirus on all machines.

        Note — the six machines I maintain as a volunteer all have fixed IP addresses set in each machine so the whole network is effected by the March April, May, and now June Updates.
        as per their explaination below.
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        There is an issue with Windows and third-party software that is related to a missing file (oem<number>.inf).
        Because of this issue, after you apply this update, the network interface controller will stop working.

        1    To locate the network device, launch devmgmt.msc; it may appear under Other Devices.
        2   To automatically rediscover the NIC and install drivers, select Scan for Hardware Changes from the Action menu.

        a. Alternatively, install the drivers for the network device by right-clicking the device and choosing Update.
        Then choose Search automatically for updated driver software or Browse my computer for driver software.”

        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        I do not see why I should have to risk breaking a complete fully functioning network and then have to rebuild NIC drivers again.
        The disruption is not necessary without a whole lot more explaination from Microsoft as to why they are doing all of this.
        I have the same brand of motherboard in each computer but different CPU,s and different Network chips.

        As I understand it the update should FIX problems without creating more. It is a matter of getting it right before releasing the update.

        To the poster with the 500 computers — My sympathy — I am upset enough with just 6 computer with no updates for 6 months.

        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        mbhelwig

        2 users thanked author for this post.
      • #199122 Reply

        anonymous

        So did you abandon KB4088878 / KB4088875 or did you approve them?

        https://www.askwoody.com/patch-list-master/master-patch-list-2018-03/

    • #197267 Reply

      Seff
      AskWoody Lounger

      I’m intrigued by this dichotomy.

      On the one hand, there are pure technical amateurs like myself here, just trying to look after our own home (or perhaps even small business) machines, in some cases for family or friends, and benefiting enormously from the unpaid support provided here by Woody and his team of experts, and for which (I trust) we make voluntary contributions to keep the site running. On the other hand, there are apparently admins responsible for looking after hundreds or even thousands of business machines, who seemingly rely on the same unpaid support. If they’re responsible for that number of machines shouldn’t they either be IT experts themselves or else employing same?

      In all honesty, I don’t see how a volunteer-run site like this one can be expected to provide advice to professional computer administrators, and I hope that it doesn’t gravitate away from its original purpose (as I understand it) namely to offer informal and unpaid advice on a largely volunteer and reader-led basis to a small and often amateur audience, without commitment or liability. I don’t believe that it was ever intended to provide more than that, but correct me if I’m wrong!

      I don’t say that in a way that is intended to criticise those large-scale professionals who rely on the advice given here, it’s just that I’m not sure why or how this site could be expected to meet the requirements of that particular audience. There is also the professional indemnity aspect to consider – if a professional administrator relies on free advice from here to keep hundreds or thousands of business computers running, what happens if and when things go wrong? Surely those administrators would want a contractual arrangement with professional IT experts to fall back on in such circumstances rather than telling the shareholders that they based their patching procedures on the word of “some guy on the internet”? Maybe the site needs to make a prominent disclaimer or two!

      On that basis, however, the site seems to me to be doing a great job as it is, and one that has only been enhanced by Susan Bradley’s arrival. I see her clearance of individual updates as being subject to the wider clearance of the month’s whole updating process as provided by the MS-DefCon ratings – and that’s fine by me, I don’t see any conflict or contradiction in that.

      1 user thanked author for this post.
      • #197289 Reply

        anonymous

        I’m IT in a mid-sized company.  I think I’m fairly typical when it comes to the position, namely I work multiple positions that would be full-fledged jobs in any larger organization.  I handle desktop updates, server updates, asset management, user support, some user training, programming, application support and application management with more on the way.  I don’t have days to go over the finer details of patches as they come out.  Usually, at least.  The Spectre and Meltdown kerfuffle ate up quite a bit of my time.

        I use AskWoody as a quick guide of whether I should yay or nay the patch.  Depending on the issues that come out I’ll dive in in order to better understand what’s breaking and whether or not we can expect fallout if we do decide to go ahead.  Before I used AskWoody, I used to use sbsdiva (Susan Bradley’s old site) as well as a half-dozen other blogs to better understand where patch issues were being seen.  Back then I’d install all patches but the ones identified and then if something broke, I’d repeal the one that was causing the issue.  For actually determining what caused the error I’d normally search TechNet and Microsoft’s patch notes.

        AskWoody has essentially replaced most of the blog crawling I used to do.  When errors occur, I use the links provided by Woody, but in general I still do my own research to determine whether or not I can justify installing a buggy patch.  The onus falls on me to verify the information given and make the risk assessment.  This isn’t something that the site can do for us.  Likewise I’m not paying the site for this service and I wouldn’t consider it liable for any of our security concerns.  There’s no contracts and no statement of intent of the site provide free services outside news and advice or opinion.  Any donations I make are just that and do not imply any form of liability on the site itself or the persons involved.

        This is a long and drawn out way of saying I still have a job to do.  I can’t rely on AskWoody or any other site to do my job for me.  The buck stops with me.

        5 users thanked author for this post.
      • #197915 Reply

        anonymous

        Seff wrote:

        On the one hand, there are pure technical amateurs like myself here
        ,,,
        On the other hand, there are apparently admins

        Yep, and I’m pretty sure Woody likes it that way.

        IT folks were hitting this board long before Susan arrived, and undoubtedly many more followed her here. And that’s a good thing. A board with too many amateurs and too few alpha-geeks would be much less useful, helpful, informative, etc. So big thanks to the IT folks for the information they provide, the advice they give, and the questions they help answer here.


        Note:
        Before you ask, no, can’t explain the message from the guy with the unpatched 500+ VMs. In a world of 7+ billion people, there are bound to be outliers. But I wouldn’t make the mistake of trying to draw meaningful conclusions re the technical chops of most IT folks by extrapolating off a single data point.

    • #197274 Reply

      zero2dash
      AskWoody Lounger

      To ME:
      I’ll throw my hat somewhat into Susan’s ring…as she said, you need to have a pilot group (or two, three, or four). Anyone in the tech dept. gets patched first/second/third (depending on # of pilot groups). You should also have different update groups for hours to patch…space it out otherwise your WAN (and/or LAN if behind SCCM/WSUS) traffic is going to slow to a crawl.

      At my old job (as an SCCM/SUS admin) we had desktop pilots and server pilots. Pilot desktop was IT support tiers 1-3, pilot server was test/lab environment. Pilots were pushed out 2 weeks after release, unless there were problems. Prod server/live desktop patches were pushed out 2 weeks later, or 1 week after pilot patches applied.

      If you have 500 machines not patched since December, I highly suggest you stagger it, prod vs non-prod, and do pilots. And then keep that schedule. You should at this point probably jump to March or April at least. In my 2nd to last position (different company, no SCCM/SUS), we generally stayed 1 month behind on patches on both prod and test. You don’t necessarily have to jump straight to May 2018, but, March or April (whichever month they finally fixed all the stuff they broke with the Smeltdown patches).

      I would also recommend patching prod systems case by case… for instance, leave app or SQL servers behind lesser non-critical stuff like file and print servers. I would patch DC’s first, assuming you have backups (but that’s another can of worms I hope you don’t have to worry about). Then do file/print servers, then app/SQL.

      To Woody:
      Even with Susan’s exceptional info, I still follow the DEFCON, as it’s easier to pay attention to (and I prefer the sledgehammer myself). The positions/jobs I’ve had that I didn’t follow the DEFCON (or was not the one who called the shots), the updates were generally done close to (if not at) the DEFCON rating change.

      2 users thanked author for this post.
    • #197277 Reply

      anonymous

      The first three months this year were a confusing mess. Ask Woody (the site) was helpful in supporting many diverse approaches to the administrator of many clients or end user during the confusion. So it is understandable to believe that holding at 12/2017 was endorsed here rather than recognized as an option. It wasn’t, but it was discussed in great detail and the volume of information could have lead to that interpretation. So thinking that way is possible.

      But even the broadest tool in the garden shed, the sledgehammer MS-DEFCON has twice advised to move ahead for all instances that may have been on hold. The more recent publication in Computerworld was introduced on June 2nd on this site here and the Computerworld article followed by the next day. A similarly prominent all-clear was sounded for the April updates in early May.

      It has been a mess, and difficulties have been documented elsewhere, referenced and discussed here. The suggestion is that Ask Woody was the source for the delay. But directions to move ahead with updates have been ignored. The Unofficial designation of volunteer regiment Group W has a legion or two. The details contributed from that contingent are helpful. But it is not an approach suggested by Woody’s articles, and the directions he gives does not endorse that system.

      Patch Lady has extended a gracious invitation to @ME for assessment and reinterpretation of goals. The repair effort may have some bumps, but it will be some of the strongest advice available.

      The mess was created elsewhere, and Microsoft was certainly a contributor with incomplete patches. Ask Woody (the site) is among the best resources for sorting the mess into appropriate piles. But the smelly work of dealing with the mess is on each of us. Unfortunate that @ME missed a couple of signals along the way, good that Woody (Da Boss) highlighted the outcome. It can be helpful to the dozens of others who are in the same situation, but have not shared.

      2 users thanked author for this post.
    • #197279 Reply

      anonymous

      Simple. Patch only what you need. You don’t need others’ input on what to patch and when. Do your homework! READ THE KB ARTICLES.

      • #197363 Reply

        Noel Carboni
        AskWoody MVP

        The documentation is more sparse than ever (a whole page that says nothing more than “this resolves issues in Windows”)… There are fewer words than ever describing these things, and a larger percentage of the words than ever are just boilerplate…

        The experience of others, when possible summarized by experts, is more valuable now than ever.

        That being said… The decisions and responsibility are yours. And sometimes the testing.

        For example, I scoured the internet for hard information on what the Spectre/Meltdown mitigations do to performance, and in the end NO ONE had measured it at the level I required. I had to actually do the updates myself to learn that it’s bad – and more importantly, just how bad for my uses.

        I’ve mulled over why…

        My needs and priorities are different than those of others. My hardware and choices of software are different than those of others. And I guess I am relatively unaffected by marketing hype, and I may be amongst only a few who consistently prefer a computer over a phone or tablet. I can’t believe it could just be because I care how well my computer systems run where others don’t.

        Yet here we are, with Intel’s/Microsoft’s patches part of every set of cumulative update available for every system.

        Even with the mitigations disabled (e.g. by InSpectre) up-to-date computer systems are just not as efficient as those running Windows patched to December’s level.

        -Noel

        5 users thanked author for this post.
    • #197292 Reply

      OscarCP
      AskWoody Lounger

      As Woody explains in the Home page, the DEFCOM is a blunt instrument. And one that, I believe, is strongly determined, these days, by when the inevitable torrent of both Windows 10  and Rollup issues starts to dwindle towards trickle status.

      For my part, being Group B, I tend to follow closely the Patch Lady’s Master Patch List, although I am in charge of only one machine, my very own old Win 7 PC, and not of hundreds.

      Because her entries on the patches du jour are binary: either they say that there are issues already reported, or “Not at this time”. Whether it is just for one, or for hundreds of machines, that there are some “issues reported”, whatever they might be, means, to even a non-techie like myself, that it’s better to “wait a bit, until this gets clarified, to see if it also applies to me.” And invariably, sooner or later, clarified it is. Issues reported so far: “Not at this time”? Well, if the “non-bad” news have remained unchanged for two or three weeks after the last Patch Tuesday, and I have not gathered, elsewhere, that there might be actual problems in spite of what it says there, then I go ahead, do my Group B “Geronimo!” stunt and patch away all those “Not at this time” updates — and all those with issues that do not apply to me.

      And so far, after months of doing this,  I still walk among the living and so does, metaphorically speaking, my Windows 7 Pro, SP1, x64, Intel I-7 “sandy bridge”, 11-year old PC… A fool’s luck, no doubt.

       

      • This reply was modified 2 months ago by  OscarCP.
      • This reply was modified 2 months ago by  OscarCP.
    • #197312 Reply

      PerthMike
      AskWoody Lounger

      I released January to April in one hit last month after patches seemed to be stable.

      The main problem seems to be that machines then showed up all over the place on which patches they needed.

      I use Security Only via WSUS, the quality rollups are denied.

      However, when trying to patch for the last four months, some workstations would just leave out one or two of the months randomly. January always seemed to get included, but then February and March showed up as being needed by about 25-30% of the servers (randomly), and just about all of them wanted April.

      Now, the machines are reporting they don’t need any more patches, even when 2018-02 and/or 2018-03 were never applied. This is on a combination of servers running 2008R2, 2012 and 2012R2.

      Even when I then let the servers scan against Microsoft’s update servers the extra patches don’t show up as needed.

      No matter where you go, there you are.

      • #197418 Reply

        anonymous

        Some of the earlier security-only might’ve been fully superseded by the April/May updates as they fix some of the NIC issues that were introduced in March’s security-only and the individual fix released mid-March.

        Basically they could break an April/May patched system.

    • #197605 Reply

      geekdom
      AskWoody Lounger

      “The Patch Lady approach is a scalpel. The MS-DEFCON approach is a sledge hammer.”

      The patch that generated dismay was the Carnac patch issued three weeks after the initial patch with instructions to uninstall the initial patch, install the Carnac patch, then install the initial patch. Sledge hammer or scalpel would have made no difference. Blowtorch, anyone?

      Group G{ot backup} Win7|64-bit|SP1|TestBeta

      • This reply was modified 2 months ago by  geekdom.

    Please follow the -Lounge Rules- no personal attacks, no swearing, and politics/religion are relegated to the Rants forum.

    Reply To: Is it OK to run patches on 500+ VMs?

    You can use BBCodes to format your content.
    Your account can't use Advanced BBCodes, they will be stripped before saving.

    Your information: