• MS-DEFCON 1: Business patchers be on alert

    Home » Forums » Newsletter and Homepage topics » MS-DEFCON 1: Business patchers be on alert

    • This topic has 39 replies, 13 voices, and was last updated 4 months ago.
    Author
    Topic
    #2415645

    ISSUE 19.02.1 • 2022-01-12 By Susan Bradley For those running a network with a domain controller, the side effects this month are extreme. Don’t patch
    [See the full post at: MS-DEFCON 1: Business patchers be on alert]

    Susan Bradley Patch Lady

    2 users thanked author for this post.
    Viewing 12 reply threads
    Author
    Replies
    • #2415682

      On my Server 2016 box, KB5009546 has already downloaded and is ‘Updates are ready to install’. How do I stop this?

      • #2415711

        Stop the windows update service.  Delete the software distribution folder.

        https://www.windowscentral.com/how-clear-softwaredistribution-folder-windows-10

        Susan Bradley Patch Lady

        2 users thanked author for this post.
      • #2415717

        First, don’t press Install Now on the Windows Update screen.

        Second, if you don’t use WSUS, set the Windows Update policies to download only and notify to install:

        Go to your GPOs, create a policy for your servers, using their names for example, and set Configure Windows Updates to option 3. This is located in Computer Configuration -> Policies -> Administrative Templates -> Windows Components -> Windows Update. Then enforce the policy by enabling it. That’s how I am set so that I have a backup first, then schedule install and reboot.

        Hope it helps.

    • #2415706

      This couldn’t come in the worst possible time, due to the CVE-2022-21907 what is marked as wormable. Those in Windows Server 2019 and later are now left between a rock and a hard place.

      Microsoft never ceases to amaze me.

    • #2415871

      If the Server 2012 R2 or 2016 or 2019 are set to download the update but set for manual install can’t you simply wait until Microsoft fixes the update and the system will download the fixed update automatically without you having to do anything further until the ALL CLEAR is given then you can proceed to manually install the update ? Thanks for any input.

       

      • #2415894

        “download but do not install” puts them in the software distribution folder. So no, you would need to clear that out first.

        Susan Bradley Patch Lady

        • #2415964

          You could also let another cumulative update override the previous update or download an MSU file with the update and use WUSA to patch the system without installing the bad update. Just ignore the update and wait for the next one.

      • #2415969

        You can wait for a better update and ignore the one displayed on Windows Update (Windows it will notify you many times) or download a cumulative MSU that replaces the bad update and install that.

        Ignore and wait it’s the best thing I can do right now. Fortunately I’m running Windows Server 2016 so the CVE-2022-21907 vulnerability shouldn’t be a concern.

    • #2415888

      Interesting. These appear to be similar to the patches released in Nov 2021 for four other AD security vulnerabilities . Are these two patches for new vulnerabilities, or the ones already “fixed” in November?

    • #2416132

      Hi.

      patched my hyper-v host server running 2016 – no issues as yet with host. No reboot loops all patches went on ok.

      fired up 3 vm devices on the above host (2016 ad dc, 2016 rras server, 2016 sccm/mecm server version 2111).

      rras server was patched first, Went through without issue at all.

      dc fired up, same patch stopped at 95% wouldn’t go on, so Went to windows catalogue and got patch 1.2gb approx, installed manually fine. Dc patched, no reboots or issues seen thus far.

      sccm/mecm single box patched fine, seems to be working.

      got 2 more 2016 servers, and 1 2019 server on hyper-v host i can patch to test. Will snapshot first, as i have played it loose with snapshots so far.

      this any use to anyone?

      seems 2016 isnt effected from what i am seeing? Anyone seen different?

       

      • #2416467

        Or there is something third party on those servers that are impacted?

        Susan Bradley Patch Lady

    • #2418065

      November Patch Tuesday took down one DC, where removing the updates did not restore stability…constant reboots every 15-60 minutes.  Had to decommission and delete.  Luckily, it stayed up long enough to finish the process.

      January Patch Tuesday started the same crash/boot cycle in 2012r2 DC.  Not as often, but still crashed 5-6 times yesterday.

      Anecdotally, I stayed RDP’d to this for long periods of time.  It would not crash while I was logged on.  But within minutes of logoff, it restarted.  Maybe a bit of Heisenberg uncertainty, or wishful thinking?

    • #2418082

      Several DC’s running 2012 R2, 2016, and 2019 are fine.

      One 2012 R2 Core (no desktop exp) running as a hypervisor host got hit.

      Disconnected network cable and logged in on a local machine admin account and uninstalled KB5009624 and all is well with the core again.

      ~ Group "Weekend" ~

    • #2418083

      New: I put off doing updates on a few more 2016 Servers until today – and it appears that the offending updates are NOT being offered now.  Microsoft may have pulled them.

      ~ Group "Weekend" ~

      1 user thanked author for this post.
    • #2418086

      UPDATE: MSFT has now pulled kb5009624 for Windows Server 2012 R2, kb5009557 for Windows Server 2019 and kb5009555 for Windows Server 2022, due to bugs… Hyper-V not starting, DC bootloops and inaccessible ReFS volumes displaying RAW filesystems.
      Still available in the Update Catalog, go figure..

      "-rw-rw-rw-" extreme computing
      3 users thanked author for this post.
    • #2418111

      Unless I’m missing something. For the consumer (non-business)  user this issue doesn’t apply, correct?

      • #2418115

        Correct.  As I said in the post, I’ve installed patches on a home PC with no issues. These are all enterprise/business problems.

        Susan Bradley Patch Lady

        2 users thanked author for this post.
        • #2418117

          P.S. that’s not to say I’m ready for consumers to install updates, just that in my personal testing on my home laptop, it survived.

          Susan Bradley Patch Lady

          4 users thanked author for this post.
    • #2418275

      Apparently there are reports that VPN is not working on machines after January updates. Can anyone confirm. I have removed the January 2022 update KB5009566 from my experimental computer running windows 11 home.

      PS I do not use VPN so it does not effect me directly.

      mbhelwig

      • #2418281

        VPN problems are happening to users using L2TP with the update. Removing it should bring connectivity back to normal.

        • #2418306

          Thank you. I will wait for MS to sort it out.

          mbhelwig

    • #2418282

      https://www.bleepingcomputer.com/news/microsoft/microsoft-pulls-new-windows-server-updates-due-to-critical-bugs/#cid22230
      Lawrence Abrams reporting that the pulled patches are back again (comments post)..uncertain whether anything has changed to patches.

      "-rw-rw-rw-" extreme computing
      • #2418301

        If the MSUs in Microsoft Catalog are the same, then, most likely, no changes.

        If someone downloaded the MSU of the 2022-12 update for one OS before they were pulled from Windows Update and then download the update again for the same OS on the catalog, you could figure it out by calculating the SHA256 hash of both files to see if they are the same.

        • #2418308

          I downloaded it for 2016 server. But this is not effected, my sql 2017 box is running on 2019 server this patched ok.

          not sure why 2016 seems to be un effected, as the updates are cumulative, and you’d think if it was fixed in 2016 it would carry over into 2019 and server 2022. As the server code base moves on.

          makes me think these 3 yearly releases aren’t always as fixed or secure as some previous release.

      • #2418314

        Given that in the case of the domain controller issue, it appears that people configuring servers with more secure settings is triggering the boot loop rather than the patch itself, this is honestly expected.

        Susan Bradley Patch Lady

    • #2419659

      Should we patch Servers (2012 R2, 2016, 2019) that are Stand Alone and NOT Domain Controllers or should we wait for the ALL Clear ??

      For Domain Controllers I get more confused the more I read and have NOT patched either of the Domain Controllers (2012R2) that I handle and I am looking for Guidance on exactly how to do this without having to worry about getting caught in a loop and having to run to customers far away and try to figure out how to back things out.

      I do all my updates remotely. I simply have whatever Microsoft has downloaded sitting in Que waiting to be installed and I am looking for step by step guidance on how to update for January.

      Thanks for any responses.

      • #2419693

        YMMV.

        Standalone servers

        We have not seen any re-boot issues with non-DC servers.  We have let many of them patch via WSUS or Microsoft Updates in their normal cycle.

        Domain Controllers

        We have had 2/3 in reboot loops, so stopped automated updates.  Now that we have updates to the updates, we need to:

        1. switch to manual updates
        2. remove what was already downloaded
        3. download appropriate manual installer
        4. install manually

        Switch to manual updates:

        • admin-level command prompt
        • SCONFIG
        • #5) Windows Update Settings: Manual
        • #15) Exit to Command Line

        Remove what was already downloaded

        1. net stop wuauserv
        2. rd /s /q %systemroot%\SoftwareDistribution\Download
        3. [you don’t need to start Windows Update Service, because you set it to manual]

        download appropriate manual installer

        1. list of out-of-band updates here.  These should take you to the Microsoft Update Catalog page.
        2. Download appropriate MSU file for your OS.  For 2012r2, the file is huge (1.6GB)
        3. copy to a folder on your DC like c:\install\[copy name from file description]

        Install manually

        1. admin-level command line interface
        2. use autocomplete to make your life easier
        3. be sure to confirm what you are seeing autocomplete before you hit the enter key:
          cd c:\install\2022[TAB][ENTER]
          [TAB][ENTER]
        4. The results should look like this:
        Microsoft Windows [Version x.x.xxxx]
        (c) 2013 Microsoft Corporation. All rights reserved.
        
        C:\Windows\system32>cd "c:\install\2022-01 Update for Windows Server 2012 R2 for x64-based Systems (KB5010794)"
        
        c:\INSTALL\2022-01 Update for Windows Server 2012 R2 for x64-based Systems (KB5010794)>windows8.1-kb5010794-x64_3b350eb34833c9a3ac71d973db3fd8ae86a6b220.msu

         

        • #2419700

          The link you provided is for the out of band for November’s Kerberos issue (we’ve had a tad too many and it’s hard to keep up)

          https://www.askwoody.com/2022/various-out-of-band-updates-out-to-fix-january-patch-issues/

          Be aware that the 2012 R2 server patch is a security only and is not cumulative.

          For those that are Plus members (and remember, a mere $1 in the donate jar gives you access) they are on the Master patch listing https://www.askwoody.com/patch-list-master/ – look in the latest Excel/CSV/PDF/HTML versions of the listing dated January 24 – I posted up early in case anyone needs to deploy this weekend.

          Susan Bradley Patch Lady

        • #2419713

          Sorry about that old link.  I had it bookmarked for earlier DC problems after Patch Tuesday.

          I don’t want to send people away from this site with links elsewhere if that’s not kosher.

          We can summarize to ‘download the appropriate MSU for your OS’.

          I’ll definitely take a look at the master list from your link.

        • #2420919

          Makes no sense to me that they do not simply pull the bad cumulative patch (or any bad patches for that matter) and replace with the fixed one especially for people who have not installed it yet instead of all this manual work. How inefficient !!! Also be nice if they do some testing once in a while. Microsoft has turned updating into a Nightmare from the depth’s of He”ll”. We are worse off now then we were years ago.

          Guess I am getting OLD……..

          Sorry for the rant – I have been Time Burgled with this !!!

           

           

           

    Viewing 12 reply threads
    Reply To: Reply #2418193 in MS-DEFCON 1: Business patchers be on alert

    You can use BBCodes to format your content.
    Your account can't use all available BBCodes, they will be stripped before saving.

    Cancel