MS-DEFCON 1: Business patchers be on alert

Topic

New Reply

ISSUE 19.02.1 • 2022-01-12 By Susan Bradley For those running a network with a domain controller, the side effects this month are extreme. Don’t patch
[See the full post at: MS-DEFCON 1: Business patchers be on alert]

Susan Bradley Patch Lady

2 users thanked author for this post.

abbodi86, IT Manager Geek

Reply | Quote

Replies

On my Server 2016 box, KB5009546 has already downloaded and is ‘Updates are ready to install’. How do I stop this?

Reply | Quote

Stop the windows update service.  Delete the software distribution folder.

https://www.windowscentral.com/how-clear-softwaredistribution-folder-windows-10

Susan Bradley Patch Lady

2 users thanked author for this post.

oldfry, WSpauldelke

Reply | Quote

First, don’t press Install Now on the Windows Update screen.

Second, if you don’t use WSUS, set the Windows Update policies to download only and notify to install:

Go to your GPOs, create a policy for your servers, using their names for example, and set Configure Windows Updates to option 3. This is located in Computer Configuration -> Policies -> Administrative Templates -> Windows Components -> Windows Update. Then enforce the policy by enabling it. That’s how I am set so that I have a backup first, then schedule install and reboot.

Hope it helps.

Reply | Quote

This couldn’t come in the worst possible time, due to the CVE-2022-21907 what is marked as wormable. Those in Windows Server 2019 and later are now left between a rock and a hard place.

Microsoft never ceases to amaze me.

Reply | Quote

There’s a mitigation workaround.

https://isc.sans.edu/forums/diary/A+Quick+CVE202221907+FAQ+work+in+progress/28234/

For Windows Server 2019 and Windows 10 Version 1809, the “HKLM:\System\CurrentControlSet\Services\HTTP\Parameter\EnableTrailerSupport” is set to 0 by default disabling trailers. You can check this registry value in Powershell (thanks, Rob)l:

Get-ItemProperty  “HKLM:\System\CurrentControlSet\Services\HTTP\Parameters” | Select-Object EnableTrailerSupport

Susan Bradley Patch Lady

1 user thanked author for this post.

nelsonhd87

Reply | Quote

Thank you very much.

Reply | Quote

https://twitter.com/wdormann/status/1480972462812770305

P.s. smart people on twitter are still trying to decipher the release notes

Susan Bradley Patch Lady

1 user thanked author for this post.

Fred

Reply | Quote

The mitigation was “clarified” sometime today:

“This mitigation only applies to Windows Server 2019 and Windows 10, version 1809 and does not apply to the Windows 21H1 and newer.”

HTTP Protocol Stack Remote Code Execution Vulnerability
CVE-2022-21907

Windows 11 Pro version 22H2 build 22621.1778 + Microsoft 365 + Edge

1 user thanked author for this post.

Fred

Reply | Quote

If the Server 2012 R2 or 2016 or 2019 are set to download the update but set for manual install can’t you simply wait until Microsoft fixes the update and the system will download the fixed update automatically without you having to do anything further until the ALL CLEAR is given then you can proceed to manually install the update ? Thanks for any input.

 

Reply | Quote

“download but do not install” puts them in the software distribution folder. So no, you would need to clear that out first.

Susan Bradley Patch Lady

Reply | Quote

You could also let another cumulative update override the previous update or download an MSU file with the update and use WUSA to patch the system without installing the bad update. Just ignore the update and wait for the next one.

Reply | Quote

You can wait for a better update and ignore the one displayed on Windows Update (Windows it will notify you many times) or download a cumulative MSU that replaces the bad update and install that.

Ignore and wait it’s the best thing I can do right now. Fortunately I’m running Windows Server 2016 so the CVE-2022-21907 vulnerability shouldn’t be a concern.

Reply | Quote

Interesting. These appear to be similar to the patches released in Nov 2021 for four other AD security vulnerabilities . Are these two patches for new vulnerabilities, or the ones already “fixed” in November?

Reply | Quote

New vulnerabilities.

Susan Bradley Patch Lady

Reply | Quote

Hi.

patched my hyper-v host server running 2016 – no issues as yet with host. No reboot loops all patches went on ok.

fired up 3 vm devices on the above host (2016 ad dc, 2016 rras server, 2016 sccm/mecm server version 2111).

rras server was patched first, Went through without issue at all.

dc fired up, same patch stopped at 95% wouldn’t go on, so Went to windows catalogue and got patch 1.2gb approx, installed manually fine. Dc patched, no reboots or issues seen thus far.

sccm/mecm single box patched fine, seems to be working.

got 2 more 2016 servers, and 1 2019 server on hyper-v host i can patch to test. Will snapshot first, as i have played it loose with snapshots so far.

this any use to anyone?

seems 2016 isnt effected from what i am seeing? Anyone seen different?

 

Reply | Quote

Or there is something third party on those servers that are impacted?

Susan Bradley Patch Lady

Reply | Quote

November Patch Tuesday took down one DC, where removing the updates did not restore stability…constant reboots every 15-60 minutes.  Had to decommission and delete.  Luckily, it stayed up long enough to finish the process.

January Patch Tuesday started the same crash/boot cycle in 2012r2 DC.  Not as often, but still crashed 5-6 times yesterday.

Anecdotally, I stayed RDP’d to this for long periods of time.  It would not crash while I was logged on.  But within minutes of logoff, it restarted.  Maybe a bit of Heisenberg uncertainty, or wishful thinking?

Reply | Quote

Several DC’s running 2012 R2, 2016, and 2019 are fine.

One 2012 R2 Core (no desktop exp) running as a hypervisor host got hit.

Disconnected network cable and logged in on a local machine admin account and uninstalled KB5009624 and all is well with the core again.

~ Group "Weekend" ~

Reply | Quote

New: I put off doing updates on a few more 2016 Servers until today – and it appears that the offending updates are NOT being offered now.  Microsoft may have pulled them.

~ Group "Weekend" ~

1 user thanked author for this post.

Microfix

Reply | Quote

UPDATE: MSFT has now pulled kb5009624 for Windows Server 2012 R2, kb5009557 for Windows Server 2019 and kb5009555 for Windows Server 2022, due to bugs… Hyper-V not starting, DC bootloops and inaccessible ReFS volumes displaying RAW filesystems.
Still available in the Update Catalog, go figure..

Keeping IT Lean, Clean and Mean!

3 users thanked author for this post.

ChrisAVWood, arbrich, Alex5723

Reply | Quote

Let’s hope that Microsoft fix their mess ASAP before a PoC for CVE-2022-21907 appears on the net.

Reply | Quote

More info on CVE202221907
https://isc.sans.edu/forums/diary/A+Quick+CVE202221907+FAQ/28234/
no exploitation yet..infocon will go Yellow should an exploit be in-the-wild
Worth keeping the bookmark

Keeping IT Lean, Clean and Mean!

Reply | Quote

Hopefully they mark them as expired in WSUS.  I just checked and they are still being offered to my server via WSUS.  I did a sync and no changes.

Reply | Quote

Curious that Windows Server 2016 was not affected.

Reply | Quote

Unless I’m missing something. For the consumer (non-business)  user this issue doesn’t apply, correct?

Reply | Quote

Correct.  As I said in the post, I’ve installed patches on a home PC with no issues. These are all enterprise/business problems.

Susan Bradley Patch Lady

2 users thanked author for this post.

Fred, sheldon

Reply | Quote

P.S. that’s not to say I’m ready for consumers to install updates, just that in my personal testing on my home laptop, it survived.

Susan Bradley Patch Lady

4 users thanked author for this post.

SueW, Chris B, Fred, sheldon

Reply | Quote

Apparently there are reports that VPN is not working on machines after January updates. Can anyone confirm. I have removed the January 2022 update KB5009566 from my experimental computer running windows 11 home.

PS I do not use VPN so it does not effect me directly.

mbhelwig

Reply | Quote

VPN problems are happening to users using L2TP with the update. Removing it should bring connectivity back to normal.

Reply | Quote

Thank you. I will wait for MS to sort it out.

mbhelwig

Reply | Quote

https://www.bleepingcomputer.com/news/microsoft/microsoft-pulls-new-windows-server-updates-due-to-critical-bugs/#cid22230
Lawrence Abrams reporting that the pulled patches are back again (comments post)..uncertain whether anything has changed to patches.

Keeping IT Lean, Clean and Mean!

Reply | Quote

If the MSUs in Microsoft Catalog are the same, then, most likely, no changes.

If someone downloaded the MSU of the 2022-12 update for one OS before they were pulled from Windows Update and then download the update again for the same OS on the catalog, you could figure it out by calculating the SHA256 hash of both files to see if they are the same.

Reply | Quote

I downloaded it for 2016 server. But this is not effected, my sql 2017 box is running on 2019 server this patched ok.

not sure why 2016 seems to be un effected, as the updates are cumulative, and you’d think if it was fixed in 2016 it would carry over into 2019 and server 2022. As the server code base moves on.

makes me think these 3 yearly releases aren’t always as fixed or secure as some previous release.

Reply | Quote

Given that in the case of the domain controller issue, it appears that people configuring servers with more secure settings is triggering the boot loop rather than the patch itself, this is honestly expected.

Susan Bradley Patch Lady

Reply | Quote

Should we patch Servers (2012 R2, 2016, 2019) that are Stand Alone and NOT Domain Controllers or should we wait for the ALL Clear ??

For Domain Controllers I get more confused the more I read and have NOT patched either of the Domain Controllers (2012R2) that I handle and I am looking for Guidance on exactly how to do this without having to worry about getting caught in a loop and having to run to customers far away and try to figure out how to back things out.

I do all my updates remotely. I simply have whatever Microsoft has downloaded sitting in Que waiting to be installed and I am looking for step by step guidance on how to update for January.

Thanks for any responses.

Reply | Quote

YMMV.

Standalone servers

We have not seen any re-boot issues with non-DC servers.  We have let many of them patch via WSUS or Microsoft Updates in their normal cycle.

Domain Controllers

We have had 2/3 in reboot loops, so stopped automated updates.  Now that we have updates to the updates, we need to:

1. switch to manual updates
2. remove what was already downloaded
3. download appropriate manual installer
4. install manually

Switch to manual updates:

• admin-level command prompt
• SCONFIG
• #5) Windows Update Settings: Manual
• #15) Exit to Command Line

Remove what was already downloaded

1. net stop wuauserv
2. rd /s /q %systemroot%\SoftwareDistribution\Download
3. [you don’t need to start Windows Update Service, because you set it to manual]

download appropriate manual installer

1. list of out-of-band updates here.  These should take you to the Microsoft Update Catalog page.
2. Download appropriate MSU file for your OS.  For 2012r2, the file is huge (1.6GB)
3. copy to a folder on your DC like c:\install\[copy name from file description]

Install manually

1. admin-level command line interface
2. use autocomplete to make your life easier
3. be sure to confirm what you are seeing autocomplete before you hit the enter key:

   cd c:\install\2022[TAB][ENTER]
   [TAB][ENTER]

4. The results should look like this:

Microsoft Windows [Version x.x.xxxx]
(c) 2013 Microsoft Corporation. All rights reserved.

C:\Windows\system32>cd "c:\install\2022-01 Update for Windows Server 2012 R2 for x64-based Systems (KB5010794)"

c:\INSTALL\2022-01 Update for Windows Server 2012 R2 for x64-based Systems (KB5010794)>windows8.1-kb5010794-x64_3b350eb34833c9a3ac71d973db3fd8ae86a6b220.msu

 

Reply | Quote

The link you provided is for the out of band for November’s Kerberos issue (we’ve had a tad too many and it’s hard to keep up)

https://www.askwoody.com/2022/various-out-of-band-updates-out-to-fix-january-patch-issues/

Be aware that the 2012 R2 server patch is a security only and is not cumulative.

For those that are Plus members (and remember, a mere $1 in the donate jar gives you access) they are on the Master patch listing https://www.askwoody.com/patch-list-master/ – look in the latest Excel/CSV/PDF/HTML versions of the listing dated January 24 – I posted up early in case anyone needs to deploy this weekend.

Susan Bradley Patch Lady

Reply | Quote

Sorry about that old link.  I had it bookmarked for earlier DC problems after Patch Tuesday.

I don’t want to send people away from this site with links elsewhere if that’s not kosher.

We can summarize to ‘download the appropriate MSU for your OS’.

I’ll definitely take a look at the master list from your link.

Reply | Quote

Makes no sense to me that they do not simply pull the bad cumulative patch (or any bad patches for that matter) and replace with the fixed one especially for people who have not installed it yet instead of all this manual work. How inefficient !!! Also be nice if they do some testing once in a while. Microsoft has turned updating into a Nightmare from the depth’s of He”ll”. We are worse off now then we were years ago.

Guess I am getting OLD……..

Sorry for the rant – I have been Time Burgled with this !!!

 

 

 

Reply | Quote