News, tips, advice, support for Windows, Office, PCs & more. Tech help. No bull. We're community supported by donations from our Plus Members, and proud of it
Home icon Home icon Home icon Email icon RSS icon
  • Can you quantify the damage done by bad patches?

    Home Forums AskWoody blog Can you quantify the damage done by bad patches?

    Tagged: 

    • Author
      Posts
      • woody
        Da Boss

        I’ve long concentrated on explaining patches’ bugs and how to work around them. Some people think that patches in the last 20 years don’t break stuff.
        [See the full post at: Can you quantify the damage done by bad patches?]

      • #2299351 Reply
        gborn
        AskWoody_MVP

        Still remember some cases I’ve seen in the internet, where Firmware Updates from Microsoft broke their Surfaces – so users had to send the devices back to get a replace.

        https://www.google.com/search?q=surface+firmware+update+bricked+device&oq=Surface+firmware+update+bricks&aqs=chrome.1.69i57j33.11696j0j7&sourceid=chrome&ie=UTF-8

        Microsoft Windows Insider MVP, Microsoft Answers Community Moderator, Blogger, Book author

        https://www.borncity.com/win/

        1 user thanked author for this post.
      • #2299377 Reply
        R
        AskWoody Lounger

        Without any doubt. Lost assignments because of bodged updates, repushing them after restoring images, fighting with ‘broken’ hardware after updates and upgrades. It’s why we left Windows 10 and moved to Macs (Office) and Linux. At least there you still have the free choice about when to upgrade. Since then, no pissed of clients anymore and no long nights trying to control or repair the damage done.

        1 user thanked author for this post.
        • #2299379 Reply
          woody
          Da Boss

          I hear ya… but can you quantify the damage?

      • #2299435 Reply
        Alex5723
        AskWoody Plus

        If you look at damages due to ransomware my opinion is the Microsoft’s update damages are 100x+.
        We are talking $$T yearly.
        Microsoft’s updates are installed monthly while ransomware is really rare on yearly basis.

        The Cost of Ransomware

        In the past year, there has been a 235% increase in cyber threats targeting businesses, and ransomware is an increasing risk for organizations of all sizes.

        A recent report from Malwarebytes says that business ransomware attacks increased 365% from Q2 in 2018 to Q2 in 2019.

        The Cost of Ransomware

        Ransomware damages were expected to exceed $11 billion in 2019.

        Kaspersky Lab discovered the average enterprise pays more than $1.2 million per attack, with small businesses paying about $120,000. For both enterprise and small businesses, costs continue to increase year over year. Kaspersky says one successful ransomware attack can cost an organization more than $713,000. In addition to the ransom other associated costs can include:

        Data loss or damaged data
        Loss of business functions and downtime
        Loss of sales and/or production
        Cost of investigation
        Damage to brand and business reputation
        Expenses related to system repair and restoration

        COSTS RELATED TO DOWNTIME

        On average, downtime costs can exceed five to 10 times the amount of a ransomware payment. In mid-2019, the average payment was more than $36,000, with many organizations paying more than $100,000.

        To quantify ransomware downtime in terms of days, the average loss is 9.6 days….

        • This reply was modified 3 weeks, 2 days ago by Alex5723.
        1 user thanked author for this post.
        • #2299445 Reply
          woody
          Da Boss

          That’s my impression, too, but I have no hard facts to back it up.

          I’m surprised this topic hasn’t garnered more reports like “My company went down for a day” or “I had to reinstall Windows from scratch and it took me all day.”

          1 user thanked author for this post.
          • #2299807 Reply
            doriel
            AskWoody Lounger

            “My company went down for a day” or “I had to reinstall Windows from scratch and it took me all day.”

            This is very close to the reality. My last real enterprise outage was caused by Windows update on CVE-2020-1472. We had to rebuild completely new Domain Controller and we spent approx 5 hours on it (me and my two celleagues). We had to troubleshoot exchange server as well, because of that update.

            So for me the quantification is clear. Its costing human time, thus costing money since IT guys are payed for that.

            And I have several notebooks from my customers, who has problems with updating. They can quantify it clearly – the ammount of money they pay us for the time we spend on their notebooks.

            Dell Latitude E6530, Intel Core i5 @ 2.6 GHz, 4GB RAM, W10 1809 Enterprise

            HAL3000, AMD Athlon 200GE @ 3,4 GHz, 8GB RAM, Fedora 29

      • #2299441 Reply
        Alex5723
        AskWoody Plus

        I don’t think that Enterprises are allowed to disclose publicly the amount of damages caused by Microsoft’s updates under their Enterprise license agreement, hence no surveys/papers on the subject.
        Microsoft has been sued by individuals, in most cases after forced Windows 10 upgrades.

        • This reply was modified 3 weeks, 2 days ago by Alex5723.
        1 user thanked author for this post.
      • #2299449 Reply
        woody
        Da Boss

        Good tweet by Florian Roth:

        I‘d recommend a series of conversations with admin teams of big companies about problems caused by early patching & how they value availability. It’s a risk assessment: is the risk of outages or unexpected service behavior due to patching higher than the risk it tries to address

      • #2299474 Reply
        LoneWolf
        AskWoody Plus

        I’ve had patches that caused Access crashes, or incompatibility with certain spreadsheets.  Had to roll them back.

        I’ve had patches that caused Excel crashes when opening files. Had to roll them back.

        I’ve had patches that caused Outlook crashes on opening. Had to roll them back.

        I’ve had Windows 10 updates that caused multiple random systems to stop printing properly; it took Microsoft a week to resolve.

        I could also tell you of our firewall vendor releasing an update that bricked firewalls by updating GRUB and breaking it, putting the OS in rescue mode and requiring reload of the entire firewall. I could note that our vendor defaults their firewalls to auto-update, and we were lucky they caught it quickly, because I lost only two firewalls -too bad they were for the same client, who lost an entire day’s productivity.  The vendor very quietly swept that under the rug with minimal notice to customers, eroding confidence; they didn’t help by releasing an automatic emergency update that overrode automatic update preferences and updated firewalls anyway, sometimes spontaneously restarting them mid-day, without notifying customers (thankfully I didn’t end up experiencing that fiasco).

        So absolutely patches do break things and cause problems, including monetary loss. There’s just little way to hold a vendor accountable other than switch, and how do you switch when the vendor is Microsoft?

        We are SysAdmins.
        We walk in the wiring closets no others will enter.
        We stand on the bridge, and no malware may pass.
        We engage in tech support, we do not retreat.
        We live for the LAN.
        We die for the LAN.

      • #2299475 Reply
        Microfix
        AskWoody MVP

        Well this is debatable whether it was a good or bad patch!
        The now redundant (and removed) KB3035583 in conjuction with various other patches, caused untold grief for the masses. Luckily askwoodydotcom came to the rescue for us (pre-forum era)

        Win8.1 Pro | Linux Hybrids | Win7 Pro O/L | WinXP O/L
      • #2299487 Reply
        bbearren
        AskWoody MVP

        Some people think that patches in the last 20 years don’t break stuff.

        Some people know that patches in the last 20 years haven’t broken any of their stuff.

        The only way you can quantify “bad” patches is (first) as a percentage of those for whom patches have had no ill effects, and (second) is the “broken stuff” a result of a single patch, or the culmination of applying a patch to a system that was not fully patched to begin with.

        There are few difficulties posted within these forums for which I can offer corrective measures, because my first question of someone with difficulties is, “Are you completely up to date with updates/patches?” which is, for the most part, out of place at AskWoody.

        But, a total number of folks within the category you’ve lined out has no relevance unless that number is placed within the greater context of those folks who have not had any problems.  Otherwise there is no comparison.

        Create a fresh drive image before making system changes/Windows updates, in case you need to start over!
        "When you're troubleshooting, start with the simple and proceed to the complex."—M.O. Johns
        "Experience is what you get when you're looking for something else."—Sir Thomas Robert Deware

      • #2299488 Reply
        Michael Austin
        AskWoody Plus

        In essence you’re asking if someone (a company or companies) has audited the extent of Microsoft’s monetary damages aggregated from categories which include diversions of valuable staff time, consultants’ paid time, damage to software or hardware, and the infrequent but not unheard of bricking of some devices. In principle I feel that someone exists who could give you decent quantifications. Long ago I happily used PatchLink on our comparatively modest, mixed OS LAN, and there will be more current analogues. Check Point technologies and other companies which need to monitor Microsoft also come to mine. As long as such a company isn’t so frequently in bed with Microsoft that they’ll make more money from that relationship, rather than by being a mostly impartial organization, like Interpol.

        Finance, social and tech founder. My planet-wide talk show for people craving new stories by which to live is Casual Saints.

      • #2299492 Reply
        wrj
        AskWoody Lounger
      • #2299497 Reply
        geekdom
        AskWoody Plus

        You are asking about the emperor’s new clothes. Most companies are unwilling to admit to operating system vulnerabilities or software difficulties. If documented, difficulties remain an in-house report and not subject to review by outsiders.

        G{ot backup} TestBeta
        offline▸ Win10Pro 1909.18363.959 x64 i3-3220 RAM8GB HDD Firefox79.0 WindowsDefender
        online▸ Win10Pro 1909.18363.1139 x64 i5-9400 RAM16GB HDD Firefox83.0b1 WindowsDefender
        TargetReleaseVersion=1909
        WUMgr
      • #2299502 Reply
        rontpxz81
        AskWoody Lounger

        I’ll never forget the first version of Windows XP had the firewall turned off by default-

      • #2299547 Reply
        RamRod
        AskWoody Plus

        Microsoft should be able to answer this question – accurately and precisely. What do you think they do with all of the telemetry they collect? What is the main purpose of the telemetry? They know exactly how much damage their updates do – and then they measure if they should pull it back or revise it – and how fast they need to work to revise it.

        Or they could simply test it better before deploying it. They must’ve calculated over the years that ex post facto revisions to updates was more effective than extensive beta testing pre-deployment. I bet they have the data to support that. I wonder if they ever published it?

        There had to be a reason to dismiss all of the beta testers and go with telemetry. Dr. Watson on steroids.

        RamRod

        • #2299846 Reply
          doriel
          AskWoody Lounger

          Very good opinion. Telemetry should be used for this purpose. It cant evaluate money, but it can evaluate time and it can also count errors. Then we can have success percentage. But this is just raw data, how many people are not sending feedback?

          There had to be a reason to dismiss all of the beta testers

          Sure there was. To save money of course. There is nice sounding fairy tale:
          You can join Windows 10 Insider ring and get in touch with cutting-edge technology!

          The truth is:
          We need you to betatest our OS. Sorry you payed money for it, but that is the reality.

          In gamimg industry, testservers and sandboxes are mostly for free. Very often you get all ingame items unlocked, so you can test them. This seems to me as the correct way of treating customers, but this seems to be utopia/unreal with Microsoft. So unethical to me.

          Dell Latitude E6530, Intel Core i5 @ 2.6 GHz, 4GB RAM, W10 1809 Enterprise

          HAL3000, AMD Athlon 200GE @ 3,4 GHz, 8GB RAM, Fedora 29

          2 users thanked author for this post.
      • #2299553 Reply
        anonymous
        Guest

        A forced “upgrade” to Windows 10 ended up frying the BIOS on a family member’s motherboard. Ended up having to install a new one.

        1 user thanked author for this post.
      • #2299579 Reply
        Kathy Stevens
        AskWoody Plus

        Yes, I can quantify the damage done by bad patches?

        In fact, it has been nominal. I have worked for and/or run large corporations, government agencies, news organizations, and for the last 20 years my own firm.

        At no time have we experienced a significant economic loss or loss in productivity as a result of bad Windows updates or bad patches. The only major computer related economic loss and loss in productivity that I have experienced was associated with software coding during Y2K.

        Our biggest problem has been with third-party “free software” recommended on Woody’s site. As a result, we no longer use off brand software featured on the website.

        At the present time we are operating 40+ Windows 10 PCs. The machines are high-end, off the shelf Lenovo’s and more recently HP’s. The only modifications we make to our computers is to install larger drives and additional RAM. All of our new drives are made by Western Digital and RAM compatible with the specifications of the manufacturers. We made the transition to Windows 10 at year-end 2019. Prior to 2019 all of our PCs were operating with Windows 7 or its predecessors without incident.

        We update Windows 10 consistent with Woody’s MS-DEFON warnings as well as driver and bios updates when push to us by the system manufacturer.

        Our primary software includes: Acronis True Image, Adobe Acrobat, CCleaner, the manufacturers suite of software shipped with each machine, graphics software, Logitech software to support our video cameras, various versions of Microsoft Office, Firefox, our VPN’s software, drivers and software for video cards, security software, MATHLAB, and the current WordPerfect office suite of programs.

        All of our computers are automatically backed up to a second internal drive twice a day by Acronis. In addition, all of our computers are backed up to external drives that are stored off-site at the end of each business day.

        In those instances when a Windows update or a patch has a negative impact on our systems, we immediately attempt to restore the system from a restore point or, worst case scenario, an Acronis backup. Time lost in people hours and the economic cost in dollars is nominal – but the machine may be off-line for a couple hours during recovery.

        So, the damage done to our firm and my previous employers due to bad Microsoft updates and patches has been nominal.

        Public corporations are required to report to their stock holders any  event that will have a significant impact on their operations and/or profitability.

        And with over 900 million Windows 10 computers operating worldwide today there would be an absolute rebellion if Windows updates or bad patches were having a significant impact on corporate productivity and/or profitability. And such disruptions would be fully covered in the business press.

         

        1 user thanked author for this post.
        • #2299667 Reply
          anonymous
          Guest

          You’ve mitigated damage done by bad patches by knowing how to repair the damage.  Downtime in people and machines have definite costs, costs that are known and tracked.

          But, unless a machine or multiple machines are directly responsible for generating revenue or verifiably caused a quantifiable business loss, the veracity of downtime losses are questionable.  What gets reported ranges from nothing through vague to clear software caused losses.

          Don’t understand how utilities recommended in Ask Woody could have been found by enough employees to cause big problems.  Good site but I’ve never seen any utility mentioned here unique to the site; they appear in many sites.  Hard to trace back to Woody.

          What is nominal?  MS, with market cap that puts them well within the list of top 10 countries’ GDP,  definitely won’t publish accurate numbers on users’ lost time; they’re not responsible for it per EULA and they don’t seem to know what to do with telemetry.  Client companies, required or not, won’t report specifics; it would be stupid to reveal details of problems competitors can read in a Quarterly.

          CCLeaner?  Huh? Avast is a data broker, take a look at their corporate site.  Read their financials; they’re very honest about what they do.

          3 users thanked author for this post.
          • #2299856 Reply
            doriel
            AskWoody Lounger

            Little bit out of context, but look at these sentences. Even if Windows somehow manage to ruin your bussiness, you will not get a single penny. And somehow I feel that this correct, bacause otherwise there would be thousands of people blaming Microsoft for ruining their bussinesses. The worst thing for me is the “unreliability” of this operating system and constantly changing envirinment with no obvious roadmap/ goal.

            Windows 10 EULA:
            “The device manufacturer or installer warrants that properly licensed software will perform substantially as described in any Microsoft materials that accompany the software. If you obtain updates or supplements directly from Microsoft during the 90-day term of this limited warranty, Microsoft provides this limited warranty for them.”

            -90 day limited warranty

            And another one:
            Except for any repair, replacement, or refund the manufacturer or installer, or Microsoft, may provide, you may not under this limited warranty, under any other part of this agreement, or under any theory recover any damages or other remedy, including lost profits or direct, consequential, special, indirect, or incidental damages.

            Dell Latitude E6530, Intel Core i5 @ 2.6 GHz, 4GB RAM, W10 1809 Enterprise

            HAL3000, AMD Athlon 200GE @ 3,4 GHz, 8GB RAM, Fedora 29

            2 users thanked author for this post.
      • #2299574 Reply
        anonymous
        Guest

        One of the earlier upgrades of Win10 took ouit two of my most used programs; Firefox and Thunderbird. It also deleted all of my folders and files. It took all of my backup drives and $370 at the repair shop to get files and folders back and had to reinstall the programs. One of my other non-MS doesn’t work any longer. It is also a graphic program like Adobe. It seems Microsoft doesn’t want any competition for their bloated win10 system. I’m not a computer graduate soi Microsoft should be fixing problems not creating them, and they seem to want those users to do that job too. I bought a cheap Chromebook hoping that will do what I need to do, rather than spend hours figuring out how to fix win10 issues.

      • #2299657 Reply
        F A Kramer
        AskWoody Plus

        As a home user with modest requirements, I can not recall any patches causing major damage except for the recent HP Omen Computer “problem” which repeatedly bombed my computer. But I am not sure that was caused by a patch or some other sort of update and whether HP or MS was responsible.

        What I can say is that I have had repeated instances of updates causing my screen colors and arrangements to change dramatically (update to 2004 is the latest to do this), my file explorer settings to change, my wireless network to fail, and my Internet browser to change its setup, home page, and even which browser was the default. While none of these cost me real money to repair, they did cost me much time and effort to correct the damage.

        1 user thanked author for this post.
      • #2299705 Reply
        anonymous
        Guest

        Since my introduction to windows 3, I have no way of calculating the number of man hours put in reinstalling, installing and troubleshooting issues caused by msft updates/upgrades/patches etc etc etc. But in light of the fact my hrly wage was north of $75 and time is money, I should guess msft owes me at least 4 first steps of children, uncounted athletic events my children and grandchildren were participants in, family reunions(not to be confused w/ PC help) hrs and hrs on telephone both conversing and on hold, miles driven to replace damaged hardware, gasoline, auto repairs, shipping and handling, searching online,- ad nauseum….I would say at least 46,9567,898 USD + tax.

        I am obviously not a business user BUT my time is as valuable as anyone elses- including Billy gates’.

        PS Not to mention University classes mucked up by shoddy coding by MSFT.

      • #2299739 Reply
        anonymous
        Guest

        I am surprised to see that there are some people that think that Windows Patches are perfect

        but well,   There’s a reason why the Msdefcon system exist here and why most people should pay attention to it….

        2 users thanked author for this post.
        • #2299758 Reply
          Kathy Stevens
          AskWoody Plus

          No, Windows patches are not perfect.

          But the question on the table is, “Can you quantify the damage done by bad patches?”

          Within the corporate environments where I have worked, Windows updates and patches have occasionally brought down a PC. But because of the ability to recover through the use of restore points or backups the economic/productivity damage caused by Windows has been nominal.

      • #2299766 Reply
        Alex5723
        AskWoody Plus

        they did cost me much time and effort to correct the damage.

        If you value your time by 1 hour = $500 and add the effort…how much it “didn’t cost you in real money” ?

        • #2299824 Reply
          F A Kramer
          AskWoody Plus

          By “real money” I mean cash out of my pocket to pay someone to repair the damage, or for the replacement of a computer. For that the answer is none. In “virtual money”, it is hard to say. Could I have been earning real money during the time spent undoing MS “revisions”? Perhaps, but even in my dreams, not $500/hour. Not even $50. But whether the expense is in real money or not, the need to undo MS damage should have been unnecessary and I think that is the point most of us are making.

      • #2299896 Reply
        Kathy Stevens
        AskWoody Plus

        The question posed by Woody at the beginning of this thread is, “Have any patches in the past five years caused you or your company “considerable damage”?”

        In our case it has not.

        We have developed and put into place a written emergency recovery program for our firm. Its chapters include, but are not limited to, addressing power failures and power surges, a fire, computer related issues, storm related damage, pandemic response, etc.

        The computer section includes hardware redundancy, data integrity and preservation, data and system recovery, an inventory of spare components including hard drives and monitors, etc.

        As a result, if a computer goes down we can pull a backup, fully configured, unit off the shelf, plug it in, recover the data files from the damaged PC, and be back to work in minutes.

        As a result of our compliance with our emergency recovery program we have not experienced “considerable damage” from bad patches in the last five years.

        2 users thanked author for this post.
        • #2300036 Reply
          doriel
          AskWoody Lounger

          In our company we are protected against data storm for example, but against patches? We have WSUS 🙂

          We have developed and put into place a written emergency recovery program for our firm.

          This enterprise risk management should be standard, good job. But what about home users? 0% has power backup source. Lets say that 10% of average users understand the principle of backing up whole partitions. Are they able to set up backing software correctly? Of course not, and why should they? They just want to use their computer. They have no intention in this patching madness.

          And home users are the MOST affected, cause they do not backup regullary. These users backup just their data – movies, photos, documents. So their recovery does not take 10 minutes, but they struggle and go to their PC guru and it takes days and money to get their PC back.

          Dell Latitude E6530, Intel Core i5 @ 2.6 GHz, 4GB RAM, W10 1809 Enterprise

          HAL3000, AMD Athlon 200GE @ 3,4 GHz, 8GB RAM, Fedora 29

          1 user thanked author for this post.
      • #2299930 Reply
        bbearren
        AskWoody MVP

        The computer section includes hardware redundancy, data integrity and preservation, data and system recovery, an inventory of spare components including hard drives and monitors, etc. As a result, if a computer goes down we can pull a backup, fully configured, unit off the shelf, plug it in, recover the data files from the damaged PC, and be back to work in minutes.

        For me, this is the bigger question.  Why isn’t everyone doing the above in some proportion relevant to their needs and resources?  I have a spare 2.5″ SSD, mSATA SSD on standby for my daily driver desktop, and three months worth of protected drive images at the ready.  I have a formatted 4TB HDD on standby to swap into my NAS should one of its four drives fail.

        I’m an inveterate tinkerer and regularly plumb Windows innards.  I can spend a good deal of time moving and breaking things to the point that Windows no longer boots, but it only takes 6 minutes to completely restore with a recent drive image.  It only requires imaging software and an external storage drive for that level of protection.

        If an ounce of prevention is worth a pound of cure, a pound of prevention can be invaluable.

        Create a fresh drive image before making system changes/Windows updates, in case you need to start over!
        "When you're troubleshooting, start with the simple and proceed to the complex."—M.O. Johns
        "Experience is what you get when you're looking for something else."—Sir Thomas Robert Deware

        1 user thanked author for this post.
      • #2300120 Reply
        Kathy Stevens
        AskWoody Plus

        One closing thought.

        No, patches have not caused considerable quantifiable economic damage to our company in the past five years.

        But there is the intangible cost of having to manage Windows 10 since migrating our systems from Windows 7 at the beginning of the year.

        Time spent following the literature to find out what are the potential problems associated with patches and updates, when is it safe to patch and update, what patches to avoid, etc.

        Time better spent dealing with our clients and doing analysis.

        1 user thanked author for this post.
    Viewing 23 reply threads

    Please follow the -Lounge Rules- no personal attacks, no swearing, and politics/religion are relegated to the Rants forum.

    Reply To: Can you quantify the damage done by bad patches?

    You can use BBCodes to format your content.
    Your account can't use Advanced BBCodes, they will be stripped before saving.