Woody Leonhard's no-bull news, tips and help for Windows, Office and more… Please disable your ad blocker – our (polite!) ads help keep AskWoody going!
Home icon Home icon Home icon Email icon RSS icon
  • Microsoft says it’s improving patch quality for a complex ecosystem

    Home Forums AskWoody blog Microsoft says it’s improving patch quality for a complex ecosystem

    Tagged: 

    This topic contains 48 replies, has 28 voices, and was last updated by  anonymous 1 month ago.

    • Author
      Posts
    • #232686 Reply

      woody
      Da Boss

      Michael Fortin just posted this on the Windows blog: While our measurements of quality show improving trends on aggregate for each successive Windows
      [See the full post at: Microsoft says it’s improving patch quality for a complex ecosystem]

      7 users thanked author for this post.
    • #232704 Reply

      GreatAndPowerfulTech
      AskWoody Lounger

      People need to read the 1954 book “How to Lie with Statistics” to recognize that metrics can be selected to make just about anything look better than it actually is.

      GreatAndPowerfulTech

      8 users thanked author for this post.
      • #232750 Reply

        samak
        AskWoody Lounger

        This book is still a classic and MS’s technique with this graph is straight from Chapter 5.

        Doesn’t the graph look good, the incidents appear to be almost at zero currently, right? Wrong! The incidents are around 0.5k but since the graph starts at 0.4k it only has a short column representing 0.1k. If it was drawn honestly the columns on the right would be 5 times the size they are.

        More MS lies and BS.

        W7 SP1 Home Premium 64-bit, Office 2010, Group B, non-techie

        8 users thanked author for this post.
        • #232776 Reply

          Chronocidal Guy
          AskWoody Lounger

          This book is still a classic and MS’s technique with this graph is straight from Chapter 5. Doesn’t the graph look good, the incidents appear to be almost at zero currently, right? Wrong! The incidents are around 0.5k but since the graph starts at 0.4k it only has a short column representing 0.1k. If it was drawn honestly the columns on the right would be 5 times the size they are. More MS lies and BS.

          I went on a bit of a statistical excursion using that chart.  Note, I have no actual data other than that chart graphic, but it’s fairly easy to estimate the numbers used, so I reconstructed the chart to play some number games.  DISCLAIMER: I have no actual data for any of this, just a visual estimate of the original chart, and other published statistics.

          Here’s an estimate of original chart, rescaled to use 0 as the baseline.

          RescaledChart1

          Not all that different really, it just shows a slowly decreasing trend.

          But wait.  What units are they even using here?  “Hundreds per millions of devices” is about as clear as mud.  I suspect that they actually meant “per hundreds of millions of devices,” but just on the off chance they’re trying to obfuscate the issue that way on purpose, I’m going to run with that worst case.

          “Hundreds per millions of devices” means we need to multiply that entire chart by 100 to get the number of reports per million devices.

          RescaledChart2

          Not looking so rosy now.  Those are really big numbers.  But in terms of percentages, I feel like the numbers might not be that far off, due to the trends at release.  Would a user complaint rate of 10-15% seem reasonable during the GWX nonsense?

          Now, consider all the announcements made concerning how many millions of devices Windows 10 is running.  To get a running total over the whole life of Windows 10, I took the dates of the assorted “Windows 10 now on X million devices!” headlines, and linearly interpolated between them, running backward to an initial estimate of 100 million at release, because it fit the overall trend.  (If anyone has hard numbers for this data, please correct me as you see fit!)

          RescaledChart3

          In all fairness, it does seem pretty reasonable to normalize the data by number of devices, due to the ever-increasing rollout pace.  But from a device-centric perspective, what’s the grand total of incidents reported?

          RescaledChart4

          No more decreasing trendline.  That total is approaching 5% of all Windows 10 devices.  Honestly though, that doesn’t sound all that remarkable from a product with this wide a userbase.  It’s still a huge number, but 5% sounds like it would be comparable to the defect rate people might expect in a product.  Of course, this doesn’t account for things like severity of the error reported, or repeat occurrences, or whether the issue was ever resolved.  But it’s interesting to guess at what the numbers behind Microsoft’s statistical misdirection really are.

          Note, I would disregard that first six months entirely, since it’s based on very flimsy estimates of Windows 10’s initial rollout.

          If anyone has corrections, observations, or complaints, I’d love to hear them.  The language behind the “hundreds per millions of devices” issue in particular could potentially remove a lot of zeroes from those totals, so I wouldn’t call this breakdown more than a WAG at best.

           

          Attachments:
          You must be logged in to view attached files.
          10 users thanked author for this post.
          • #232794 Reply

            anonymous

            While I applaud your efforts to reinterpret the graph, it doesn’t  require that level of commitment to know that MS is a Master of Obfuscation.

          • #232813 Reply

            anonymous

            I read many years ago that up to five percent failure rate for a electronic product was (maybe still is) an acceptable incurred loss. Is that their measurement of success?

          • #232843 Reply

            lurks about
            AskWoody Lounger

            Thanks for the effort and proving the blather was marketing ahem. I would say for an OS a reported error rate of 5% is way too high; that is 1 in 20 installs. Also, I would expect the errors to be underreported, even the most serious ones. So the real rate is probably well north of 5% and is anyone’s guess. Your analysis shows that whatever MS is using for QA it is a dismal failure.

            1 user thanked author for this post.
            • #233073 Reply

              anonymous

              Severity is what matters.  There’s many times where an OS with an error (or defect) can still run normally in most circumstances, while hardware would just fail.  Due to the complexity, software in general will have more defects but they can usually run with them and remove them at a later date.  Defect rate is higher than 5%

              Once something manufactured is shipped, it’s shipped.  It can’t be changed and will generally wear out given time.  5% “failure” rate is just that, failures not defects.  Manufactured goods generally will have defects in them that do not cause failure during their intended lifespan, however they may be what causes the device to fail after its guaranteed or expected lifespan.

    • #232707 Reply

      anonymous

      You know the term “work-around?” Microsoft has invented something new: the “talk-around.” This announcement is just another, obfuscating, over-complicating talk-around, making it seem as though the simple and obvious solution–rehire and adequately fund a well-resourced quality-control/beta-test department–wasn’t right there in front of them all the time.

      4 users thanked author for this post.
      • #232837 Reply

        Noel Carboni
        AskWoody MVP

        How to use up a company’s reputation, built over a lifetime: Say things, instead of doing things. It’s cheaper! People will take a Long Time to let go of belief.

        -Noel

        3 users thanked author for this post.
    • #232714 Reply

      anonymous

      “We shifted the responsibility for base functional testing to our development teams in order to attempt to deliver higher quality code from the start, but often fell short as exemplified by the numerous issues both business and end users have experienced including, but not limited to, data loss.” 
      There, fixed THAT sentence.

      2 users thanked author for this post.
    • #232724 Reply

      Mr. Natural
      AskWoody Lounger

      The very first sentence tries to minimize a huge problem among a few other serious problems noted with the 1809 release. My eyes started glassing over as soon as I read that.

      In paragraph 3 he mentioned all the variables Microsoft has to deal with. It certainly is a very difficult situation to manage and deal with. It’s true that there’s bound to be some systems affected negatively by an update when dealing with that kind of volume. With all those variables to deal with maybe Microsoft should focus more on the issues at hand instead of this constant push for new features that most people have no interest in.

      Looking at the “quality improvement” chart doesn’t really tell the history of how patch quality has steadily declined over time leading to much more serious issues resulting from the more recent updates.

      We already know Microsoft has ignored warnings from whatever testers they have now. I would like to know how testing now differs from how it was managed before WaaS. I’ll be interested to read the promised forthcoming post giving more details on the current “testing process”.

       

      2 users thanked author for this post.
    • #232729 Reply

      anonymous

      Unbelievable… this chart is misleading to the extreme.

      For accurate figures, note how the vertical scale stops at “0.4k”. So in your head add about the vertical size of Dec ’15 in the chart to each of the bars. That paints a different picture, no? It would be nice if someone created the alternative chart based at zero to paint the true picture.

      Let us also assume that “0.4k” means “400 per million devices” and not “400 hundreds (40000) per million devices”, which would be correct under a strict interpretation of the chart’s scale description. I refuse to believe they would be that misleading…

      Also note that in absolute numbers, we’re still talking ~400 per million devices for 700 million W10 devices, i.e. 280k incident reports in last October alone!

      And the customer satisfaction alluded to in the article can equally be explained by many people who are so disappointed that they seek ways to avoid updates altogether, instead of providing feedback on their marvellous experience. “You recently scored us 0/10. Please take a moment to give us your feedback again.” Uh-huh…

      2 users thanked author for this post.
    • #232738 Reply

      glenznet
      AskWoody Lounger

      This is nothing but a crock and a bunch of excuses for MS nerds that have lost touch with their customer base – especially the corporate customer. I am sick of Microsoft making excuses rather than listening and making the changes necessary for stability. I don’t give a d*** how pretty a window looks – just that it consistently works even after patching

      2 users thanked author for this post.
    • #232743 Reply

      geekdom
      AskWoody Lounger

      quality
      complex
      ecosystem
      validate
      trends
      approach
      critical
      attention
      care
      mission
      empower
      scale
      dynamic
      key
      achieve
      spectrum
      goal
      indicator
      stabilization
      evaluate

      Were any buzzwords missed?

      Group G{ot backup} Win7 · x64 · SP1 · i3-3220 · TestBeta
      3 users thanked author for this post.
      • #232768 Reply

        MikeFromMarkham
        AskWoody Lounger

        They missed “truth”, “accuracy” and “honesty” to name at least three… but then those aren’t terms with which the current Microsoft regime seem to be familiar.

        4 users thanked author for this post.
      • #232812 Reply

        woody
        Da Boss

        Seamless

        Going forward

        🙂

        2 users thanked author for this post.
      • #232815 Reply

        anonymous

        progress

        change

        1 user thanked author for this post.
    • #232746 Reply

      JCCWsusser
      AskWoody Lounger

      Looks to me like that chart shows a declining acceptance of each new release.

      2 users thanked author for this post.
      • #233053 Reply

        anonymous

        My thoughts exactly.  You can’t have any reported issues if you have no customers.  Very smart of them.

    • #232755 Reply

      Mr. Natural
      AskWoody Lounger

      I think Mr. Fortin’s post would make an excellent thesis for his PHD at Baghdad Bob University.  🙂

      3 users thanked author for this post.
      • #232846 Reply

        lurks about
        AskWoody Lounger

        MS market drones wouldn’t be hired by Honest Abe’s Used Cars because Abe has some scruples.

        1 user thanked author for this post.
    • #232756 Reply

      zero2dash
      AskWoody Lounger

      “While our measurements of quality show improving trends on aggregate for each successive Windows 10 release, if a single customer experiences an issue with any of our updates, we take it seriously…

      Yet your actions and words over the last several years show the complete opposite.

      In fact, I’d say that history indicates that the only time you take anything seriously is when the proverbial poo hits the fan and the media chastises you for it. 1809 is arguably the first time I think I’ve seen actual, real change from your company in probably a decade.

      Susan Bradley’s experiment and subsequent proof showed to you months ago that things needed to change, yet you were too busy to do anything other than send several canned indifferent responses.

      You have grown apathetic, and it clearly shows. So here we are.

      8 users thanked author for this post.
      • #232839 Reply

        Noel Carboni
        AskWoody MVP

        It’s just all the more “say it” instead of “do it”, saving much money.

        -Noel

        1 user thanked author for this post.
    • #232757 Reply

      Steve D.
      AskWoody Lounger

      “experience”. “Experience” or “experiences” was used 14 times in that post. A couple of years ago a highly-publicized marketing study reported that millenials preferred spending their money on experiences rather than possessions. By that they meant things like travel, culture, etc. Not operating systems. Someone should tell Microsoft.

      2 users thanked author for this post.
    • #232753 Reply

      anonymous

      Maybe, just maybe…

      1703-1709 (even 1607) is more because people don’t want upgrade into a unstable build.

      1809 is almost zero because no-one-install-it, or at least, no one want that.

      …and…

      1803 is the people who jumped into the “new stable version”

      perhaps?…

    • #232789 Reply

      RetiredGeek
      AskWoody MVP

      Hey Y’all,

      Here’s the ones that really had me saying “are they really saying that?”.

      “Self-host” means that employees working on Windows run the latest internal versions on their machines to ensure they are living with Windows.

      The “aggressive” part refers to the tenacious push to make sure local teams run their own builds and pursue any issues found.

      Really, they’re developing fixes on new builds that haven’t been integrated with all the teams builds REALLY!

      We obsess over these metrics as we strive to improve product quality, comparing current quality levels across a variety of metrics to historical trends and digging into any anomaly.

      Maybe, just maybe, they should be obsessing over the quality of the software and not the metrics that are so obviously failing to deliver trustworthy patches and updates.

      Just my two cents worth.

      May the Forces of good computing be with you!

      RG

      PowerShell & VBA Rule!
      Computer Specs

      4 users thanked author for this post.
      • #232814 Reply

        woody
        Da Boss

        Sounds like they came up with a new definition of “dog food” — a staple of the Windows testing regime since Windows 3.1 (if not earlier).

        2 users thanked author for this post.
        • #232818 Reply

          anonymous

          Yes, but you do not ever read the text “New Look! Now Lower Quality” on the label.

          • #233064 Reply

            Mark
            AskWoody Lounger

            Yes, and even lesser noticed after the “New Look…!” is the other change along with that is the size change from 24 oz. to 18 oz.

            But hey, it is a “New Look”!

        • #233066 Reply

          Mark
          AskWoody Lounger

          Woody, it definitely was earlier…I remember running Windows 286 and Windows 386 (the precursors to Windows 3.1).  Those versions are what put the infamous BSOD on the map.

          1 user thanked author for this post.
    • #232793 Reply

      ax kramer
      AskWoody Lounger

      I bet they also believe in the Tooth Fairy!

    • #232836 Reply

      Noel Carboni
      AskWoody MVP

      Seems to me that graph describes declining expectations.

      A social engineering success for Microsoft!

      -Noel

      1 user thanked author for this post.
    • #232849 Reply

      The Surfing Pensioner
      AskWoody Lounger

      I was intrigued by the expression “complex ecosystem” in this context. I looked up “complex ecosystem”:-

      “An ecosystem is a complex system composed of organisms living in a given habitat. Plants and animals are the biotic components of the ecosystem, while the subsoil, water, air, light, temperature, the climate, rains are part of the abiotic components.” (http://www.eniscuola.net/en/argomento/what-is-an-ecosystem/terrestrial-biomes/the-ecosystem-a-complex-system/)

      That sounds very impressive, but unfortunately my P.C. does not fit the description. It’s a machine, bless it. Despite certain appearances, I’ve yet to find organisms in its software. It has no consciousness independent of its programmer and has yet to be caught procreating. What on earth are M/S thinking? That Windows 10 is a virtual reality universe?

      4 users thanked author for this post.
    • #232858 Reply

      JohnW
      AskWoody Lounger

      Well it’s obviously too complex for Microsoft to understand and manage!  🙂

      1 user thanked author for this post.
    • #232861 Reply

      gkarasik
      AskWoody Lounger

      How to use up a company’s reputation, built over a lifetime: Say things, instead of doing things. It’s cheaper! People will take a Long Time to let go of belief.

      -Noel

      I never mistook Microsoft for Mother Teresa, but while Gates and Ballmer were there the company did pay some attention to their public. That changed radically when Gates/Ballmer left.

      GaryK

      2 users thanked author for this post.
    • #232862 Reply

      Elly
      AskWoody MVP

      Another problem with the statistics here… the burden of being beta testers lays heavily on W10 Home users… since the people I know use Home, that might account for their higher reporting of problems (everyone that I know that use W10 Home has had problems)… but Microsoft may be using their unfortunate experiences to protect a larger number of Education and Enterprise machines… thus showing less problems overall. But its killing the Home users! No one recommends buying/using W10 Home now. If you mention problems with W10 Home at Best Buy they will steer you to a Chromebook… without hesitation… they want satisfied customers.

      Win 7 Home, 64 bit, Group B

      2 users thanked author for this post.
      • #232902 Reply

        mindwarp
        AskWoody Lounger

        The other problem might be, since Home users are the unpaid and UNTRAINED (caps for emphasis) beta testers, how many problems go unreported because Home users don’t know how to report them?  There’s a reason my IT department likes me – I did tech support in a previous job, so when I email them, they know that they’ll actually get what they need to know, properly written up.  I know what to write when reporting a bug, and can track down how to do so – but I’m not an average Home user (on rare cases I use my PC).  Are average Home users up to that?

        • #233057 Reply

          anonymous

          That’s why they need to spy on people so they can get the information they need rather than trust the user to report it.

    • #232874 Reply

      Cybertooth
      AskWoody Lounger

      It’s a machine, bless it. Despite certain appearances, I’ve yet to find organisms in its software.

      Well, people do keep finding bugs in Windows, don’t they?  😉

      But more seriously, it grates on me too that people are using the word “ecosystem” to refer to things where it clearly doesn’t belong. Just call it (as the case may be) a “market,” an “industry,” a “company,” an “OS,” what have you. All these nouns are sufficient to describe what they refer to. There’s no need to get figurative about what you’re taking about. Saying “ecosystem” is lazy, sloppy thinking.

      And if you want to use the most all-encompassing, general term that you can, then for heaven’s sake just say “system” and save yourself (and us) two syllables. You sound neither hip nor knowledgeable when you say “ecosystem”.

       

      • #232881 Reply

        OscarCP
        AskWoody Lounger

        “Ecosystem” is a good long word to use for impressing the rubes. Which, in the opinion of the PR people who use it, is what the vast majority of us are. It is up there with “momentarily” used to mean “shortly” or “soon” (but Yuck! those are short words). We live in a pretentious age where self-important people use long words to impress others and each other.

        2 users thanked author for this post.
      • #233069 Reply

        anonymous

        Ecosystem’s a good analogy for software development, especially at the scale Microsoft works.  As software “evolves” it creates new pressures within the ecosystem that other software devs react and adapt to.  You have evolutionary lines (eg, Microsoft Office), and you have dinosaurs (around the 80s).  For larger applications, code constantly has to change to keep up with the times and make sure that everything stays working within its ecosystem.

        So no, your computer isn’t alive, but it has continuously changed.  Generally for the better, but sometimes like now for the worse.

         

    • #232896 Reply

      anonymous

      I suspect that Microsoft’s filings with the Securities and Exchange Commission (Forms 10K, 10Q and 8k) do not adequately disclose the dissatisfaction and problems with the Windows update system.

    • #232901 Reply

      gkarasik
      AskWoody Lounger

      “We shifted the responsibility for base functional testing to our development teams in order to attempt to deliver higher quality code from the start, but often fell short as exemplified by the numerous issues both business and end users have experienced including, but not limited to, data loss.” There, fixed THAT sentence.

      When the jargon starts, truth is usually the first casualty.

      GaryK

      1 user thanked author for this post.
    • #232915 Reply

      ojiisang
      AskWoody Lounger

      I think the graph shows that people stopped complaining over time because they already know that their complaints won’t be heard.

      2 users thanked author for this post.
    • #232942 Reply

      HiFlyer
      AskWoody Lounger

      M$ multi-syllable bafflegab reveals more than it conceals.  Pitiful!

      Words of one syllable work.

       

    • #232944 Reply

      anonymous

      Netmarketshare numbers for Windows desktop OS:
      Sep 2016 = 90.45%
      Oct 2016 = 89.97%
      Oct 2017 = 88.00%
      Oct 2018 = 87.27%

      Since the cessation of the 1-year free upgrade(from Win 7/8.1 to Win 10) promotion in Aug 2016, Windows has lost more than 3% in world marketshare = about 60 million desktop/laptop computer users left Windows = these ex-Windows users just up and leave = not feed-backed or recorded by M$ or the numbers were just ignored by M$.

      The above numbers likely prove that Win 10 patching has gone down the drain. […]

      2 users thanked author for this post.
    • #232999 Reply

      Elrod
      AskWoody Lounger

      We also changed the focus of the teams that still report to me who are responsible for end-to-end validation,

      My wife uses Win10 Home.  Does she now report to Mr. Fortin?

      And, if so, what’s she pulling down, after taxes?  Just curious.

      Group "L": Linux Mint

      1 user thanked author for this post.
    • #233047 Reply

      EP
      AskWoody MVP

    Please follow the -Lounge Rules- no personal attacks, no swearing, and politics/religion are relegated to the Rants forum.

    Reply To: Microsoft says it’s improving patch quality for a complex ecosystem

    You can use BBCodes to format your content.
    Your account can't use Advanced BBCodes, they will be stripped before saving.

    Your information:


    Comments are closed.