The saga of Microsoft testing

Topic

New Reply

Buried in a Reddit thread, there’s an interesting (and, I believe, accurate) retelling of the recent history of Microsoft testing. Here are the high p
[See the full post at: The saga of Microsoft testing]

9 users thanked author for this post.

lurks about, Noel Carboni, Karlston, SueW, WildBill, PhotM, Elly, satrow, Microfix

Reply | Quote

Replies

Volt creates the MS current so there has to be resistance!
nothing seems in parallel, just a series of patch …… ups.

Windows - commercial by definition and now function...

Reply | Quote

A long, long time ago, I worked in retail, and you are not allowed to ring yourself out for sales.

That’s basically what they’re doing, if they’re letting their devs test their own code.
If Tom codes incorrectly but thinks he did it right, and he looks over his code (“testing”), he’s not going to see a problem. The code gets pushed to prod. This is about the most backwards bottom thing I’ve ever heard. But it fits, and it makes sense. Like the saying goes, “where there’s smoke, there’s fire”.

Nearly everything in life should have a checks and balances system; software is definitely one of them. This is asinine! But they don’t. They have telemetry data now, so they can release whatever they want, with little to no repurcussion, because they have that data. It’s similar to how a lot of games come out nowadays and are broken pre-launch so they do what every good (snark) company does and puts out a day 1 patch to fix what they didn’t catch during RTM.

5 users thanked author for this post.

BobbyB, lurks about, Noel Carboni, SueW, WildBill

Reply | Quote

zero2dash wrote:

If Tom codes incorrectly but thinks he did it right, and he looks over his code (“testing”), he’s not going to see a problem.

Hopefully Tom at least waits a day or so before looking over his code. But even better would be to have a totally different set of eyes looking at the code. I know when I was a programmer, I had a preferred way to code, and I felt very strongly about it. That can be a major blind spot, which is why you need different people testing the code than the ones who wrote the code.

Actually, the programmer should test his own code. But someone else should also test that same code.

Group "L" (Linux Mint)
with Windows 10 running in a remote session on my file server

7 users thanked author for this post.

Bill C., GoneToPlaid, BobbyB, Noel Carboni, SueW, WildBill, Elly

Reply | Quote

MrJimPhelps wrote:

I know when I was a programmer, I had a preferred way to code, and I felt very strongly about it. That can be a major blind spot, which is why you need different people testing the code than the ones who wrote the code. Actually, the programmer should test his own code. But someone else should also test that same code.

I tested my own code, but sometimes when it was a tricky bug that kept reappearing, I would get a co-worker to review & test. A 2nd set of eyes helps. Also, even if it might throw a project a little off-schedule, I didn’t rush it into production. Programming, especially for a bank, you don’t want to be called in at night to fix something you could have fixed in testing. Sadly, Microsoft is rushing everything & without taking time to do it right, or having other people test, is a recipe for the disaster we have.

Bought a refurbished Windows 10 64-bit, currently updated to 22H2. Have broke the AC adapter cord going to the 8.1 machine, but before that, coaxed it into charging. Need to buy new adapter if wish to continue using it.
Wild Bill Rides Again...

2 users thanked author for this post.

AlexEiffel, BobbyB

Reply | Quote

Review – i.e., looking over one’s software work with at least one peer – is VERY powerful and can get a programmer quite a long way toward creating correct and high quality code. I hope and imagine that Microsoft has built plenty of peer review into their processes.

That being said, because the complexity of this stuff is WAY beyond what humans can manage without a methodical, layered approach, there is always going to be code that makes it all the way to the point where it would be released without discovery of problems. Even with a review-rich environment I find testing is still necessary.

-Noel

2 users thanked author for this post.

MrJimPhelps, BobbyB

Reply | Quote

I haven’t done coding in years. Yet I too developed my own preferred way to code — formatting, comments and all. And therein lies the problem when I would review my own code. It all looked so nice and clean, since it was my own coding style, that sometimes I could not spot my own coding mistake. Yet a second person could look at my code, read my comments about what this section of code was supposed to do, and then instantly spot my coding mistake. The result usually was a slap of my own palm to my forehead. Everyone who codes is wearing blinders when reviewing one’s own code.

1 user thanked author for this post.

Noel Carboni

Reply | Quote

Spot on! Especially the part about the nice and clean formatting and comments. I failed on a job interview once, because I inadvertently revealed that I was very obsessive about making my code look perfect, with perfect comments, spacing, etc etc. They knew that it took me forever to write code because of this, so they hired someone else.

By the way, well-tested and perfect code does not occur often in most custom software companies. They know that if they don’t hurry and get the product out, their competitor will beat them to the punch. So they get it “substantially” correct and then release it. If they can convince the customer that they have met the requirements, they can then call bug fixes “enhancements” and charge the customer for fixing them!

Microsoft is not a “custom software” company; they don’t do custom programming for specific customers, but rather they produce one product which is sold to all their customers. So the previous paragraph is off-topic; it doesn’t apply to “the saga of Microsoft testing”.

Group "L" (Linux Mint)
with Windows 10 running in a remote session on my file server

Reply | Quote

Guess they’re part of the “move fast and break things” culture now.  And now, with Windows 10, you can’t keep them at bay.  Business users beware.  This isn’t your father’s Microsoft.

Reply | Quote

The “investment” world is not connected to the “run the business” world, and most certainly not the “deliver quality products” world. That seems to be the root of all this. If you’re a consumer, you want a flawless, high quality product that’s going to deliver good value and last a long time. Most investors, however, want the company they’ve invested in to be ruthless in pursuit of their dividends.

Using perpetually unstable software (whether because of constant updates or because of bugs), feels a bit like having a new car that you take back to the dealer for warranty work and they deliver it back to you with another, different problem that they’ve introduced, and they then expect you to pay them to fix it because THAT part isn’t under warranty. It’s just not a low stress way to live life. But that’s okay – we all need more stress, right?

-Noel

9 users thanked author for this post.

Bill C., ve2mrx, AlexEiffel, GoneToPlaid, BobbyB, lurks about, SueW, woody, WildBill

Reply | Quote

There are times when the obvious is not obvious. Even the most ardent and accomplished coders know that. The Test Team at a software company have a set of test criteria and they are more likely to spot a problem with the code itself. Beta testers will uncover that which was not anticipated, but they may not report it, especially if it is not important to them.

AI will take care of testing the code at Microsoft – is that the plan? Silicon versus carbon!

2 users thanked author for this post.

columbia2011, Noel Carboni

Reply | Quote

A quote from https://itvision.altervista.org/why-windows-10-sucks.html

You may probably want to know why Windows 10 feels so buggy. Here’s a very nice <abbr title=”https://news.ycombinator.com/item?id=10937526″>quote</abbr&gt;:

Full Disclosure: I worked at M$ from 2014-2015.

MS has some very talented programmers. They’re not very common, but they exist. The problem is that the entire company is completely and totally focused on developing an absurd number of new features and products, giving them completely unrealistic deadlines, and then shipping software on those deadlines no matter how half-assed or buggy it is.

The idea is that everything is serviceable over the internet now, so they can just “fix it later”, except they never do. This perpetuates a duct-tape culture that refuses to actually fix problems and instead rewards teams that find ways to work around them. The talented programmers are stuck working on code that, at best, has to deal with multiple badly designed frameworks from other teams, or at worst work on code that is simply scrapped. New features are prioritized over all but the most system-critical bugs, and teams are never given any time to actually focus on improving their code. The only improvements that can happen must be snuck in while implementing new features.

As far as M$ is concerned, all code is —-, and the only thing that matters is if it works well enough to be shown at a demo and shipped. Needless to say, I don’t work there anymore.

(edited for coarse language by me)

Hanlon's Razor: Never attribute to malice that which can be adequately explained by stupidity.

3 users thanked author for this post.

AlexEiffel, Noel Carboni, Elly

Reply | Quote

New features are prioritized over all but the most system-critical bugs, and teams are never given any time to actually focus on improving their code. The only improvements that can happen must be snuck in while implementing new features.

This is EXACTLY what I have been saying. It’s very obvious, even from the outside.

Reply | Quote

I read the other articles, that guy is observant and logical. Unable to disagree with any of his collected points with which there has been experience to date, both with Windows & Linux (GNU) systems.

Reply | Quote

I fully recognize the issues with testing. Yes, a developer should test his (or her) own code. However, it’s essential for others to test as well — fresh set of eyes, different sets of assumptions, etc. When you’re developing code, it’s really easy to get so accustomed to working with your own code that you miss exceptional cases, that others are far more likely to find. Or something as simple as forgetting to remove various sorts of debugging things that should never be in production code. As noted previously, there’s lots of blind spots.

I can see a reasonable case for including people on the Insider Track for participating in debugging, because it allows for better identifying stuff in production environments. However, that kind of participation must be voluntary, and Microsoft is forcing that on a much wider scale, especially Home users, as well as the massive amounts of privacy-invading and resource-consuming telemetry. But that’s all a part of “Windows as a Service”, where it’s emphatic that Microsoft owns Windows, and users are merely renting licenses to use Windows, under Microsoft’s boilerplate contract terms (take it or leave it).

The underlying problem to all of this is that the release cadence is being driven by the marketing people, where stuff is being done on a calendar that they’re dictating, regardless of whether stuff is ready or not. Thus, we have the semi-annual feature releases (and limited window of support for any specific release), and monthly Patch Tuesday updates, all of which are now difficult to reject or delay, even if there’s compelling reason to do so.

The saga of the 1803 release is especially instructive. It was all ready to go, and then at the last minute, pulled out of the release channel and reverted back to beta, because there were serious problems. And even when 1803 was released, there were known problems that were still unfixed. And to add insult to injury, the release to the Current Business Branch (or whatever they’re calling it this month) was done not because it was truly ready for enterprise use (and still with serious flaws), but because that was the original schedule, and delaying that any further would have caused unacceptable delays in releasing 1809.

In the end, what Microsoft is doing is not because there’s any significant benefit to its customers (even the large enterprise customers) for making them more effective in running their operations, but because the constant changes meet Microsoft’s revenue needs. And for the customers that aren’t paying big money for enterprise contracts, the contempt for everybody else is pretty telling.

As long as Microsoft’s marketing people have the final say, nothing is going to change, not even what Susan Bradley did with her incredibly useful surveys and subsequent Open Letter to Microsoft. If they’re going to ignore a friendly request from somebody that’s that prominent, it’s likely that they’re not listening to anybody but themselves.

3 users thanked author for this post.

AlexEiffel, lurks about, Noel Carboni

Reply | Quote

There are two systemic problems with W10. First is the lack of proper testing and thus releasing very buggy software with users bearing the brunt of the problems. The other is a complete failure to understand how a rolling release works. Google and Apple do variation of it with their OSes in that they release a new version regularly but a slow enough cadence to get the code right. Arch Linux and its progeny are true rolling releases with updates to any and all the installed code to the newest version coming out throughout the month. Note most of these updates are bug fixes and minor improvements not massive new features which take time to implement correctly.

Another point is Google’s idea all software is in perpetual beta is widely misunderstood to mean the release beta or worse quality software. It really means software is evolving and can not be consider fixed for a long period of time. An example is Chrome. It is has frequent releases but each release is polished and stable. Whether the idea of perpetual beta is always valid is another issue, I tend to think for some software it is not true they must be considered in perpetual beta.

1 user thanked author for this post.

AlexEiffel

Reply | Quote

Daniel Miessler’s view on software:
https://danielmiessler.com/blog/the-reason-software-remains-insecure/

3 users thanked author for this post.

AlexEiffel, lurks about, woody

Reply | Quote

As a developer you need someone independent to test your code. It does not matter if you do your own unit testing and have peer review neither will catch every mistake. But a well thought out test program often will catch them. The more that are caught while the code is being developed the fewer issues with release. Essentially firing your testers will eventually bite you hard as the code quality drops.

Reply | Quote

I hope this is a real exchange because it is long overdue, now if we could get the dirt on the GWX malware creation that would be awesome. Did anybody archive this conversation?

Reply | Quote

MrJimPhelps wrote:

zero2dash wrote:

If Tom codes incorrectly but thinks he did it right, and he looks over his code (“testing”), he’s not going to see a problem.

Hopefully Tom at least waits a day or so before looking over his code. But even better would be to have a totally different set of eyes looking at the code. I know when I was a programmer, I had a preferred way to code, and I felt very strongly about it. That can be a major blind spot, which is why you need different people testing the code than the ones who wrote the code.

Actually, the programmer should test his own code. But someone else should also test that same code.

Great comment!

I am/was not a coder, but was tasked with testing and writing about testing as well as policy documents, and end user guides. I also had a voluminous email inbox that was urgent. I have found that when time permits, the best course is to draft, edit, and then IF time permits, age the writings and revisit it. This is critical for important and/or controversial issues. Funny how your replies get better, you do not get the “yeah, but…” replies and your solutions tend to get adopted.

Telling coders the first thing off your mind as a tester is only slightly productive unless you have an ongoing working relationship. Thinking of what you are telling them tends to save time as they do not go down the wrong path or have to guess what you mean. In our project we actually had short tester/user/coder sitdowns that greatly reduced rewrites and changes.

Remember, before you send out that blue rocket email, age it. It weill save time (and maybe your job) in the long run.

Now if we could just get Twitter users to do the same thing….

Reply | Quote

They have now more world wide testers than ever before, so it was a wise move.

Of course the developers are brilliant and all that and as already stated, they can only test so far. Next step is of course to release to a selected group of user and get instant feedback via telemetry.

So money saved on testing and whatever way the released code impacts the recipients is of no consequence for Microsoft.

Win-win.

Except if it’s your machine that goes tits up, of course.

Added, without snark: I think they’re right. – Woody

Should you ever be in any need of snark, just give a call. I’ve got tons…

Reply | Quote