Windows 7 Experience Index hard drive numbers changed

Comparing the Windows Experience Index performance details for Windows 7 Build 7000 (the official “beta” version) and Build 7057 (the latest leaked build), there’s no question that Microsoft is changing both the performance tests and the way they’re interpreted.

The WEI – Microsoft’s performance index – is put together by a program called WinSAT, the Windows System Assessment Tool. If you’re curious about the details, I talk about WinSAT in Windows Vista Timesaving Techniques For Dummies, Technique 1. (The Windows Experience Index originated with Vista; you can run the tests on Windows XP machines, but they don’t mean much.)

WinSAT stores the results of its tests in XML files in c:\Windows\Performance\WinSAT\DataStore. If you look in that folder you can see the results of all of the WinSAT tests that have been run on your PC since day one.

I compared the WinSAT reports run under Build 7000 with the WinSAT reports under Build 7057, and found that not only did the performance scores change, the tests themselves changed.

To re-state that in plain English: Microsoft has changed the way it’s measuring hard drive performance in Windows 7, no doubt in response to voluminous complaints from customers that the hard disk performance number in the Windows 7 beta didn’t match customers’, uh, experience. (I, too, railed about it in my Windows Secrets Newsletter article.) The powers-that-be may have also been influenced by the fact that turning off disk write caching – thus presumably slowing down the hard drive – drives up the Windows 7 beta hard disk performance number. An embarrassing bit of techie tinkering.

My Windows 7 beta (32-bit) machine lists these hard drive scores:

<AvgThroughput kind=”Sequential Read” units=”MB/s” ioSize=”65536″ score=”5.8″>62.56167</AvgThroughput>
<AvgThroughput kind=”Random Read” units=”MB/s” ioSize=”16384″ score=”3.5″>1.45000</AvgThroughput>
<MeanLatency Kind=”Read With Sequential Background Writes” units=”us” score=”4.3″>9070</MeanLatency>
<MaxLatency Kind=”Read With Background Writes” units=”us” percentile=”95″ score=”1.9″>68724</MaxLatency>
<MaxLatency Kind=”Read With Background Writes” units=”us” percentile=”100″ score=”7.1″>257208</MaxLatency>
<MeanLatency Kind=”Read With Random Background Writes” units=”us” score=”2.6″>15010</MeanLatency>

The same computer, same hard drive, same everything, running under Windows 7 Build 7057 logs these performance numbers:

<AvgThroughput kind=”Sequential Read” units=”MB/s” ioSize=”65536″ score=”5.7″>62.35333</AvgThroughput>
<AvgThroughput kind=”Random Read” units=”MB/s” ioSize=”16384″ score=”3.3″>1.17000</AvgThroughput>
<Responsiveness Kind=”AverageIORate” units=”ms/IO” score=”5.6″ factor=”0.0″>4.64000</Responsiveness>
<Responsiveness Kind=”GroupedIOs” units=”units” score=”6.7″ factor=”0.0″>12.23557</Responsiveness>
<Responsiveness Kind=”LongIOs” units=”units” score=”6.9″ factor=”0.0″>15.52148</Responsiveness>
<Responsiveness Kind=”Overall” units=”units” score=”6.4″ factor=”0.0″>189.91417</Responsiveness>
<Responsiveness Kind=”Cap”>FALSE</Responsiveness>

See how the numbers change, as do the tests?

The bottom line is that my hard drive, which scored a 3.0 in the Windows 7 beta, now scores a 5.7. It didn’t get twice as fast. Microsoft changed the way it measures and reports performance.

Don’t get me wrong. The Windows 7 Beta disk performance test is a fine test. Mathematically it’s backed to the hilt with surveys, cross-checks, experiments, and lots of real-world correlation with the test results. The problem is that the test didn’t measure what I do with my hard drive. I spend a lot of time on sequential reads – streaming media, sucking in photos, leafing through songs. I spend very little time with random writes: I don’t modify picture files in great quantities, or perform massive database updates.

There’s good reason for stress-testing and reporting the results of stress tests as part of the WEI. For example, I red-line my CPU from time to time, and it’s worthwhile knowing whether one CPU can handle the load better than another. I stress out my video card all the time. The fact that there are two different WEI numbers for video cards doesn’t bother me a bit – I can understand where Microsoft is coming from, and adjust my expectations accordingly. I don’t expect the memory test to tell me much except whether it’d help to add more memory. And the WEI score reflects that expectation well enough.

But the hard drive WEI test in the Windows 7 beta just didn’t reflect my reality. And I bet it doesn’t reflect yours, either.

If I have one hard disk with a WEI of 3.0 and another with a WEI score of 5.7, I expect the second one to perform significantly faster than the first, doing the kinds of things I usually do. I should be able to stream movies with fewer drop-outs. Run backups faster. See thumbnails of my photos and album covers faster. Open picture files faster. Start programs faster. The Windows 7 beta WEI disk score didn’t measure any of that. So, for me, it was worthless. Worse than worthless, actually, because a low, meaningless WEI disk score can pull down the rating of my whole PC. That’s the problem I talked about in my Windows Secrets Newsletter article.

It appears as if Microsoft is listening, and making significant last-minute changes in Windows 7 based on reasonable comments and complaints from customers. I, for one, find that refreshing. And different.