The recent benchmarking tests performed by Devil Mountain Software, and made public via the exo.blog, comparing the Release Candidate builds of Windows XP Service Pack 3 and Windows Vista Service Pack 1 are irrelevant, claimed Microsoft. Nick White, a Product Manager with the Vista launch team contested the relevance of the tests based on the bad timing and the lack of synchronization with the two flagship products' lifecycles. The Devil Mountain Software company concluded that XP SP3 was no less than twice as fast as Windows XP in terms of performance, after a comparison of the Release Candidate version of XP SP3 and a Preview of the Release Candidate for Vista SP1. White gave as an example of a proper test a study it commissioned to Principled Technologies for XP SP2 and Vista RTM.

"My point is that we waited to conduct these benchmarking tests until Windows Vista had reached the RTM milestone in the product cycle, as this allowed us to provide our customers the most meaningful data available at the time -- the data most likely to directly affect their decision to upgrade to Windows Vista. We do a whole range of performance tests at every stage of the OS development process but, as a general rule, we avoid sharing benchmark tests of software that hasn't gone RTM (i.e., final code). This explains why we have not to date published any findings of benchmark tests (nor commissioned anyone to do so) on performance improvements brought about by Windows Vista SP1. Publishing benchmarks of the performance of Windows Vista SP1 now wouldn't be a worthwhile exercise for our customers, as the code is still in development and, to the degree that benchmarking tests are involved, remains a moving target," White stated.

At the bottom of this article I have embedded a Vista benchmarking test designed to illustrate "a window-open, window-close routine at accelerated speed." White emphasized the fact that, while the test is a valid way to measure performance, it is in fact little representative of how the end users would be running the actual operating system in real life scenarios. White also stressed the fact that Vista SP1 is growing from development milestone to development milestone, and that the service pack's characteristics also improve following this trend. But in the end, White did nothing more than to respond to the claims from Devil Mountain Software, and to defend Vista and especially SP1 in the context of XP SP3.

"In Windows many operations can require additional processing time for work that is done for reasons that benefit the customer; these can include security, reliability or application compatibility checks conducted when a program launches. These operations may add microseconds to an individual application's launch that under real usage isn't perceivable to the human eye. When thousands of such operations are strung together through automation, those few microseconds can have a cumulative effect on the benchmark result, causing performance to appear much better or worse than expected," White added.


Video: Windows Vista benchmark testing