14 DAY TRIAL // At the end of the past week, Microsoft made available to the public the results of a “browser measuring contest” involving Internet Explorer 8, Firefox 3.05 and Google Chrome 1.0. What the Redmond company did was to measure the browser load times for the top 25 websites worldwide. In the software company's own comparison, IE8 managed to come out on top of its rivals, besting the open source Firefox and Chrome which were released in 2008, and not the current 3.1/3.5 version of Firefox or the 2.0 flavor of Chrome. But the real question is does it matter?
Obviously, Microsoft is rather a new comer to the browser measuring game. Simply because Internet Explorer, while considered by rival browser makers a reference point, never actually needed to compare itself to competitive products, especially from the perspective of the dominant browser on the market. IE continues to account for the lion's share when it comes down to browser usage share worldwide, but Firefox has made a dent in its dominance that is harder and harder to ignore. And fact is that for the first time in the long life of Internet Explorer, Microsoft has a release that it’s worth comparing, as it is bound to score at least on par with rivals, if not even best them in certain scenarios.
“Did you know that IE 8 Loads faster than Firefox 3.05 and Chrome 1.0 on the top 25 popular web sites? Is the faster on 5 out of the top 10 sites; 12 of the top 25; 3 times as many as Firefox; and 1/3 more than Chrome,” revealed Microsoft's Rob Margel. The table included on the left, from the “Measuring Browser Performance: Understanding issues in benchmarking and performance analysis” study, contains all the results published by Microsoft.
“In line with the testing guidelines outlined in this paper we have used the browser “Done” indicator for timing when the page is completely loaded at that point. For pages which continue to load and change after the “Done” indication we have used common visual cues to generate the timings. Timing is started when the Go button is pressed. These timings were captured in January 2009; because Internet content is always changing you may get different timings when you run these tests,” Microsoft revealed.
The question remains. Does it matter that Google.com is loaded in 0.28 seconds by Chrome, 0.22 seconds by Firefox and 0.20 seconds by IE8? What does a mere 0.02-second difference in browser load times represent to the end user? Well, absolutely nothing. You would need a high speed camera just to notice the difference. And at the same time, differences such as 5.48 seconds for Chrome, 6.37 seconds for Firefox and 8.03 seconds for IE8 when loading Sina.com.cn are more than visible, and irking to the end users. Still, what Microsoft does is establishing that benchmarking is in no way representative of real-life.
“Many benchmarking tests cite results from tools such as SunSpider, Celtic Kane, V8 and iBench. Each is designed, at least in some way, to test only a portion of the end-user scenario. SunSpider primarily tests JavaScript performance and also some other popular functions used on websites today. Other popular benchmarks, Celtic Kane and V8, are also geared at testing only the JavaScript engine. Meanwhile, the iBench suite relies on a browser reporting when it’s finished loading a page. Since browsers differ on when they define a page is done loading, this test also yields unreliable results,” the company stated.
Fact is that any measurement performed involving browsers has to take into account a multitude of variables from caching behavior, to overhead and resource competition, but also connectivity and network device latency, Internet-based data, browser-targeted content, machine configurations, inconsistent definitions, and even extensibility. In this context, benchmarks and benchmark results remain largely a marketing tool more than anything else. Due to the sometimes minute differences between the browser load times in the tests performed by Microsoft, with some minor tweaks, rival browser makers Mozilla and Google could turn the same set of tests to their advantage easily. Such a situation certainly diminishes the relevance of browser measuring contests.
Internet Explorer 8 (IE8) Release Candidate 1 (RC1) is available for download here.
Firefox 3.1 Beta 3 for Windows is available here.
Firefox 3.1 Beta 3 for Linux is available here.
Firefox 3.1 Beta 3 for Mac OS X is available here.
The latest development milestone of Google Chrome is available for download here.
