14 DAY TRIAL // It's hard to explain why a browser 'feels' better or faster than another, but it's very easy to compare clear numbers. Which is why benchmarks are always cited to 'show' that one browser is faster (and therefore better?) than the next. But benchmarks can't really be trusted.
This is nothing new, synthetic tests can be quite useful but they should never be relied upon as the definite yardstick that separates browsers, or any other piece of software or hardware for that matter.
That said, when utilized properly, benchmarks are a great tool. One area where this is the most obvious, when it comes to web browsers, is JavaScript performance.
JavaScript performance has always been important, but ever since Chrome came along and Google 'declared' JavaScript as the de facto coding language of the web, how well browsers handle JavaScript-heavy websites has become one of the major ways a browser maker can assert its superiority.
Three JavaScript benchmarks are mainly utilized to test performance, unsurprisingly coming from some of the major browser makers. None of them are perfect though.
Google isn't satisfied with the state of the current benchmarks, it's why it created its own V8 benchmark suite in the first place. But now it's taken to fixing the other two as well.
This isn't just so that Chrome can rank better in all benchmarks, accurate results ensure that developers know what to focus on and what the problem areas are.
Google says that it constantly updates and improves its own V8 benchmark so it has no major gripes with it. Since these are the benchmarks on which Chrome development is based on, it's in Google's best interest to ensure that they are relevant and accurate.
But Google believes the other two major benchmarks, SunSpider and Kraken, could see some improvements.
Sun Spider is the oldest but still the most respected JavaScript benchmark and was initially developed by Apple. It's an open source project, so anyone can alter it, and it has seen several updates along the way, but it is starting to show its age.
The big problem is that modern browsers are so much faster than they were four years ago that they get through the tests too fast for it to be relevant.
"Even for the more relevant tests, JavaScript has gotten so fast that many finish in only a few milliseconds. This just isn’t long enough to figure out which engine is faster--a golf cart and a Tesla will finish a 10-foot race in nearly the same time," Google writes.
There's not much you can do to really fix this, except to fully rewrite the tests, but Google has created a modified version that runs each of the individual tests in the SunSpider benchmark 50 times over.
It's not an ideal solution, but Google says it does provide more relevant results. It also says that with this modified version Chrome is 30 percent faster than with the regular one and that it's four times as fast on some tests.
Mozilla's Kraken, the latest entry to the bunch, fares a lot better, but there are still issues. The version that is offered for testing, Kraken 1.0 is outdated and has bugs that affect its accuracy, Google says. As such, the Chrome team started using the very latest version, Kraken 1.1, pulled straight from the Mozilla repositories.
Google is now using these modified versions for internal testing to make sure developers have accurate data to go on. Google is also hosting and offering both of them to anyone interested in JavaScript performance testing.