Adds test validation and removes power management inconsistencies

May 1, 2013 07:40 GMT  ·  By

The SunSpider JavaScript benchmark suite is more than five years old. That might as well be a million years as far as the web is involved. In those five years, JavaScript engines have become orders of magnitude faster, and you can thank Google Chrome for that.

There are also newer and more comprehensive benchmarks, like Google's Octane and Mozilla's Kraken. Still, the WebKit team believes SunSpider still has its uses.

Because it relies on short snippets of codes and because it benchmarks JavaScript operations that are hard to optimize, the team believes SunSpider has some unique capabilities.

This is where the SunSpider 1.0 release comes in. The latest version comes with bug fixes and better accuracy.

There are some bigger additions as well, and one is test validation. When SunSpider came out, JavaScript engines were relatively simple and predictable.

Now, most use one or two just-in-time compilers and all manners of optimization algorithms. This is why it's become important to test the results of the benchmark and not just how fast the code is processed.

23 out of the 26 SunSpider tests now have validation checks. These shouldn't slow down the test too much, as the team tried to keep overhead under two percent.

Another area that has seen improvements is interference with the score due to power management schemes. SunSpider added a 500 ms delay between tests to give the browser time to complete asynchronous tasks.

But that delay is enough for CPUs with aggressive power management schemes to revert to a lower clock. The lower clock, as well as the overhead of having to ramp up clock speed when another test started, could lead to poorer and inconsistent scores.

The delay was shortened to 10 ms as of SunSpider 0.9.1, but that soon became too much as well. So the delay was removed altogether. In testing, the team found that the move improved repeatability of the scores.

Of course, without the delay between each test, the SunSpider benchmark now runs faster than ever before. That's a good thing, but it also means scores aren't comparable with previous versions, if this needed mentioning.