14 DAY TRIAL // Google will no longer support the Octane browser benchmark after the company determined that the test was actually becoming detrimental to web performance.
Widely used since about 2012 by devs everywhere in the world, Octane tries to assess how well a browser runs JavaScript, which is an important component of the web.
Over time, however, browsers got better and better at taking the test. This, Google says, led to other problems. "We began to notice that JavaScript optimizations which eked out higher Octane scores often had a detrimental effect on real-world scenarios."
Much of this issue comes from the fact that the test was designed for the web as it was five years ago, which is far from where we are now. Browsers are now much more efficient at dealing with JavaScript, and websites are currently being built in different ways, which makes the test obsolete for the most part.
"Using Octane to measure V8 performance didn’t capture important use cases for the modern web, such as loading frameworks quickly, supporting large applications with new patterns of state management, or ensuring that ES2015+ features are as fast as their ES5 equivalents," Google writes in the blog post.
Cheating for a better score
It also reached a point where developers were cheating to get higher scores on the test. This, Google says, resulted in real-world performance issues. For instance, the company points out, developers were taking advantage of a bug that gave them a 15% performance bump in Octane. It did nothing on the actual web.
In balance, the test penalized some optimizations that were actually good for real-world performance, optimizations that were discovered in the past few years and that Octane didn't take into account.
For these and many other reasons, Google is ending support for Octane. "Unfortunately, similar issues exist in other static or synthetic benchmarks," Google writes, issuing a sort of warning to developers against using such tools to mark their work.