Benchmarks are not always a good indicator of performance

Sep 28, 2017 05:09 GMT  ·  By

The first real test for the latest iPhone 8 Plus is here, and it looks like the massive advantage the new phones seemed to have in various benchmarks, doesn’t’ exist when using actual applications.

The initial benchmarks showed a massive lead for the new iPhone 8 Plus, with its A11 Bionic processor. In fact, it practically destroyed the competition, in this case, Samsung Galaxy Note 8. They weren’t even close, and everyone hurried to say that Apple simply built the best phone on the market.

On the other hand, the problem with benchmarks is that they don’t accurately portray what’s happening in real life with these devices. It’s also quite possible that companies make small optimizations so that benchmarks just run better. It might not be ethical, but it’s been done before.

Benchmarks don’t actually matter

Artificial benchmarks aim to show the raw power of the hardware, but they don’t take into account what people do with their devices. This is the reason why the tests from PhoneBuff are important because they show how people will interact with the phones.

The setup is pretty simple, as you can see in the video below. A number of apps are started in a particular order, and some actions are taken along the way. After all of the apps for the test are opened in the background, the user goes back, through each one, to see if they are still working.

While the iPhone 8 Plus managed to finish the initial part of the test in the first position, with a few seconds ahead, it lost that advantage because the multitasking is not working all the great. Some of the apps didn’t resume as they should, and the Galaxy Note 8 finished the test first.

Most likely, the fact that the Galaxy Note 8 has 6GB of RAM, which is double the memory of the iPhone 8 Plus, helped it a lot during this test. It goes to show that benchmarks can’t always be trusted completely, or at least user should be cautious when seeing massive disparities between results.