Lies, damned lies, and Android benchmarks

I've been making, evaluating, and using benchmarks for decades. Along the way I've seen vendor after vendor try to play games with them so that their products would look better than the competition. So, I wasn't surprised when the news broke that Samsung had fudged recent Galaxy Note 3 benchmarks to make its hot new smartphone look better. I'm even less surprised that AnandTech has subsequently discovered that almost all Android device OEMs have rigged their devices to show their best on certain benchmarks.

AnandTech, which has specialized in hardcore technology reviews since the 1990s, found that, "With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device" that checks for specific benchmark tests. When such a device finds that it's being tested, it automatically drives its "CPU voltage/frequency to their highest state right away."

The other exception to the rule that everyone cheats was that none of the various Nexus devices showed signs of tampering. That's because these devices use Google's stock Android Open Source Project (AOSP) for their operating system and "the optimization isn't a part of AOSP."

In each case, it's not that the device was feeding the benchmark fake numbers. Instead, as the devices detected that a benchmark was being used they'd simply turn on all their cores and run at the fastest possible speed. The result was that, for the purposes of performance tests, they were showing their best possible results rather the results they'd show under normal use.

AnandTech also found that each OEM optimized their devices for different benchmarks. Only in one case, Qualcomm's Vellamo Mobile Benchmark, did AnandTech find that all vendors' flagship devices "seem to game this test, which sort of makes the point of the optimization moot" since when everyone is cheating the field is, ironically enough, back to being level.

The smartphone and tablet OEMs, and not the chip manufacturers, seem to be the ones behind this latest round of benchmark tampering. AnandTech wrote, "I know internally Intel is quite opposed to the practice (as I’m assuming Qualcomm is as well), making this an OEM level decision and not something advocated by the chipmakers."

The benchmark vendors have also been long aware that vendors and OEMs will try to game their tests. Some, such as FutureMark, make a point of stating that trying to detect their benchmarks take advantage of the test is not allowed.

While the benchmarking companies and publications strive to make testing transparent and above board, the vendors are constantly trying to put their best tech foot forward by any means they can. So, as Michael J. Miller, CIO of private investment firm Ziff Brothers Investments and former editor-in-chief of PC Magazine, pointed out in 2011, "Every benchmark needs to be taken with a grain of salt."

AnandTech also noted that, "The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0-5 percent, and somewhere south of 10 percent on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors."

There's nothing new about any of this. I've been in the benchmarking game for almost 30-years now and while AnandTech hopes that the OEMs will see the error of their ways and stop trying to game benchmarks I can say with perfect confidence that they won't.

People want a magic number that will tell them with a glance if they're buying the best PC, tablet, smartphone, car, what have you. For computing devices that magic number is a benchmark number. So long as people want a quick, one-number answer, testers, benchmark makers, and vendors will continue to play a tug of war over benchmark results.

As I tell anyone looking to buy any technology, look beyond the benchmarks and to the entire package to decide if it's the right device for you. There are no magic, absolute numbers that can tell you if something is the right buy for you. Only you, after looking over benchmarks, reviews, and your own hands-on experience, can decide what tech is really right for you.

Related stories:

• Samsung caught fudging benchmarks (again)
  
• Benchmarks: HTC One, Sony Xperia Z, Google/LG Nexus
  
• New virtualization benchmark for clouds and datacenters
  
• TOP500 leaders announce new supercomputer benchmark
  
• Tech giants join forces to create Android gaming benchmark