How to fool benchmarking apps

[Permalink]

The same tricks are being used in voting machines, I hear.

I don’t think the second type, automatically using 100% of speed and cpu cores when benchmark apps are detected, is really cheating. That’s optimization. They wouldn’t call it cheating on a battery life test if the OS aggressively took steps to reduce power consumption when certain scenarios were detected.

Clearly the solution is to test battery power while running the benchmarks. If you see “Battery life 36 hours standby, 10 hours browsing the web, 7 hours watching video, 90 minutes running benchmarks” then it’s pretty clear that there are shenanigans going on.

It’d serve them right if the quick look at the point then said “battery life is 90 minutes.”

I would agree if the test were one of efficiency, but it’s not. The test is of computing horsepower. However, an efficiency benchmark would be a really great addition. It would also fit with Apple’s push towards battery life as their primary benchmark for phones and laptops. I think they’re right to focus more on battery life now that cpu/gpu performance has gotten to a level where most consumers will never experience a practical difference. So, it would be nice for the industry to focus more on efficiency, i.e. “which phone does more with less?”

Apple’s primary benchmark is thickness. If it was battery life they wouldn’t be shaving shave a millimetre off a phone, and would give us more battery instead.

True. Apple’s priority benchmark is cachet. In terms of technical specs, they’ve been touting battery life lately.

it’s time to stop trusting benchmarking apps altogether.

Good luck with that. They’ve existed since the beginning of (computer) time because people really want them. Like fortune tellers, astrology, and computer ‘analysts’ like Pachter, it doesn’t even matter if they’re mostly wrong. Trust is entirely beside the point.

Partisans want them as confirmation bias, normal users want them as a score on the decision they made. If your device is near the top that’s bragging rights. If it’s average, well, it’s pretty good. If it’s near the bottom that can either feed into a victimization complex or a hate-on for your device (another form of victimization complex). Or you can perversely decide to revel in it. LG Optimus owners know what I’m talking about.

Well if they leave benchmarking apps, we get more marketing. Now that’s scary and completely false. =(.

I agree entirely. The fact is that very very few apps use the ridiculous power of the computers many of us carry around in our pockets. My phone has a quad core 1.7GHz processor and 2GB of RAM… how many apps do people think really take advantage of that hardware? The answer is: very few, so it entirely makes sense to turn off some of this hardware if it’s not being used. There is nothing that the benchmarking apps are doing to game the results and the same settings could be turned on by any app to get at that power. The reason they don’t leave it on the whole time is so the battery doesn’t drain in an hour, so the worst possible accusation one could make is that they’re messing with non-power-hungry apps to game battery life results (also known as optimisation).

I’d also like to add that Charles Arthur is a fool who writes scaremongering linkbaity crap. I’m shocked that he remains The Guardian’s tech editor and that his opinions are repeated here on BB.

PS: On a recent ep of All About Android they mentioned that if you want to get “unoptimised” results out of the benchmarking apps you simply have to rename the apk before installing because the S4 (and others) apparently use the apk title as a cue to turn on optimisation.

1 Like

Web browsers use up a surprising amount of memory.

This topic was automatically closed after 5 days. New replies are no longer allowed.