Futuremark Begins Delisting "Cheating" Mobile Devices from 3DMark Databaseby Anand Lal Shimpi on November 26, 2013 9:37 PM EST
Over the summer, following a tip from @AndreiF7, we documented an interesting behavior on the Exynos 5 Octa versions of Samsung's Galaxy S 4. Upon detecting certain benchmarks the device would plug in all cores and increase/remove thermal limits, the latter enabling it to reach higher GPU frequencies than would otherwise be available in normal games. In our investigation we pointed out that other devices appeared to be doing something similar on the CPU front, while avoiding increasing thermal limits. Since then we've been updating a table in our reviews that keeps track of device behavior in various benchmarks.
It turns out there's a core group of benchmarks that seems to always trigger this special performance mode. Among them are AnTuTu and, interestingly enough, Vellamo. Other tests like 3DMark or GFXBench appear to be optimized for, but on a far less frequent basis. As Brian discovered in his review of HTC's One max, the list of optimization/cheating targets seems to grow with subsequent software updates.
In response to OEMs effectively gaming benchmarks, we're finally seeing benchmark vendors take a public stand on all of this. Futuremark is the first to do something about it. Futuremark now flags and delists devices caught cheating from its online benchmark comparison tool. The only devices that are delisted at this point are the HTC One mini, HTC One (One max remains in the score list), Samsung Galaxy Note 3 and Samsung Galaxy Tab 10.1 (2014 Edition). In the case of the Samsung devices, both Exynos 5 Octa and Qualcomm based versions are delisted.
Obviously this does nothing to stop users from running the benchmark, but it does publicly reprimand those guilty of gaming 3DMark scores. Delisted devices are sent to the bottom of the 3DMark Device Channel and the Best Mobile Devices list. I'm personally very pleased to see Futuremark's decision on this and I hope other benchmark vendors follow suit. Honestly I think the best approach would be for the benchmark vendors to toss up a warning splash screen on devices that auto-detect the app and adjust behavior accordingly. That's going to be one of the best routes to end-user education of what's going on.
Ultimately, I'd love to see the device OEMs simply drop the silliness and treat benchmarks like any other application (alternatively, exposing a global toggle for their benchmark/performance mode would be an interesting compromise). We're continuing to put pressure on device makers, but the benchmark vendors doing the same will surely help.