The AnandTech Podcast: Episode 26

28 Comments

View All Comments

• dylan522p - Friday, October 4, 2013 - link

  Why don't you guys make your own benchmark that is actually good? You guys did it in the SSD space and the benchmarking scene there was not nearly as big nor bad.

• dishayu - Friday, October 4, 2013 - link

  I don't think it's even possible to cheat in SSD benchmarks. I believe they made the SSD benchmark to simulate real-world performance rather than purely synthetic numbers. And I think I read Anand's tweet/comment somewhere that they do plan to make their secret, undisclosed (to phone manufacturers) benchmark in the future.

• Anand Lal Shimpi - Friday, October 4, 2013 - link

  It absolutely is possible, which is why I've never given out our tests (or even low level details of our tests) to manufacturers even though they've asked over the years.
  
  The same is true for our smartphone/tablet battery life tests.
  
  Take care,
  Anand

• Anand Lal Shimpi - Friday, October 4, 2013 - link

  It's extremely expensive :)
  
  What we've done in the SSD space is ideally what we'd want to do here, but it's not quite that simple.
  
  This isn't a no, just means it's something that's going to require a lot more thought.
  
  Take care,
  Anand

• Kevin G - Friday, October 4, 2013 - link

  Have you considered the open source route for a mobile benchmark? While having the source open would let manufacturers know exactly what tests and how they're done, it would enable peer review of the code and allow other site to do a custom build that'd evade basic application detection.

• dylan522p - Friday, October 4, 2013 - link

  They would detect certainly code and almost certainly cheat.

• Kevin G - Friday, October 4, 2013 - link

  The ability to modify/build it yourself can be used to make this increasingly difficult.

• dylan522p - Friday, October 4, 2013 - link

  Hope you guys can afford one. Maybe kickstarter it and offer nothing in return. Just use the money to build the benchmark. In either case I just hope you don't let anyone touch it so they can't cheat it.

• Callitrax - Friday, October 4, 2013 - link

  First, when discussing how devices feel in hand, I expect analyses of polar moment of inertia from now on.
  
  I think one of the big problem is you need unique benchmark suites for phones, or browsers.
  The PC used to have some of these issues, but not so much anymore because most of the programs used to test computer CPUs are the same programs the user actually uses. When you tested Haswell it was with Excel, 7zip, Visual Studio, Photoshop and so on. Those are real programs that people use, so if a vendor "cheats" at those, well it just makes those programs actually faster for users. When benchmarking video cards at this point, its pretty much exclusively with video games (if there are synthetic benches in a video card review I just skip it). At this point its just understood that AMD and NVidia tweak their drivers to improve performance in big name games, but you know what - that means those games are actually performing faster, so its actually a good thing. (as long as they aren't degrading video quality like the old nvidia quake problems)
  
  So really the problem appears to come down to a lack commonly used apps and games on phones? Because then "cheating" on those would just make life better for everyone, well except that the current Android cheating method kills battery life.

• teiglin - Friday, October 4, 2013 - link

  I think the problem is twofold. First, there is a distinct lack of automation tools. Windows automation is a mature, well-developed market, but this is much less true in the mobile/ARM space. Not to mention any such benchmark could well require root privileges in order to launch other apps and profile them. Is it even possible to write something like fraps as a non-root app for Android or iOS?
  
  The other problem is basically what you mention--a lack of obvious benchmarking targets. Desktop benchmarks hit obvious, common tasks that people do with computers, like file compression, video encoding/decoding, rendering, and code compilation. The only tasks besides gaming that I can see being remotely applicable to mobile from a desktop benchmark are photo editing and maybe spreadsheet calculations. And mobile game developers don't include benchmarking tools, so I have to refer you back to my first point, which is that I at least don't know how you would go about benchmarking mobile games.
  
  I'd love to see Anand's and Brian's approach to a good mobile benchmark, but it's definitely not a simple problem.