As I mentioned in our coverage of GL/DXBenchmark 2.7, with the arrival of Windows RT/8 we'd finally see our first truly cross-platform benchmarks. Kishonti was first out of the gate, although Futuremark was first to announce its cross platform benchmark simply called 3DMark.

Currently available for x86 Windows 8 machines, Futuremark has Android, iOS and Windows RT versions of 3DMark nearing release. Today the embargo lifts on the Android version of 3DMark, with iOS and Windows RT to follow shortly.

Similar to the situation with GL/DXBenchmark, 3DMark not only spans OSes but APIs as well. The Windows RT/8 versions use DirectX, while the Android and iOS versions use OpenGL ES 2.0. Of the three major tests in the new 3DMark, only Ice Storm is truly cross platform. Ice Storm uses OpenGL ES 2.0 on Android/iOS and Direct3D feature level 9_1 on Windows RT/8.

The Android UI is very functional and retains a very 3DMark feel. There's an integrated results brower, history of results and some light device information as well:

There are two options for running Ice Storm: the default and extreme presets.

3DMark - Ice Storm Settings
  Default Extreme
Rendering Resolution 1280x720 1920x1080
Texture Resolution Normal High
Post-processing Quality Normal High

Both benchmarks are rendered to an offscreen buffer at 720p/1080p and then scale up to the native resolution of the device being tested. This is a very similar approach we've seen by game developers to avoid rendering at native resolution on some of the ultra high resolution tablets. The beauty of 3DMark's approach here is the fact that all results are comparable, regardless of a device's native resolution. The downside is we don't get a good idea of how some of the ultra high resolution tablets would behave with these workloads running at their native (> 1080p) resolutions.

Ice Storm is divided into two graphics tests and a physics test. The first graphics test is geometry heavy while the second test is more pixel shader intensive. The physics test, as you might guess, is CPU bound and multithreaded.

Before we get to the results, I should note that a number of devices wouldn't complete the tests. The Intel based Motorola RAZR i wouldn't run, the AT&T HTC One X (MSM8960) crashed before the final score was calculated so both of those devices were excluded. Thankfully we got the Galaxy S 3 to complete, giving us a good representative from the MSM8960/Adreno 225 camp. Thermal throttling is a concern when running 3DMark. You have to pay close attention to the thermal conditions of the device you're testing. This is becoming something we're having to pay an increasing amount of attention to in our reviews these days.

Graphics Test 1

Ice Storm Graphics test 1 stresses the hardware’s ability to process lots of vertices while keeping the pixel load relatively light. Hardware on this level may have dedicated capacity for separate vertex and pixel processing. Stressing both capacities individually reveals the hardware’s limitations in both aspects.

In an average frame, 530,000 vertices are processed leading to 180,000 triangles rasterized either to the shadow map or to the screen. At the same time, 4.7 million pixels are processed per frame.

Pixel load is kept low by excluding expensive post processing steps, and by not rendering particle effects.

Although the first graphics test is heavy on geometry, it features roughly 1/4 the number of vertices from GL/DXBenchmark 2.7's T-Rex HD test. In terms of vertex/triangle count, even Egypt HD is more stressful than 3DMark's first graphics test. That's not necessarily a bad thing however, as most Android titles are no where near as stressful as what T-Rex and Egypt HD simulate.

3DMark - Graphics Test 1

Among Android smartphones, Qualcomm rules the roost here. The Adreno 320 based Nexus 4 and HTC One both do very well, approaching 60 fps in the first graphics test. The Mali 400MP4, used in the Galaxy Note 2 and without a lot of vertex processing power, brings up the rear - being outperformed by even NVIDIA's Tegra 3. ARM's Mali-T604 isn't enough to pull ahead in this test either; the Nexus 10 remains squarely behind the top two Adreno 320 based devices.

Graphics Test 2

Graphics test 2 stresses the hardware’s ability to process lots of pixels. It tests the ability to read textures, do per pixel computations and write to render targets.

On average, 12.6 million pixels are processed per frame. The additional pixel processing compared to Graphics test 1 comes from including particles and post processing effects such as bloom, streaks and motion blur.

In each frame, an average 75,000 vertices are processed. This number is considerably lower than in Graphics test 1 because shadows are not drawn and the processed geometry has a lower number of polygons.

3DMark - Graphics Test 2

As you'd expect, shifting to a more pixel shader heavy workload shows the Galaxy Note 2 doing a lot better - effectively tying the Tegra 3 based HTC One X+ and outperforming the Nexus 7. The Mali-T604 continues to, at best, tie for third place here. Qualcomm's Adreno 320 just seems to deliver better performance in 3DMark for Android.

3DMark - Graphics

The overall score pretty much follows the trends we saw earlier. Qualcomm's Adreno 320 leads things (Nexus 4/HTC One), followed by ARM's Mali-T604 (Nexus 10), Adreno 225 (SGS3), Adreno 305 (One SV), Tegra 3 (One X+/Nexus 7) and finally Mali 400MP4 (Note 2). The only real surprise here is just how much better Adreno 320 does compared to Mali-T604.

Physics Test

The purpose of the Physics test is to benchmark the hardware’s ability to do gameplay physics simulations on CPU. The GPU load is kept as low as possible to ensure that only the CPU’s capabilities are stressed.

The test has four simulated worlds. Each world has two soft bodies and two rigid bodies colliding with each other. One thread per available logical CPU core is used to run simulations. All physics are computed on the CPU with soft body vertex data updated to the GPU each frame. The background is drawn as a static image for the least possible GPU load.

The Physics test uses the Bullet Open Source Physics Library.

3DMark - Physics

3DMark - Physics Test

The physics results give us an indication of just how heavily threaded this benchmark is. The quad-core devices are able to outperform the dual-core Cortex A15 based Nexus 10, despite the latter having far better single threaded performance. The Droid DNA/Optimus G vs. Nexus 4 results continue to be a bit odd, perhaps due to the newer drivers included in the Nexus 4's use of Android 4.2 vs. 4.1.2 for the other APQ8064 platforms.

The overall Ice Storm results don't paint a vastly different picture from what we've seen thus far:

3DMark - Ice Storm

We will likely have to revisit this as non-Nexus devices get updated to Android 4.2, especially given the strange delta between the Nexus 4 and Optimus G/Droid DNA. When the One gets updated to Android 4.2.x, it'll be interesting to see if it pulls ahead of the Nexus 4 in 3DMark.

It's also interesting to note that 3DMark seems to value the Adreno 320 more than ARM's Mali-T604 in the Nexus 10. I tossed results from the HTC One into the T-Rex HD benchmark data from GL/DXBenchmark 2.7 and found that the two benchmarks seem to agree on this point as well:

GL/DXBenchmark 2.7 - T-Rex HD (Offscreen)

The magnitude of difference between the One (Adreno 320) and Nexus 10 (Mali-T604) isn't as large, but we are looking at two very different benchmarks here.

3DMark includes an optional demo run that will generate a score as well (although these results aren't included in the overall Ice Storm score):

3DMark - Demo

Just for kicks I ran the extreme (1080p, high quality textures and post processing) presets on a couple of devices to see how the numbers changed:

3DMark for Android - Default vs. Extreme Settings Comparison
  HTC One Google Nexus 10 Google Nexus 7
  Default Extreme Default Extreme Default Extreme
Ice Storm Score 11309 6190 7990 5499 3539 1895
Graphics Score 11250 5942 7952 5030 3130 1581
Physics Score 11520 7246 8127 8165 6527 6196
Graphics Test 1 49.2 fps 29.8 fps 36.9 fps 27.3 fps 12.2 fps 7.5 fps
Graphics Test 2 48.7 fps 22.8 fps 32.5 fps 18.2 fps 15.3 fps 6.4 fps
Physics Test 36.6 fps 23.0 fps 25.8 fps 25.9 fps 20.7 fps 19.7 fps
Demo 52.5 fps 25.7 fps 35.2 fps 21.5 fps 16.8 fps 7.0 fps

Moving up to the Extreme presets definitely increases the workload on these devices. It's important to note that today's flagships are close to running 3DMark at extreme presets at 30 fps in some cases. On average, we're looking at low to mid 20 fps for the best of the best on Android today.

3DMark for Android seems to be another good data point to have. While the holy grail remains a good collection of actual games with integrated benchmarks representative of game play, at least having a good selection of simulated game tests is a lot better than where we were just a year ago.

The fact that all of 3DMark's benchmarks are comparable across devices regardless of display resolution is a nice touch, although I do appreciate GLBenchmark's ability to target the native resolution of any device (especially when dealing with very high res tablets). The inclusion of a dedicated physics test is very nice, however I'd like to see equal emphasis placed on single (or lightly) threaded performance as the real world tends to not be as thread happy as 3DMark's physics test implies.

Overall, the 3DMark Ice Storm workload is clearly less stressful than GL/DXBenchmark, which makes me wonder how it'll age over time. On the bright side, 3DMark will make a nice companion to GL/DXBenchmark - offering a glimpse at what GPU performance would look like in workloads closer to what is presently available, rather than forecasting what may be possible in the future. We plan on using both benchmarks in our reviews going forward.

POST A COMMENT

56 Comments

View All Comments

  • g00ey - Tuesday, April 02, 2013 - link

    The Google Nexus 10 has a considerably higher screen resolution (2560x1600) than the Nexus 4 (768x1280) or the HTC One (1080p). This might explain why the Mali T604 doesn't look so good on these benchmarks. Reply
  • MonkeyPaw - Tuesday, April 02, 2013 - link

    If I understand the article right' resolution doesn't matter. Scenes are drawn at a set res and then scaled to native. The only overhead will be from the scaling. Reply
  • bearxor - Tuesday, April 02, 2013 - link

    Uhhh... Yeah.

    Someone didn't read the article and instead decided to skim the charts.
    Reply
  • theduckofdeath - Tuesday, April 02, 2013 - link

    The GPU and processor still has to work harder to actually output the extra pixels, even though the rendering is done at a specific resolution. With a bit of reverse calculations, someone with a bit of extra time should be able to calculate how much of an impact actual output resolution has on a benchmark like this. though, maybe it's not that easy, as different devices with "the same internals" but different resolution displays might have memory with different performance, for instance. Reply
  • Exophase - Tuesday, April 02, 2013 - link

    Mobile SoCs often have support hardware for scaling a screen overlay to native resolution. This is often built into the display controller so there's no performance impact even to available memory bandwidth.

    No idea if such a thing is actually used here though.
    Reply
  • ellis2323 - Friday, May 03, 2013 - link

    I'm a gamedev (DX, GL, OGLES...).This a OpenGL ES 2.x scene. Rendering is done on a FBO and your draw it on screen with 2 triangles. So the resolution impacts the memory bandwidth. The hardware scaling (if available) is available for compositing but for Android UI, not applications. Reply
  • blackmagnum - Tuesday, April 02, 2013 - link

    Very interesting. Will people pick their mobile devices only based on performance, or they are more concerned about looks. What is the optimum balance? Reply
  • xTRICKYxx - Tuesday, April 02, 2013 - link

    I think everyone wants a fast device; but as we all know, benchmarks don't mean everything. I would prefer the Nexus 10's dual A15 over the quad krait cores. Reply
  • alexvoda - Wednesday, April 03, 2013 - link

    Performance is not the only important factor.
    In addition to performance (CPU & GPU) features are very important (screen resolution, microSD, battery size&removability, radios).
    Also basic characteristics (device size) are probably the most important.
    And almost forgot, the software.
    Add to this the price and performance isn't such a major criteria any more.

    For example: I want a device that I can use single handedly so all 1080p 5" devices are way too big for me. As such I prefer a 4.2"-4.3" screen size. I don't really care about 1080p but I really want no less than 720p. I find MicroSD to be a must. I accept built-in batteries but I prefer removable ones.
    For my use case no phone available (at the time of purchase - December) is good enough so top performance is not as important since it isn't available. (I watch anime encoded in h264 Hi10P which can not be hardware decoded and must be software decoded. Not even the Exynos A15 Dual and the Snapdragon S4 Pro can handle it. Maybe just the new maybe just the Exynos Octa and Snapdragon 600).
    I currently have a Sony Xperia T which was very cheap when I bought it. It kinda satisfies those criteria but not perfectly. At 4.6" it is still too big, and the battery drains very fast.
    Right now the ideal device for me would probably be the Xiaomi M2S (http://www.engadget.com/2013/04/03/xiaomi-m2s-with... but I'm not so sure about the software and I doubt I could easily get service in case it breaks. I will probably wait for this years nexus and decide if it is worth sacrificing the microSD slot.
    Reply
  • alexvoda - Wednesday, April 03, 2013 - link

    link didn't work because of extra ")".
    Xiaomi M2S http://www.engadget.com/2013/04/03/xiaomi-m2s-with...
    Reply

Log in

Don't have an account? Sign up now