A Look At Triple-GPU Performance And Multi-GPU Scaling, Part 1
by Ryan Smith on April 3, 2011 7:00 AM ESTFitting Three Video Cards in an ATX Case
I thought we’d flip our normal GPU review style on its head by starting with Power, Temperature, and Noise first. NVIDIA and AMD have both long recommended against placing high-end video cards directly next to each other, in favor of additional spacing between video cards. Indeed this is a requirement for their latest dual-GPU cards, as both the GTX 590 and 6990 draw relatively massive amounts of air using a fan mounted at the center of the card and exhaust roughly half their air inside of the case. Their reference style single-GPU cards on the other hand are fully exhausting with fans mounted towards the rear of the card. Thus multi-GPU configurations with the cards next to each other is supposed to be possible, though not ideal.
There’s a reason I want to bring this up first, and a picture is worth a thousand words.
While AMD and NVIDIA’s designs share a lot in common – a rear-mounted blower fan pushes air over a vapor chamber cooler – the shrouds and other external equipment are quite different. It’s not until we see a picture that we can appreciate just how different they are.
With the Radeon HD 6000 series, AMD’s reference designs took on a very boxy design. The cards fully live up to the idea of a “black box”; they’re enclosed on all sides with a boxy cooler and a black metal backplate. As a GPU reviewer I happen to like this design as the GPUs are easy to stack/store, and the backplate covers what would normally be the only exposed electronics on the card. The issue with this boxy design is that AMD is taking full advantage of the PCIe specification, leading to the 6900 series being the full width allowed.
NVIDIA on the other hand has always had some kind of curve in their design, normally resulting in a slightly recessed shroud around the blower intake. For the GTX 580 and GTX 570 they took a further step in recessing the shroud around this area, leading to the distinct wedge shape. At the same time NVIDIA does not use a backplate, saving precious millimeters of space. The end result of this is that even when packed like sardines, the GTX 580 and GTX 570 blowers have some space reserved for air intake.
The Radeon HD 6970 does not, and this is our problem. The picture of the 6970 in triple-CF really paints the picture, as the middle card is directly pressed up against the top card. Because these cards are so large and heavy the rear ends tend to shift and dip some when installed against a vertical motherboard – in fact this is why we can normally get away with a dense dual-CF setup since the bottom card dips a bit more – but in a triple-CF configuration the end result is that one of the cards will end up getting up-close and personal with another one.
Without outside intervention this isn’t usable. We hit 99C on the middle card in Crysis when we initially installed the three cards, and Crysis isn’t the hardest thing we run. For the purposes of our test we ultimately resorted to wedging some space between the cards with wads of paper, but this isn’t a viable long-term solution.
Unfortunately long-term alternatives are few if you want to give a triple-GPU setup more space. Our testbed uses an Asus Rampage II Extreme, which features three PCIe slots mixed among a total of 6 slots; the way it’s laid out makes it impossible to have our triple-GPU configuration setup in any other manner. Even something like the ASRock P67 Extreme4 can’t escape the fact that the ATX spec only has room for 7 slots and that when manufacturers actually use the 7th and topmost slot that it’s a short PCIe x1 slot. In short you won’t find an ATX motherboard that can fit three video cards and at the same time gives each one a slot’s worth of breathing room. For that you have to use a larger than ATX form factor.
So what’s the point of all of this rambling? With AMD’s current shroud design it’s just not practical to do triple-CF on air on an ATX motherboard. If you want to play with three AMD boards you need to think outside of the box: either use water cooling or use a larger motherboard.
Facebook
Twitter
RSS
97 Comments
View All Comments
DanNeely - Sunday, April 3, 2011 - link
Does DNetc not have 4xx/5xx nVidia applications yet?Pirks - Sunday, April 3, 2011 - link
They have CUDA 3.1 clients that work pretty nice with Fermi cards. Except that AMD cards pwn them violently, we're talking about an order of magnitude difference between 'em. Somehow RC5-72 code executes 10x faster on AMD than on nVidia GPUs, I could never find precise explanation why, must be related to poor match between RC5 algorithm and nVidia GPU architecture or something.I crack RC5-72 keys on my AMD 5850 and it's amost 2 BILLION keys per second. Out of 86,000+ participants my machine is ranked #43 from the top (in daily stats graph but still, man...#43! I gonna buy two 5870 sometime and my rig may just make it to top 10!!! out of 86,000!!! this is UNREAL man...)
On my nVidia 9800 GT I was cracking like 146 million keys per second, this very low rate is soo shameful compared to AMD :)))
DanNeely - Monday, April 4, 2011 - link
It's not just dnetc, ugly differences in performance also show up in the milkeyway@home and collatz conjecture projects on the boinc platform. They're much larger that the 1/8 vs 1/5(1/4 in 69xx?) differences in FP64/FP32 between would justify; IIRC both are about 5:1 in AMD's favor.Ryan Smith - Sunday, April 3, 2011 - link
I love what the Dnet guys do with their client, and in the past it's been a big help to us in our articles, especially on the AMD side.With that said, it's a highly hand optimized client that almost perfectly traces theoretical performance. It doesn't care about cache, it doesn't care about memory bandwidth; it only cares about how many arithmetic operations can be done in a second. That's not very useful to us; it doesn't tell us anything about the hardware.
We want to stick to distributed computing clients that have a single binary for both platforms, so that we're looking at the performance of a common OpenCL/DirectCompute codepath and how it performs on two different GPUs. The Dnet client just doesn't meet that qualification.
tviceman - Sunday, April 3, 2011 - link
Ryan are you going to be using the nvidia 270 drivers in future tests? I know they're beta, but it looks like you aren't using WHQL AMD drivers either (11.4 preview).Ryan Smith - Sunday, April 3, 2011 - link
Yes, we will. The benchmarking for this article was actually completed shortly after the GTX 590, so by the time NVIDIA released the 270 drivers it was already being written up.ajp_anton - Sunday, April 3, 2011 - link
When looking at the picture of those packed sardines, I had an idea.Why don't the manufacturers make the radial fan hole go all the way through the card? With three or four cards tightly packed, the middle card(s) will still have some air coming through the other cards, assuming the holes are aligned.
Even with only one or two cards, the (top) card will have access to more fresh air than before.
semo - Sunday, April 3, 2011 - link
Correct me if I'm wrong but the idea is to keep the air flowing through the shroud body and not through and through the fans. I think this is a moot point though as I can't see anyone using a 3x GPU config without water cooling or something even more exotic.casteve - Sunday, April 3, 2011 - link
"It turns out adding a 3rd card doesn’t make all that much more noise."Yeah, I guess if you enjoy 60-65dBA noise levels, the 3rd card won't bother you. Wouldn't it be cheaper to just toss a hairdryer inside your PC? You'd get the same level of noise and room heater effect. ;)
slickr - Sunday, April 3, 2011 - link
I mean mass effect 2 is a console port, Civilisation 5 is the worst game to choose to benchmark as its a turn based game and not real time and Hawx is 4 years outdated and basically game that Nvidia made, its not even funny anymore seeing how it gives the advantage to Nvidia cards every single time.Replace with:
Crysis warhead to Aliens vs predator
battleforge to shogun 2 total war
hawx to Shift 2
Civilization 5 to Starcraft 2
Mass Effect 2 with Dead Space 2
Wolfeinstein to Arma 2
+add Mafia 2