Testing the latest x86 rack servers and low power server CPUs
by Johan De Gelas on July 22, 2009 2:00 AM EST- Posted in
- IT Computing
Final Verdict: the Servers
Before we can draw conclusions, we need to look at the pricing of these barebones servers. We assume that hardware costs matter to you. In other words, we assume that you are not using expensive software like the enterprise versions of ESX and Oracle.
The European pricing is the end-user pricing as provided by Triple-S, a server and storage (BTO) vendor that is active in France, Belgium, and the Netherlands. The US pricing was gathered by looking at the pricewatch site. All prices exclude any taxes.
| Server Pricing Comparison | |||||
| Server Chassis | Comments | European pricing | US Price | ||
| Price (Euro) | Price per Server (Euro) | Price | Price per Server (US Dollar) | ||
| Supermicro Twin2 Nehalem | |||||
| SYS-6026TT-FF | No hot pluggable nodes, no Infiniband | € 2116 | € 529 | $2429 | $607 |
| SYS-6026TT-iBRF | Hot pluggable nodes, no Infiniband | € 2413 | € 603 | $2845 | $711 |
| SYS-6026TT-iBXF | No hot pluggable nodes, DDR Infiniband | € 3138 | € 785 | $3659 | $915 |
| SYS-6026TT-BIBXRF | Hot pluggable nodes, DDR Infiniband | € 3297 | € 824 | $4095 | $1024 |
| SYS-6026TT-BIBQF | Hot pluggable nodes, QDR Infiniband | € 3503 | € 876 | $4159 | $1040 |
| Supermicro Twin2 Istanbul | |||||
| PIO-2012TM-BTRF | € 2117 | € 529 | N/A | ||
| PIO-2012TM-BiBXRF | € 3056 | € 764 | N/A | ||
| ASUS | |||||
| ASUS RS700D6-E6/PS8 "Duo" | Dual node | € 1526 | € 763 | $1500 | $750 |
| ASUS RS700-E6/RS4 | Single node, cableless | € 1138 | € 1138 | $1150 | $1150 |
| Chenbro RM130204 "Willowbrook" | € 700 | € 700 | $750 | $750 | |
| Supermicro A+ Server 1021M-UR | € 795 | € 795 | $998 | $998 | |
The ASUS RS700D6-E6/PS8 or "ASUS Duo" offers very low power consumption and high quality power regulation. To really stand out, the ASUS Duo server should adopt the cableless design and the "easy to remove fans" of the ASUS RS700-E6/RS4. The latter is a bit more expensive but still one of the best x86 rack servers on the market. If you need more than one expansion slot and want optimum serviceability, the ASUS RS700-E6 is an excellent choice. The ASUS RS700D6-E6/PS8 is well suited for the HPC market but falls short in its other target market: virtualization. The two power supplies should be redundant and hot swappable. The Duo server should also be as easy to service as its single node brother.
The Chenbro RM130204 plus Intel Willowbrook comes with excellent ideas to lower power consumption, but the implementation can't compete with the best Supermicro and ASUS servers. Low power is nice, but as you can see in the results in this article, a clever design such as the dual PSUs in the Supermicro Twin2 that power four nodes can lower power even more. The Willowbrook based server is limited to only 32GB if you do not want to use expensive 8GB DIMMs and does not offer power supply redundancy. When it comes to virtualization, the Chenbro Willowbrook combination is rather mediocre. However, if you are looking to run a server version of Windows on this one, it is close to impossible to find a server that consumes so little power. Idle power can be as low as 70W if you use Windows 2003 SP2 and the hot patches that support the deep C-states of the latest Xeons. This server is almost inaudible when running under load and is whisper silent when running idle. The Chenbro RM130204 is a server you can afford to run outside the data center.
The Supermicro Twin2 offers very low power consumption and PSU redundancy at a very attractive price per node. This server is a serious alternative for all those who like the density of blade servers but don't want to pay the high prices and yearly software licenses. A typical blade server (with 10 blades) is about three times more expensive per server, and I/O modules (such as a gigabit Ethernet switches) easily cost twice as much as the stand-alone versions. The HPC people will be very pleased with the Infiniband solutions (the DDR version is able to come close to its maximum bandwidth). Supermicro reports that they already have several large-scale (2000+ servers) deployments.
The Twin2 is probably the best solution on the market if you are searching for a low cost, high density, and redundant server… but it is not perfect. Fans should be easier to remove and hot swappable. We would like to see support for 8GB DIMMs across all Twin2 versions, not just a few SKUs. We also want a version with 2.5" disks so we can use SSDs if necessary. Supermicro tells us that the Twin2 will be available with 2.5" disks later this year.
The Supermicro A+ Server 1021M-UR is AMD's current favorite for low power Opteron servers, but not for long. It is clear that the NVIDIA based platform is getting old, feature- and power-wise. We look forward to seeing the Opteron EE and especially the six-core HE in action on a "Fiorano Kroner" platform. Just like Intel's Willowbrook, the power regulation will be optimized for low power CPUs and will not support the more power hungry CPUs.
Facebook
Twitter
RSS
12 Comments
View All Comments
Doby - Thursday, July 23, 2009 - link
I don't understand why virtualization benchmarking is done with 16 or fewer VMs. With the CPU power of the newer CPU you can consolidate far more on there. Why aren't the benchmarks done with VMs with varying workloads, around 5% or less utilization, and then see how many VMs a particular server can handle. It would be far more real world.I have customers running over 150 VMs on a 4 CPU box, the performance compison of which CPU can handle 16 VMs better is completely bogus. It's all about how many VMs can I get without overloading the server (60-80% utilization).
JohanAnandtech - Thursday, July 23, 2009 - link
As explained in the article, we were limited with the amount of DDR-3 we have available. We had a total of 48 GB of DDR-3 and had to test up to servers. It should not be too hard to figure out what the power consumption could have been with twice or even four times more memory. Just add 5 Watt per DIMM.BTW, 150 VMs on one box is not extremely rare in the realworld. Are those VDI VMs?
"the performance comparison of which CPU can handle 16 VMs better is completely bogus"
On a dual socket machine it is not. Why would it be "bogus"? I agree that in a perfect world we would have loaded that machine up to 48 GB per Server (that is a fortune of 192 GB of RAM) and have run like 20-30 VMs per server. A little bit of understanding for the limitations we have to face would make my day....
uf - Thursday, July 23, 2009 - link
What power consumption is for low loaded server (not idle!) say at 10% and 30% average cpu utilization per core?MODEL3 - Wednesday, July 22, 2009 - link
in your comment:If AMD would apply the methodology of Intel to determine TDP they would end up somewhere between ACP and the current "AMD TDP"
You referring exclusively to the server CPUs?
Because if not, the above statement is false and unprofessional.
I don't have access to server CPUs but from my experience with mainstream consumer CPUs tells me the exact opposite:
65nm dual core (same performance level) 65W max TDP:
both 6420 (2,13GHz)& 4600 (2,4GHz) has lower* actual TDP than 5600 (2,9Ghz)
45nm dual core (same performance level) 65W max TDP:
both 7200 (2,53GHz)& 6300 (2,8GHz) has lower* actual TDP than Athlon 250 (3,0Ghz)
45nm Quad core (same performance level) 65W max TDP:
Q8200S (2,33 GHz) has lower* actual TDP than Phenom II 905e (2.5GHz)
I don't even need to give details for system configurations everyone knows these facts.
* not by much but nevertheless lower (so from that point to the point of " AMD has actual TDP somewhere between AMD's ACP and Intel's TDP " there is a huge gap
JohanAnandtech - Wednesday, July 22, 2009 - link
Correct. I only checked for server CPUs (see the pdf I linked).JarredWalton - Wednesday, July 22, 2009 - link
There are several issues at work, particularly with desktop processors. For one, AMD and Intel both have a range of voltages on desktop parts, so (just throwing out numbers) one CPU might run at 1.2V and another with the same part might run at 1.225V - it's a small difference but it can show up.Next, Intel and AMD both seem to put out numbers that are a theoretical worst case, and clock speed and voltage of a given chip help determine where the CPUs actually fall. The stated TDP on a part might be 65W, and with some 65W chips you can get very close to that while with others you might never get above 50W, but they'll both still state 65W.
The main point is that AMD's ACP ends up lower than what is realistic and their TDP ends up as essentially the worst-case scenario. (AMD parts are marketed with the ACP number, not TDP.) Meanwhile, Intel's TDP is higher than AMD's ACP but isn't quite the worst-case scenario of AMD's TDP.
I believe that's the way it all works out: Intel reports TDP that is lower than the absolute maximum but is typically higher than most users will see. AMD reports ACP that is more like an "average power" instead of a realistic maximum, but their TDP is pretty accurate. Even with this being the general case, processors are still released in families and individual chips can have much lower power requirements than the stated ACP/TDP - basically they should always come in equal to or lower than the ACP/TDP, but one might be 2W lower and another might be 15W lower and there's no easy way to say which it is without testing.
MODEL3 - Wednesday, July 22, 2009 - link
I mostly agree with what you 're saying except 2 things:1.AMD's TDP ends up as essentially the worst-case scenario (not true in all the cases e.g. Phenom X4 9350e (it has actual TDP higher than 65W)
2.In all the examples I gave, Intel & AMD had the same "official" TDP (also same more or less performance & same manufacturing proccess) so with your logic AMD should have lower than Intel actual TDP which is not true.
I live in Greece, here we pay 0,13€ (inc. VAT) per KW, so...
In another topic did you see the new prices for AMD Athlon II X2 245 (66$) & 240 (60$)? (while Intel 5300 cost 64$ & 5400 74$)
They should have priced them at 69$ & 78$.
No wonder why AMD is loosing so much money, they have to fire immediately those idiots who dit it (it reminds me the days before K8 when AMD used these methods)
JPForums - Wednesday, July 22, 2009 - link
I'm having a hard time correlating your chart and your assessment."Notice how adding a second L5520 CPU and three DIMMs of DDR3-1066 to our Chenbro server only adds 9W."
Found that one. However, on the previous page you make this statement:
"So adding a Xeon X5570 adds about 58W (248W - 175W - three DIMMs of 5W), while adding an Opteron 2435 2.6GHz adds about 47W (243 - 181 - three DIMMs of 5W)."
This implies to me that just adding the 3 DIMMs should have raised the power 15W.
"Add an Opteron EE to our AMD server and you add 22W."
Check. Did you add the 3 DIMMs here as well?
"The result is that the best AMD platform consumes about 20W more than the best Intel platform when running idle."
Can't find this one. There are 3W difference between the Xeon L5520 and the Opteron 2377 EE. There are 16W difference for the dual CPU counter parts (closer). All the other comparisons leave the Intel platform consuming more power than the AMD counterpart. Is this supposed a comparison of the platform without the CPU? It is unclear to me given the words chosen. I was under the impression that the CPU is generally considered part of the platform.
"Intel's power gating is the decisive advantage here: it can turn the inactive cores completely off. Another indication is that the dual Opteron 2435 consumes about 156W when we turn off dynamic power management, which is higher than the Xeon X5570 (150W)."
An explanation of dynamic power management would be helpful. It sounds like you're saying that Intel's power management techniques are clearly better because when you turn both their power management and AMD's power management off, the Intel platform works better. The only way your statements make sense is if the dynamic power management you are talking about isn't a CPU level feature like clock gating. In any case, power management techniques are worthless if you can't use them.
As a side question, when the power management support issue with the Xeon X5570 is addressed and AMD has a new lower power platform, where do you predict the power numbers will end up? I'd still expect the "Nahalem" Xeons to win in performance/power, though.
JohanAnandtech - Wednesday, July 22, 2009 - link
Part 2 :-)"The result is that the best AMD platform consumes about 20W more than the best Intel platform when running idle."
Can't find this one. "
135W - 119W = 16W. I made a small error there (spreadsheet error).
"It sounds like you're saying that Intel's power management techniques are clearly better because when you turn both their power management and AMD's power management off, the Intel platform works better. "
More or less. There are two ways the CPU can save power: 1) lower voltage and clockspeed or 2) Shut down the cores that you don't need. In case of the Intel part, it is better at shutting down the cores that it don't need. They simply are completely shut off and consume close to 0 W. In case of AMD, each core still consumes a few watt.
So if you turn Speedstep and power now! off, you can see the effect of the 2nd way to save power. It confirms our suspicion of why the Opteron EE is not able to beat the L5520 when running idle.
JohanAnandtech - Wednesday, July 22, 2009 - link
I'll chop my answers up to keep these comments readable.quote:
"Notice how adding a second L5520 CPU and three DIMMs of DDR3-1066 to our Chenbro server only adds 9W."
Found that one. However, on the previous page you make this statement:
"So adding a Xeon X5570 adds about 58W (248W - 175W - three DIMMs of 5W), while adding an Opteron 2435 2.6GHz adds about 47W (243 - 181 - three DIMMs of 5W)."
This implies to me that just adding the 3 DIMMs should have raised the power 15W. "
No. Because the 9W is measured at idle. It is too small to measure accurately, but DIMMs do not consume 5W per DIMM in idle. Probably more like 1W or so.