Real-world virtualization benchmarking: the best server CPUs compared
by Johan De Gelas on May 21, 2009 3:00 AM EST- Posted in
- IT Computing
Introduction
TCO and ROI have been abused repeatedly by sales representatives, in the hope of getting you to swallow the sometimes outrageously high pricing on their quotation for a trendy new technology. However, server virtualization is one of the few ICT technologies that really lives up to its hype. The cost savings are real and the TCO is great, as long as you obey a few basic rules like not installing bloatware or extremely large memory limited databases. There is more.
Server consolidation is superb for the IT professional who is also a hardware enthusiast (and thus reads it.anandtech.com ?). Hardware purchases used to be motivated by the fact that the equipment was written off or because the maintenance contract was at the end of its life. Can you even think of a more boring reason to buy new hardware? The timeframe between the beginning of the 21st century and the start of commercially viable virtualization solutions was the timeframe where the bean counters ruled the datacenter. Few people were interested in hearing how much faster the newest servers were, as in most cases the extra processing power would go to waste 95% of the time anyway.
Now with virtualization, we hardware nerds are back with a vengeance. Every drop of performance you wring out of your servers translates into potentially higher consolidation ratios (more VMs per physical machine) or better response time per VM. More VMs per machine means immediate short- and long-term cost savings, and better performance per VM means happier users. Yes, performance matters once again and system administrators are seen as key persons, vital to accomplishing the business goals. But how do you know what hardware you should buy for virtualization? There are only two consolidation benchmarks out there: Intel's vConsolidate and VMware's VMmark. Both are cumbersome to set up and both are based on industry benchmarks (SPECJbb2005) that are only somewhat or even hardly representative of real-world applications. The result is that VMmark, despite the fact that it is a valuable benchmark, has turned into yet another OEM benchmark(et)ing tool. The only goal of the OEMs seems to be to produce scores as high as possible; that is understandable from their point of view, but not very useful for the IT professional. Without an analysis of where the extra performance comes from, the scores give a quick first impression but nothing more.
Yes, this article is long overdue, but the Sizing Servers Lab proudly presents the AnandTech readers with our newest virtualization benchmark, vApus Mark I, which uses real-world applications in a Windows Server Consolidation scenario. Our goal is certainly not to replace nor to discredit VMmark, but rather to give you another data point -- an independent second opinion based on solid benchmarking. Combining our own testing with what we find on the VMmark page, we will be able to understand the virtualization performance landscape a bit better. Before we dive into the results, let's discuss the reasoning behind some of the choices we made.
Facebook
Twitter
RSS
66 Comments
View All Comments
JohanAnandtech - Friday, May 22, 2009 - link
We are definitely interesting in doing this, but of course we like to do this well. I'll update as soon as I can.pc007 - Thursday, May 21, 2009 - link
I agree, CPU & RAM usage are not really bottlenecks in my experience. Processes hamering slow disk and making everything else slower is the main concern.SeanG - Friday, May 22, 2009 - link
There are 300 million people in this country and you're surprised that some of them are ignorant/jerks/crazy? We're all supposed to be ashamed because not everyone from this country is mentally stable? It's insulting to people like me who care about this country to hear you talk about being ashamed over something that is a problem with humanity in general and not only in the US.lopri - Thursday, May 21, 2009 - link
It is said to see such a fascist persona in this comment section of such a fascinating article. I feel ashamed as one residing in the U.S.JohanAnandtech - Friday, May 22, 2009 - link
Don't be. Tshen must be the first US citizen that I have encountered that hates Belgians :-). All other US people I have met so far were very friendly. In fact, I am very much astonished how hospitable US people are. Sometimes we have only spoken over the phone or via e-mail with each other and the minute I arrive in the US, we are having a meal and chatting about IT. When you arrive in Silicon valley, one can only be amazed about the enormous energy and entrepreneurship this valley breathes.tshen83 - Friday, May 22, 2009 - link
You are a slave, Johan, whether you realize it or not. The people in Silicon Valley are "nice to you" because they are in the process of negotiating a purchased piece of publication from you.You don't know anything about the Silicon Valley nor are you qualified to talk about it. If anything is true, Silicon Valley is in the toilet right now with bankruptcies everywhere. The state is broke, with Arnold Schwarzenegger begging for Federal bailouts. The last two big "entrepreneurships" coming out of Silicon Valley: Facebook and Twitter are both advertising scams without a viable business model.
I don't hate Belgians. I do hate retards like you whether you come from Belgium or not.
[BANNED]
[FROM JARRED: We are proponents of free speech, but repeated name calling and insults with little to no factual information to back up claims will not be tolerated. There was worse, and I'm leaving this text so you can see how it started.]
tshen83 - Thursday, May 21, 2009 - link
As what I have expected, Johan, your sorry ass came up with a benchmark that invalidates VMmark.On page 9 "Nehalem vs Shanghai" http://it.anandtech.com/IT/showdoc.aspx?i=3567&...">http://it.anandtech.com/IT/showdoc.aspx?i=3567&...
Where is the Nehalem vs Shanghai benchmarks? All I see is a chart pumping Opteron 8389.
Let me dissect the situation for you [EDITED FOR VULGARITY].
The 100% performance per watt advantage witnessed by the Nehalem servers was the result of 3 factors: triple channel DDR3 IMC, HyperThreading, and Turbo Boost. The fact that Opterons can no longer compete because they lack the raw bandwidth and the "fake Hyper Threaded" cores that performs like a real core.
What would AMD do in this situation? Of course, invalidate an industrially accepted benchmark by substituting it with a "paid third party" benchmark that isn't available to the the public. I wonder what kind of "optimizations" were done?
You know what killed the GPU market? HardOCP.com. That's right, they invalidated the importance of 3DMark by doing game by game FPS analysis. The problem with this approach is that the third party game developers really don't have the energy or resources to make sure that each GPU architecture is properly optimized for. As long as the games run about 30fps on both Nvidia and ATI GPUs, they are happy. What results from this lackluster effort is that there is no Frames per Second differentiation on the GPU vendors, causing prices to free fall and the idiots choosing an architecturally inferior ATI GPU that gave a similar FPS performance.
Same methodology can be applied here. Since the Opterons lack raw memory bandwidth and core count visible to the OS, why not have a benchmark that isn't threaded well enough, and stress on high CPU utilization situations where memory bandwidth and core count matter less? That is what this new benchmark is doing, hiding Opteron architectural difficiencies.
The reason why VMmark stresses high number of VMs is to guage the hardware acceleration of VM switching. Having lesser number of VMs doing high CPU workload helps the worse performer(Opteron) by hiding and masquerading the performance difficiency. Nobody runs 100 VMs on one physical machine, but the VMmark does show you a superior hardware implementation. Nobody really prevents AMD from optimizing their CPUs for VMmark.
Let me be even more brutal with my assessment of your ethics, Johan. Why do you feel you are qualified to do what you do? The people who actually know about hardware are doing the CPU designs themselves in the United States, so the Americans would be the first to know about hardware. When the CPU samples are sent to Taiwan for motherboard design, the Asians would be the second batch of people dealing with hardware. By the time hardware news got to freaking Europe(Fudzilla, The Inq), the information usually was fudged up to the wazoos by Wall Street analysts. Consider yourself lucky that the SEC isn't probing you [EDITED FOR VULGARITY] because you reside in Belgium.
So Johan, my suggestion for you personally, is that you should consider the morality of your publications. In today's day and age, every word you ever say is recorded for eternity. Thirty years from now, do you want people to call you a [EDITED FOR VULGARITY] for pumping an inferior architecture by fudging benchmark results? Of course, I personally run the same risks. What I can guarantee you is that by June of next year, Johan, you [EDITED FOR VULGARITY] would be pumping Via instead.
JohanAnandtech - Friday, May 22, 2009 - link
As long as you are not able to discuss technical matters without personal attacks, I won't waste much time on you. Leave the personal attacks out of your comments, and I'll address every concern you have.But for all other readers, I'll show how shallow your attacks are (but they probably figured that one out a long time ago).
"Why not have a benchmark that isn't threaded well enough"
Yes, Tshen. In your world, Oracle and MS SQL server have few threads. In the realworld however...
"http://it.anandtech.com/IT/showdoc.aspx?i=3567&...">http://it.anandtech.com/IT/showdoc.aspx?i=3567&...
Where is the Nehalem vs Shanghai benchmarks? All I see is a chart pumping Opteron 8389. "
All other readers have seen a chart that tries to show how the benchmark reacts to cache size and memory bandwidth. All other readers understand that we only have one Nehalem Xeon, and that is a little hard to show empirically how for example different cache sizes influence the benchmark results.
Lastly, as long as I publish AMD vs Intel comparisons, some people will call me and Anandtech biased. This article shows that Nehalem is between 50 to 80% faster in typical server apps.
http://it.anandtech.com/IT/showdoc.aspx?i=3536&...">http://it.anandtech.com/IT/showdoc.aspx?i=3536&...
For some people that meant that we were biased towards Intel. In your case, we are biased towards AMD if Intel does not win by a huge percentage. For the rest of the world, it just means that we like to make our benches as realworld as possible and we report what we find.
whatthehey - Friday, May 22, 2009 - link
At least we can be grateful he's kind enough to include his IQ in his user name. You know what's interesting? tshen83 isn't exactly a common user name, and he happens to troll elsewhere:http://www.google.com/search?hl=en&q=tshen83">http://www.google.com/search?hl=en&q=tshen83
"Thirty years from now, do you want people to call you a fucking asshole...?" No need for you to wait 30 years, tshen; we'll be happy to call you a fucking asshole right now. As the saying goes: if the shoe fits....
JarredWalton - Thursday, May 21, 2009 - link
Frankly, you make me sad to be an American - as though just because someone is located in Belgium they are not qualified to do anything with hardware? Let's see, Belgium has higher average salaries than the US, so they surely have to be less qualified. And with you as a shining example we can certainly tell EVERYONE in the US is more qualified than in Europe.To whit, your assertion that HardOCP - or any other site - "killed the GPU market" is absurd in the extreme. The GPU market has seen declining prices because of competition between ATI and NVIDIA, and because the consumer isn't interested in spending $500 every 6-12 months on a new GPU. However, ATI and NVIDIA are hardly dying... though ATI as part of AMD is in a serious bind right now if things don't improve. Thankfully, AMD has helped the CPU market reach a similar point, but with Core i7 we're going back to the old way of things.
Your linking to page nine of this article as though that's somehow proof of bias is even better. Johan shows that Nehalem isn't properly optimized for in ESX 3.5, while Shanghai doesn't have that problem. That's a potential 22% boost for AMD in that test, which we outright admit! Of course, as Johan then points out, there are a LOT of companies that aren't moving to ESX 4.0 for a while yet, so ESX 3.5 scores are more of a look at the current industry.
Again, we know that VMmark does provide one measurement of virtualization performance. Is it a "catch-all"? No more so than the vApus Mark I tests. They both show different aspects of how a server/CPU can perform in a virtualized environment. We haven't even looked at stuff like Linux yet, and you can rest assured that the performance of various CPUs with that environment are all over the place (due to optimizations or lack of optimizations). Anyhow, I expect Nehalem will stretch its legs more in 2-tile and 3-tile testing, even with our supposedly biased test suite.
Since you're so wise, let me ask you something: what would happen if a large benchmark became highly used as a standard measurement of performance in an industry where companies spend billions of dollars? Do you think, just maybe, that places like Intel, Dell, HP, Sun, etc. might do a bunch of extra optimizations targeted solely at improving those scores? No, that could never happen, especially not in the great USA where we alone are qualified to know how hardware works. Certainly NVIDIA and ATI never played any optimization games with 3DMark.
In short, the responses to your comments should give you a good idea of how reasoned your postings are. Cool your jets and learn to show respect and thoughtful posting. I don't know why you're so worried about people showing Intel in the best light possible, but you post on (practically) every Intel or AMD article pumping the joys of Intel, and lambasting AMD.
The fact is, many reviews of Nehalem show inflated benefits for the architecture relative to the real world. VMmark with ESX 4.0 definitely falls into that category - or do you think a range of 14.22@10 tiles with ESX 3.5 Update 4 to 24.24@17 tiles with ESX 4.0 is perfectly normal? I'm not sure anyone actually runs a real workload that mimics VMmark to the point where simply an update to ESX 4.0 would boost performance and virtualization potential by 70%.
Does Intel make the currently better CPU? Of course they do. Does that mean AMD isn't worth a look? Hardly. There are numerous reasons an architecture might perform better/worse. VMmark - or any benchmark - will at best show one facet of performance, and thus what we really need are numerous tests showing how systems truly perform.