Bulldozer for Servers: Testing AMD's "Interlagos" Opteron 6200 Series
by Johan De Gelas on November 15, 2011 5:09 PM ESTIntroducing AMD's Opteron 6200 Series
When virtualization started to get popular (ca. 2005-2007), there was a fear that this might slow the server market down. Now several years later, the server market has rarely disappointed and continues to grow. For example, IDC reported a 12% increase in revenue when comparing Q1 2010 and Q1 2011. The server market in total accounted for $12 billion revenue and almost two million shipments in Q1 2011, and while the best desktop CPUs generally sell for $300, server chips typically start at $500 and can reach prices of over $3000. With the high-end desktop market shrinking to become a niche for hardcore enthusiasts--helped by the fact that moderate systems from several years back continue to run most tasks well--the enterprise market is very attractive.
Unfortunately for AMD, their share of the lucrative server market has fallen to a very low percentage (4.9%) according IDC's report early this year (some report 6-7%). It is time for something new and better from AMD, and it seems that the Bulldozer architecture is AMD's most server-centric CPU architecture ever. We quote Chuck Moore, Chief Architect AMD:
By having the shared architecture, reducing the size and sharing things that aren’t commonly used in their peak capacity in server workloads, “Bulldozer” is actually very well aligned with server workloads now and on into the future. In fact, a great deal of the trade-offs in Bulldozer were made on behalf of servers, and not just one type of workload, but a diversity of workloads.
This alginment with server workloads can also be found in the specs:
|
Opteron 6200 "Interlagos" |
Opteron 6100 "Magny-cours" |
Xeon 5600 "Westmere" |
|
| Cores (Modules)/Threads | 8/16 | 12/12 | 6/12 |
| L1 Instructions | 8x 64 KB 2-way | 12x 64 KB 2-way | 6x 32 KB 4-way |
| L1 Data | 16x 16 KB 4-way | 12x 64 KB 2-way | 6x 32 KB 4-way |
| L2 Cache | 4x 2MB | 12x 0.5MB | 6x 256 KB |
| L3 Cache | 2x 8MB | 2x 6MB | 12MB |
| Memory Bandwidth | 51.2GB/s | 42.6GB/s | 32GB/s |
| IMC Clock Speed | 2GHz | 1.8GHz | 2GHz |
| Interconnect | 4x HT 3.1 (6.4 GT/s) | 4x HT 3.1 (6.4 GT/s) | 2x QPI (4.8-6.4 GT/s) |
The new Opteron has loads of cache, faster access to memory and more threads than ever. Of course, a good product is more than a well designed microarchitecture with impressive specs on paper. The actual SKUs have to be attractively priced, reach decent clock speeds, and above all offer a good performance/watt ratio. Let us take a look at AMD's newest Opterons and how they are positioned versus Intel's competing Xeons.
| AMD vs. Intel 2-socket SKU Comparison | |||||||||
| Xeon |
Cores/ Threads |
TDP |
Clock (GHz) |
Price | Opteron |
Modules/ Threads |
TDP |
Clock (GHz) |
Price |
| High Performance | High Performance | ||||||||
| X5690 | 6/12 | 130W | 3.46/3.6/3.73 | $1663 | |||||
| X5675 | 6/12 | 95W | 3.06/3.33/3.46 | $1440 | |||||
| X5660 | 6/12 | 95W | 2.8/3.06/3.2 | $1219 | |||||
| X5650 | 6/12 | 95W | 2.66/2.93/3.06 | $996 | 6282 SE | 8/16 | 140W | 2.6/3.0/3.3 | $1019 |
| Midrange | Midrange | ||||||||
| E5649 | 6/12 | 80W | 2.53/2.66/2.8 | $774 | 6276 | 8/16 | 115W | 2.3/2.6/3.2 | $788 |
| E5640 | 4/8 | 80W | 2.66/2.8/2.93 | $774 | |||||
| 6274 | 8/16 | 115W | 2.2/2.5/3.1 | $639 | |||||
| E5645 | 6/12 | 80W | 2.4/2.53/2.66 | $551 | 6272 | 8/16 | 115W | 2.0/2.4/3.0 | $523 |
| 6238 | 6/12 | 115W | 2.6/2.9/3.2 | $455 | |||||
| E5620 | 4/8 | 80W | 2.4/2.53/2.66 | $387 | 6234 | 6/12 | 115W | 2.4/2.7/3.0 | $377 |
| High clock / budget | High clock / budget | ||||||||
| X5647 | 4/8 | 130W | 2.93/3.06/3.2 | $774 | |||||
| E5630 | 4/8 | 80W | 2.53/2.66/2.8 | $551 | 6220 | 4/8 | 115W | 3.0/3.3/3.6 | $455 |
| E5607 | 4/4 | 80W | 2.26 | $276 | 6212 | 4/8 | 115W | 2.6/2.9/3.2 | $266 |
| Power Optimized | Power Optimized | ||||||||
| L5640 | 6/12 | 60W | 2.26/2.4/2.66 | $996 | |||||
| L5630 | 4/8 | 40W | 2.13/2.26/2.4 | $551 | 6262HE | 8/16 | 85W | 1.6/2.1/2.9 | $523 |
The specifications (16 threads, 32MB of cache) and AMD's promises that Interlagos would outperform Magny-cours by a large margin created the impression that the Interlagos Opteron would give the current top Xeons a hard time. However, the newest Opteron cannot reach higher clock speeds than the current Opteron (6276 at 2.3GHz), and AMD positions the Opteron 6276 2.3GHz as an alternative to the Xeon E5649 at 2.53GHz. As the latter has a lower TDP, it is clear that the newest Opteron has to outperform this Xeon by a decent margin. In fact most server buyers expect a price/performance bonus from AMD, so the Opteron 6276 needs to perform roughly at the level of the X5650 to gain the interest of IT customers.
Judging from the current positioning, the high-end is a lost cause for now. First, AMD needs a 140W TDP chip to compete with the slower parts of Intel's high-end armada. Second, Sandy Bridge EP is coming out in the next quarter--we've already seen the desktop Sandy Bridge-E launch, and adding two more cores (four more threads) for the server version will only increase the performance potential. The Sandy Bridge cores have proven to be faster than Westmere cores, and the new Xeon E5 will have eight of them. Clock speeds will be a bit lower (2.0-2.5GHz), but we can safely assume that the new Xeon E5 will outperform its older brother by a noticeable margin and make it even harder for the new Opteron to compete in the higher end of the 2P market.
At the low-end, we see some interesting offerings from AMD. Our impression is that the 6212 at 2.6-2.9GHz is very likely to offer a better performance per dollar ratio than the low-end Xeons E560x that lack Hyper-Threading and turbo support.
Okay, we've done enough analyzing of paper specs; let's get to the hardware and the benchmarks. Before we do that, we'll elaborate a bit on what a server centric architecture should look like. What makes server applications tick?
Facebook
Twitter
RSS
106 Comments
View All Comments
duploxxx - Thursday, November 17, 2011 - link
Very interesting review as usual Johan, thx. It is good to see that there are still people who want to thoroughly make reviews.While the message is clear on the MS OS of both power and performance i think it isn't on the VMware. First of all it is quite confusing to what settings exactly have been used in BIOS and to me it doesn't reflect the real final conclusion. If it ain't right then don't post it to my opinion and keep it for further review....
I have a beta version of interlagos now for about a month and the performance testing depending on bios settings have been very challenging.
When i see your results i have following thoughts.
performance: I don't think that the current vAPU2 was able to stress the 2x16core enough, what was the avarage cpu usage in ESXTOP during these runs? On top of that looking at the result score and both response times it is clear that the current BIOS settings aren't optimal in the balanced mode. As you already mentioned the system is behaving strange.
VMware themselves have posted a document for v5 regarding the power best practices which clearly mentions that these needs to be adapted. http://www.vmware.com/files/pdf/hpm-perf-vsphere5....
To be more precise, balanced has never been the right setting on VMware, the preferred mode has always been high performance and this is how we run for example a +400 vmware server farm. We rather use DPM to reduce power then to reduce clock speed since this will affected total performance and response times much more, mainly on the virtualization platform and OEM bios creations (lets say lack of in depth finetuning and options).
Would like to see new performance results and power when running in high performance mode and according the new vSphere settings....
JohanAnandtech - Thursday, November 17, 2011 - link
"l it is quite confusing to what settings exactly have been used in BIOS and to me it doesn't reflect the real final conclusion"http://www.anandtech.com/show/5058/amds-opteron-in...
You can see them here with your own eyes.
+ We configured the C-state mode to C6 as this is required to get the highest Turbo Core frequencies
"performance: I don't think that the current vAPU2 was able to stress the 2x16core enough, what was the avarage cpu usage in ESXTOP during these runs?"
93-99%.
"On top of that looking at the result score and both response times it is clear that the current BIOS settings aren't optimal in the balanced mode."
Balanced and high performance gave more or less the same performance. It seems that the ESX power manager is much better at managing p-states than the Windows one.
We are currently testing Balanced + c-states. Stay tuned.
duploxxx - Thursday, November 17, 2011 - link
thx for answers, i read the whole thread, just wasn't sure that you took the same settings for both windows and virtual.according to Vmware you shouldn't take balanced but rather OS controlled, i know my BIOS has that option, not sure for the supermicro one.
quite a strange result with the ESXTOP above 90% with same performance results, there just seems to be a further core scaling issue on the vAPU2 with the performance results or its just not using turbo..... we know that the module doesn't have the same performance but the 10-15% turbo is more then enough to level that difference which would still leave you with 8 more cores
When you put the power mode on high performance it should turbo all cores for the full length at 2.6ghz for the 6276, while you mention it results in same performance are you sure that the turbo was kicking in? ESXTOP CPU higher then 100%? it should provide more performance....
Calin - Friday, November 18, 2011 - link
You're encrypting AES-256, and Anand seem to encryrpt AES-128 in the article you liked to in the Other Tests: TrueCrypt and 7-zip pagetaltamir - Friday, November 18, 2011 - link
Conclusion: "Intel gives much better performance/watt and performance in general; BD gives better performance/dollar"Problem: Watts cost dollars, lots of them in the server space because you need to some some pretty extreme cooling. Also absolute performance per physical space matters a lot because that ALSO costs tons of money.
UberApfel - Sunday, November 20, 2011 - link
A watt-year is about $2.The difference in cost between a X5670 & 6276; $654
On Page 7...
X5670: 74.5 perf / 338 W
6276: 71.2 perf / 363 W
adjusted watt-per-performance for 6276: 363 * (74.5 / 71.2) = 380
difference in power consumption: 42W
If a server manages an average of 50% load over all time; the Xeon's supposed superior power-efficiency would pay for itself after only 31 years.
Of course you're not taking into consideration that this test is pretty much irrelevant to the server market. Additionally, as the author failed to clarify when asked, Anandtech likely didn't use newer compilers which show up to a 100% performance increase in some applications ~ looky; http://www.phoronix.com/scan.php?page=article&...
Thermalzeal - Monday, November 21, 2011 - link
Good job AMD, you had one thing to do, test your product and make sure it beat competitors at the same price, or gave comparable performance for a lower price.Seriously, wtf are you people doing?
UberApfel - Tuesday, November 22, 2011 - link
Idiots like this is exactly why I say the review is biased. How can anyone with the ability to type be able to scan over this review and come to such a conclusion. At least with the confidence to comment.zappb - Tuesday, November 29, 2011 - link
completely agree - some very strange comments along these lines over the last 11 pageszappb - Tuesday, November 29, 2011 - link
posted by ars technica - incredibly tainted in intels favourThe title is enough:
"AMD's Bulldozer server benchmarks are here, and they're a catastrophe"