Investigating Cavium's ThunderX: The First ARM Server SoC With Ambition
by Johan De Gelas on June 15, 2016 8:00 AM EST- Posted in
- SoCs
- IT Computing
- Enterprise
- Enterprise CPUs
- Microserver
- Cavium
Selecting the Competition
In setting up our benchmarks, we chose four different Intel SKUs to compete with the Cavium ThunderX. Our choices are not ideal (as we only have a limited number of SKUs available) but there is still some logic behind the SKU choice.
The Xeon E5-2640 v4 (10 cores @2.4 GHz, $939) has Intel's latest server core (Broadwell EP) and features a price tag in the ballpark of the ThunderX ($800) along with a low 90W TDP.
The Xeon E5-2690 v3 (12 cores @2.6 GHz, $2090) is a less optimal choice, but we wanted an SKU with a higher TDP, in case that the actual power consumption of the Thunder-X is higher than what can be expected from the official 120W TDP. To be frank, it was the only SKU that was faster than the E5-2640 v4 that we had. The Xeon E5-2699v4 ($4115, 145W TDP) did not make much sense to us in this comparison... so we settled for the Xeon E5-2690v3.
And then we added all the Xeon Ds we had available. At first sight it's not fair to compare a 45W TDP SoC to our 120W ThunderX. But the Xeon D-1557 is in the same price range as the Cavium ThunderX, and is targeted more or less at the same market. And although they offer fewer network and SATA interfaces, Cavium has to beat these kind of Xeon Ds performance wise, otherwise Intel's performance per watt advantage will steal Cavium's thunder.
The Xeon D-1581 is the most expensive Xeon D, but it is Intel's current server SoC flagship. But if the ARM Server SoCs start beating competitively priced Xeon Ds, Intel can always throw this one in the fray with a lower price. It is the SoC the ARM server vendors have to watch.
Configuration
Most of our testing was conducted on Ubuntu Server 14.04 LTS. We did upgrade this distribution to the latest release (14.04.4), which gives us more extensive hardware support. However, to ensure support for the ThunderX, the gcc compiler was upgraded to 5.2. In case of the ThunderX, the kernel was also 4.2.0, while the Intel systems still used kernel 3.19.
The reason why we did not upgrade the kernel is simply that we know from experience that this can generate all kinds of problems. In the case of the ThunderX using a newer kernel was necessary, while for the Intel CPUs we simply checked that there were no big differences with the new Ubuntu 16.04. The only difference that we could see there is that some of our software now does not compile on 16.04 (Sysbench, Perlbench). As we already waste a lot of time with debugging all kinds of dependency trouble, we kept it simple.
Gigabyte R120-T30 (1U)
The full specs of the server can be found here.
| CPU | One ThunderX CN8890 |
| RAM | 128GB (4x32GB) DDR4-2133 |
| Internal Disks | 2x SanDisk CloudSpeed Ultra 800GB |
| Motherboard | Gigabyte MT30-GS0 |
| BIOS version | 1/28/2016 |
| PSU | Delta Electronics 400w 80 Plus Gold |
Supermicro X10SDV-7TP8F and X10SDV-12C-TLN4F (2U case)
| CPU | Xeon D-1557 (1.5 GHz, 12 cores, 45 W TDP) Xeon D-1581 (1.8 GHz, 16 cores, 65 W TDP) |
| RAM | 64 GB (4x16 GB) DDR4-2133 |
| Internal Disks | 2x Intel SSD3500 400GB |
| Motherboard | Supermicro X10SDV-7TP8F Supermicro X10SDV-12C-TLN4F |
| BIOS version | 5/5/2016 |
| PSU | Delta Electronics 400w 80 Plus Gold |
Hyperthreading, Turbo Boost, C1 and C6 were enabled in the BIOS.
Intel's Xeon E5 Server – S2600WT (2U Chassis)
This is the same server that we used in our latest Xeon v4 review.
| CPU | Xeon E5-2640 v4 (2.4 GHz, 10 cores, 90 W TDP) Xeon E5-2690 v3 (2.6 GHz, 12 cores, 135 W TDP) |
| RAM | 128GB (8x16GB) Kingston DDR-2400 |
| Internal Disks | 2x Intel SSD3500 400GB |
| Motherboard | Intel Server Board Wildcat Pass |
| BIOS version | 1/28/2016 |
| PSU | Delta Electronics 750W DPS-750XB A (80+ Platinum) |
Hyperthreading, Turboost, C1 and C6 were enabled in the BIOS.
Other Notes
All servers are fed by a standard European 230V (16 Amps max.) power line. The room temperature is monitored and kept at 23°C by our Airwell CRACs in our Sizing Servers Lab.
Facebook
Twitter
RSS
82 Comments
View All Comments
Spunjji - Wednesday, June 15, 2016 - link
Well, this is certainly promising. Absent AMD, Intel need some healthy competition in this market - even if it is in something of a niche area.niva - Wednesday, June 15, 2016 - link
This is the area where profits are made, not "something of a niche area."Shadow7037932 - Wednesday, June 15, 2016 - link
Yeah, I mean getting some big customers like Facebook or Google would be rather profitable I'd imagine.JohanAnandtech - Thursday, June 16, 2016 - link
More than 30% of Intel's revenue, and the most profitable area for years, and for years to come...prisonerX - Wednesday, June 15, 2016 - link
This is the future. Single thread performance has reached a dead end and parallelism is the only way forward. Intel's legacy architecture is a millstone around its neck. ARM's open model and efficient implementation will deliver more cores and more performance as software adapts.The monopolists monopolise themselves into irrelevance yet again.
CajunArson - Wednesday, June 15, 2016 - link
" Intel's legacy architecture is a millstone around its neck."I wouldn't call those Xeon-D parts putting up excellent performance at lower prices and vastly lower power consumption levels to be any kind of "millstone".
"ARM's open model and efficient implementation "
What's "open" about these Cavium chips exactly? They can only run a few specialized Linux flavors that don't even have the full range of standard PC software available to them.
What is efficient about a brand-new ARM chip from 2016 losing at performance per watt to the 4.5 year old Sandy Bridge parts that you were insulting?
As for monopolies, ARM has monopolized the mobile market and brought us "open" ecosystems like the iPhone walled-garden and Android devices that literally never receive security updates. I'd take a plain x86 PC that I can slap Linux on any day of the week over the true monopoly that ARM has over locked-down smartphones.
shelbystripes - Wednesday, June 15, 2016 - link
You're right to criticize the "millstone" comment, Intel has done quite well achieving both high performance and high performance-per-watt in their server designs.But your comment about a "true monopoly" in the "locked-down smartphone" market is ridiculous. The openness (or lack thereof) that you're complaining about has nothing to do with the CPU architecture at all. An x86 smartphone or tablet can just as easily be locked down, and they are. I own a Dell Venue 8 7000, which is an Android tablet with an Intel Atom SoC inside. It's a great tablet with great hardware. But it's got a bunch of uninstallable crapware installed, Dell abandoned it after 5.1 (it's ridiculous that a tablet with a quad-core 2GHz SoC and 2GB RAM will never see Marshmallow), and the locked smartphone-esque bootloader means I can't repurpose it to a Linux distro even if one existed that supported all the hardware inside this thing.
On the flipside, the most popular open-source learning/development solution out there right now is the ARM-based Raspberry Pi. There are a number of Linux distros available for it, and everything is OSS, even the GPU driver.
TheLightbringer - Thursday, June 16, 2016 - link
You haven't done your homework.Some mobile devices were coming with Intel. But like Microsoft it entered the market too late, without offering any real value. The phrase "Too little, too late" fit them both.
ARM didn't do a monopoly. They just simply saw an opportunity and embrace it. In the early IBM clone days Intel licensed their architecture to allow competition and broad arrange of products. After the market was won, they went greedy, didn't licensed the architecture anymore and cut a lot of players out, leaving a need for a chip licensing scheme. And that's where ARM got in.
Google develops Android OS, but is up to phone vendors and carriers to deploy them. And they don't want to for economic reasons. They prefer to sell you a new phone for $$$.
Intel and MS got in the mobile/car market exactly what they deserve, nothing else.
junky77 - Friday, June 17, 2016 - link
they all greedy. Some just play it smartly or have more luck in decision makingBut, yea, when you read about the way IBM behaved when things were fresh - it's quite amazing. They had much of the market and could do a lot of stuff, but they simply had a very narrow mind set
soaringrocks - Wednesday, June 15, 2016 - link
You make it sound like it's mostly a SW problem, I think it's more complex than that. Actual performance is very dependent on the types of workload and some tasks fit Intel CPUs nicely and the performance per watt for ARM is lacking despite the hype of that architecture being uniquely qualified for low-power. It will be fun to watch how the battle evolves though.