The very first independent Nested Paging Virtualization tests
by Johan De Gelas on July 10, 2008 12:00 AM EST- Posted in
- IT Computing general
The first generation of hardware virtualization was nothing to write home about from a performance point of view. In many cases, hardware virtualization was disabled by the OEMS to avoid problems, in other cases hardware virtualization was simply slower than the current forms of software virtualization. It was a step back from the highly optimized VMWare binary translation, but of course a step back to take many steps forward.
We explained this in great detail here. One of the problems was that software virtualization tackled (virtual) memory management so much better than hardware virtualization. First generation Hardware virtualization - as can be found in the Xeon 53xx series and second generation Opterons - should be seen as way of making it easier to write a slim hypervisor. One of the examples is Linux KVM, which makes good use of hardware virtualization while it avoids the complexities of binary translation and paravirtualization. Basically many of the new hypervisor solutions would not be even possible without the availabiltiy of hardware virtualization capable CPUs.
The second generation of hardware accelerated virtualization which was first introduced in the third generation of Opterons (AMD "Barcelona" aka K10) is a lot better from a performance point of view. It offers massive tagged TLBs that can support the memory management of the different guest OS and the hypervisor. Intel's Nehalem will use this technique too.
The past months, we have been working very hard on a massive virtualization benchmarking effort. It will take a while to analyze the huge amount of data we generated so far, but we could not resist to give you already a taste.
The benchmarks below have been done on a dual socket Opteron 8356 (so 8 cores at 2.3 GHz) machine. Our quad socket board is not supported by ESX, so that is why we limited our Opteron 8356 to 2 CPUs for now. The specific benchmarks below were done on Xen 3.2.0 which is distributed with SLES 10 SP2. Four virtual machines are running at the same time, and each virtual machine has 2 virtual CPUs which are firmly linked to 2 physical Cores. Two webservers, one Oracle OLTP test and one DSS MySQL test are all running in parallel on our dual opteron server. Each virtual machine has 4 GB of RAM and runs Windows 2003 R2.We will give a lot of more benchmarking detail in the upcoming article.
We enabled and disabled HAP in the kernel parameters of Xen 3.2.0 (SUSE SLES 10 SP2). Oh yes, Hardware Assisted Paging is the same thing as Nested Paging, RVI (AMD marketing) and EPT (Intel marketing). Confused? Blame Intel and AMD marketing :-). The No-HAP or No NPT benchmarks are normalized to 100%.
There are two very interesting analyses to be made from the graph above. First of all, Xen 3.2.0 which can be found in the newest Novell SLES 10 SP2, is capable of running Windows 2003 R2 under heavy stress.
In theory this has been the case for quite some time as it was claimed from the start of Xen 3.0 that Xen would be able to run Windows based virtual machines thanks to hardware virtualization. In practice, all Xen versions 3.0.x have a lot of trouble with Windows as a guest OS, freezing the virtual machines up etc. Simply put, running Windows on a Xen hypervisor before Xen 3.2 is simply asking for trouble.
But now Xen is capable of running Windows 2003 machines. Our in house developed software really stresses the virtual machines well over their maximum, so we have good reason to believe that if a software survives the stresstest hell we put it through, it is stable. We believe Xen is now a viable (and probably cheap) alternative to VMWare ESX and Hyper-V.
Secondly, Nested Page Tables, also know as Rapid Virtualization Indexing (Yep, that is the marketing name :-) can give a serious boost to some memory intensive webservers. 31% better performance is nothing to sneeze at. Let me remind you that these webservers are not some industry standard benchmarks for which AMD can sent over a team of engineers to optimize it.
Now, we just used a realworld webserver for this benchmarking. If you like to get an idea of what kind of webserver software we tested, look here.
The OLTP test ("Oracle Swingbench") is not limited by the harddisk performance. We test with six 15000 rpm seagate SAS 300 GB disks (more or less the fastest harddisk you can get) in RAID-0 and we have performed the same test with twelve disks. The extra disks gave a 1% boost, so our disk system is not the bottleneck.
The Oracle Swingbench OLTP tests shows a smaller performance boost thanks to NPT, 7%. Not bad at all.
The two benchmarks shows that the second generation of hardware virtualization does make a difference. So hardware virtualization is one of those new technologies that really has a promising future. Seeing how NPT really pushes hardware virtualization forward feels a bit similar to the way the Voodoo 2 opened a new world of 3D graphics, or how the first dualcore CPUs started the multicore race. Pretty exciting if you ask us, and we feel the same way about our upcoming articles.
Oh and in case you haven't noticed, it.anandtech.com really offers you some very special content on virtualization. If you are sick an tired of reading the same woolly - copied from the software vendors - texts about virtualization, you should start reading right here. We have
explained for you. Read those 3 articles and you are up to speed on what is going in the virtualization market. Next some very unique benchmarks on Hyper-V, VMWare ESX and Novell/Xen. AMD versus Intel will be - of course - in there too...
Facebook
Twitter
RSS
20 Comments
View All Comments
andy987s - Tuesday, August 19, 2008 - link
Red Hat magazine have an interesting benchmark showing the combination of NPT hardware support plus Para-Virtualised drivers makes a huge difference. See http://www.redhatmagazine.com/2008/07/03/red-hat-e...">http://www.redhatmagazine.com/2008/07/0...na-with-...In short it's demonstrating that NPT or FV alone provide useful boosts but still land you with an inefficient VM. Whereas the combination provides a huge boost and makes the VM come close to native performance!
I think it's important that all the benchmarks should show native machine performance where possible. eg run workload 1 on server X. Run workload 2 on server X. Now run workload 1 and/or 2 on server X in a VM.
7oby - Friday, August 15, 2008 - link
You wrote:The OLTP test ("Oracle Swingbench") is not limited by the harddisk performance. We test with six 15000 rpm seagate SAS 300 GB disks (more or less the fastest harddisk you can get) in RAID-0 and we have performed the same test with twelve disks. The extra disks gave a 1% boost, so our disk system is not the bottleneck.
You might want to consider 6x Non-RAID or 3x RAID-0 for database benchmarking. Databases are smart and the parallelism is achieved inside the database and I/O has to be parallel as well. One big 6x RAID0 is slow.
If you look at what VMMark benchmarks are typically performed:
http://www.vmware.com/products/vmmark/results.html">http://www.vmware.com/products/vmmark/results.html
They use a RAID10 setup with 7 independet drives.
Here's another article about the superiour performance of a RAID1 setup compared to a RAID10 in db benchmarking due to the additional parallesim:
http://blogs.zdnet.com/Ou/?p=484&page=2">http://blogs.zdnet.com/Ou/?p=484&page=2
If you maxed out the parallism (by as many independent drives as possible), you may use RAID0 or RAID10 for at most two drives just to be sure to have always a bandwidth that's >1Gb/s networking performance over the entire platters.
JohanAnandtech - Monday, August 18, 2008 - link
Excellent Suggestion. Although in this setup we are sure that we are definitely not disk (DQL checked and no disk wait cycles) limited, we'll redo the more disk intensive tests.gtrider - Wednesday, July 23, 2008 - link
This is interesting from a kid system point of view but I would be seriously interested in comparing this virtualization to system z with PR/SM && LPAR technologies or z/VM and z/Linux.Its a shame that the techie hardware sites don't talk about the grown up systems more.
mino - Thursday, July 24, 2008 - link
The reasoning is simple.System/Z plays a different league, period.
You have allready mentioned it Yourself, the whole x86 virtualization is just kids play compared to mainframes.
But hell, the mainframes are also a different league from cost POW - while not so expensive anymore, for casual mid-sized company that may consider this site it is usually just an overkill.
Also, a mainframe (even the smallest one) is not somethig You go out and buy to test ...
mabellon - Tuesday, July 22, 2008 - link
In the article it is stated:"It offers massive tagged TLBs that can support the memory management of the different guest OS and the hypervisor. Intel's Nehalem will use this technique too."
Obviously the TLBs (standard and nested-extended) are of finite size. Couldnt excessive context switching of processes within an OS result in thrashing the TLB cache (exceeding the address space tag maximum for instance)?
Similarily, are there limitations to the NPT-EPT with regards to the maximum number of hosted VMs, or the context switch rate between processes and VMs? In the future it would be nice to see how this technology scales with VM count and activity. I sincerely doubt businesses intend to run a single Win2k3 virtual machine at a time.
Also, now that this is done in hardware, what kind of memory savings can be expected compared to previous software implementations?
Thanks, Interesting read.
JvdW - Wednesday, July 16, 2008 - link
Just to show what hardware can do for stability, my recent horror story.I have 3 Sun X2200 M2 two CPU dual Core fitted with 16Gb and one with 12Gb and setting up a Xen based environment (Virtual Iron 4.3.8)
Things seemed to work smoothly until I noticed that once in a while the node itself, not the VM, seemed to crash. This started an investigation which led to the following conclusion:
All 3 nodes were upgraded to the then latest BIOS/BMC (tools CD version 1.6), mainly because of security fixes and functionality in the eLOM. I can consistently crash a node by stress testing a XP VM using a HDBenchmark together with a CPU benchmark (HD Speed and 7-Zip). Once I degrade a node to a BIOS/BMC version of the 1.3 Tools CD (2.70 3B13 BMC/BIOS) that VM runs perfectly well while under stress.
Another pecularity of the higher BIOS/BMC version than 1.3 is that the crashes occur more or less at the same time, for example when it first crahes around 10:30 then it will crash again around xx:30 whether that is 1 hour later or 10 hours, on average it runs 2-3 hours.
So be careful to state stability statements without reasoning behind it.
Those servers are very stable with VI as long as I stay on the 1.3 Tools release but sadly that looks like an impossiblity in the future.
Any of you techies having an explanation for this, how weird it may seem I'm in for any explaination.
Greetings,
Joop
mino - Tuesday, July 15, 2008 - link
1) I REALLY appreciate your effort to making a reasonable performance benchmark. AFAIK no such 3rd party benchmark with reasonable value is available for free.2) Please be careful talking about Xen "hypervisor (in)stability".
XEN is as a name You should sorely reserve for Citrix Xen Server or the deep-technical talk about hypervisors.
The stability AND performance of an virtualized solution is mostly affected by the I/O. I/O is NOT part of the hypervisor as occurs in so-called dom0/service-partition (or whatever PR guys rename it) that is essentially na I/O router for all the VM's on the machine.
Also, as You are well aware, there are many virtualization solution utilizing the Xen hypervisor.
In brief, in order of maturity:
Virtual Iron
Citrix's Xen Server (former XenSource XenServer)
Novel's; Red Hat's; and other "Xen-based virtualization" solution from Linux distribution vendors.
Please, do not reproduce the same mistake I see all over the web - using the hypervisor name for describing a virtualization solution characteristics.
(I know Citrix does it intentionally in their PR, but we are technologists not Avareage Joe's to be misled)
3) What I would like to see is have benchmarked the real solution that compete for the market i.e. ESX vs. Xen Server vs. Virtual Iron vs. Hyper-V.
However You take it, the solutions from the Linux OS Vendors have a very limited market penetration. Microsoft too has not market, but everybody knows it will get some so it makes sense to include Hyper-V.
Comparison of Red Hat's Xen solution against their KVM solution may be interesting but the real value for IT people out there is to compare the existing solutions one is going to buy for his possible project.
Perkida - Tuesday, July 15, 2008 - link
I appreciate someone looking into such benchmarks, it's about time.When comparing enterprise level solutions such as VMware ESX and Hyper-V it is probably more directly comparable to the enterprise level Xen put out by Citrix, XenServer, than the free version. The embedded product that ships on select HP systems may be of particular interest as it has the ability to display up to 9 VM's in a grid locally using the servers display driver instead of being required to remote in (also still an option). There is something special about being able to view 4, 6, or 9 VM's at once when you're sand-boxing a QA build, etc.
Just a thought as my experience with CTX has always been a positive one.
Take it away Brian Madden.
mino - Tuesday, July 15, 2008 - link
Second that, did not see You post comming ;)