Sixteen Cores, Four Sockets

Java Servers SPECjbb2005

SPECjbb2005 from SPEC (Standard Performance Evaluation Corporation) evaluates the performance of server-side Java by emulating a three-tier client/server system with emphasis on the middle tier. Instead of testing with a possible disk intensive database system, SPECjbb uses tables of objects, implemented by Java Collections, rather than a separate database. A longer description can be found here.

Again, it is not our objective to show the best possible scores. Very few people will take the time to fully tune the JVM and take the risk that some of the ultra aggressive optimizations backfire. So we tested with some decent but rather generic tuning that we could use on all systems. We used almost the exact same setup as we described here in great detail. The only changes were:

• We use the BEA JRockit p27.5.0-5 Linux x86_64, which has optimizations for both the "Barcelona" and the "Tigerton" CPUs.
  
• We have enabled large pages of 2MB each. This delivers a big performance jump.
  
• We do not use the SUN JVM as Sun is about to release a new JVM version which is significantly faster than the current one. This new JVM (1.6.0_06-prelease) allows Sun to claim the crown in SPECjbb2005 as you can see here.
  
• We changed one of our java parameters: -Xms1800m -Xmx1800m -Xns1300m -XXaggressive -XXlazyunlocking -Xgc:genpar -XXtlasize:min=4k,preferred=512k -XXcallprofiling
  
• We use Xns = 1.3GB instead of 1.5GB (used in previous reviews) as this leaves more room for the old space (Xms minus Xns) which avoids excessive garbage collecting.

Below you can find the final score which SPECjbb2005 reports, which is an average of the last four runs.

The scores on spec.org do not agree with our scores. Dell, for example, reports that the Quad Xeon 7350 should be able to reach up to 446209 BOPS. HP reports that the Quad Opteron can reach 368534 BOPS. Who is right? We're probably both right, as we use very different settings. It shows how confusing a benchmark can become when software optimization is pushed to insane heights just to get the best score.

We feel that it is likely that our results are closer to the real world. We keep our optimizations consistent over the different architectures, and it is a bit weird to see that the current scores of the Xeon 7350 are so high when it only has 9GB/s of bandwidth available. Considering that SPECjbb2005 uses between 8 and 32GB of RAM very actively, it is no surprise that memory bandwidth makes a big difference in performace. In fact, if you do not use Large Pages of 2MB, the benchmark is severely bottlenecked by the TLBs - another indication of how this benchmark uses memory.

We welcome feedback and our testing methods and results can be reproduced.