CPU Tests: SPEC

SPEC2017 and SPEC2006 is a series of standardized tests used to probe the overall performance between different systems, different architectures, different microarchitectures, and setups. The code has to be compiled, and then the results can be submitted to an online database for comparison. It covers a range of integer and floating point workloads, and can be very optimized for each CPU, so it is important to check how the benchmarks are being compiled and run.

We run the tests in a harness built through Windows Subsystem for Linux, developed by our own Andrei Frumusanu. WSL has some odd quirks, with one test not running due to a WSL fixed stack size, but for like-for-like testing is good enough. SPEC2006 is deprecated in favor of 2017, but remains an interesting comparison point in our data. Because our scores aren’t official submissions, as per SPEC guidelines we have to declare them as internal estimates from our part.

For compilers, we use LLVM both for C/C++ and Fortan tests, and for Fortran we’re using the Flang compiler. The rationale of using LLVM over GCC is better cross-platform comparisons to platforms that have only have LLVM support and future articles where we’ll investigate this aspect more. We’re not considering closed-sourced compilers such as MSVC or ICC.

clang version 10.0.0
clang version 7.0.1 (ssh://git@github.com/flang-compiler/flang-driver.git
 24bd54da5c41af04838bbe7b68f830840d47fc03)

-Ofast -fomit-frame-pointer
-march=x86-64
-mtune=core-avx2
-mfma -mavx -mavx2

Our compiler flags are straightforward, with basic –Ofast and relevant ISA switches to allow for AVX2 instructions. We decided to build our SPEC binaries on AVX2, which puts a limit on Haswell as how old we can go before the testing will fall over. This also means we don’t have AVX512 binaries, primarily because in order to get the best performance, the AVX-512 intrinsic should be packed by a proper expert, as with our AVX-512 benchmark. All of the major vendors, AMD, Intel, and Arm, all support the way in which we are testing SPEC.

To note, the requirements for the SPEC licence state that any benchmark results from SPEC have to be labelled ‘estimated’ until they are verified on the SPEC website as a meaningful representation of the expected performance. This is most often done by the big companies and OEMs to showcase performance to customers, however is quite over the top for what we do as reviewers.

For each of the SPEC targets we are doing, SPEC2006 rate-1, SPEC2017 speed-1, and SPEC2017 speed-N, rather than publish all the separate test data in our reviews, we are going to condense it down into a few interesting data points. The full per-test values are in our benchmark database.

(9-0a) SPEC2006 1T Geomean Total(9-0b) SPEC2017 1T Geomean Total(9-0c) SPEC2017 nT Geomean Total

 

CPU Tests: Synthetic Gaming Tests: Chernobylite
Comments Locked

126 Comments

View All Comments

  • YB1064 - Monday, January 4, 2021 - link

    Thermals look horrendous! 102 C even when throttled is bad. Even with water, I don't think you will see much improvement. Better have a chiller on hand!
  • GeoffreyA - Monday, January 4, 2021 - link

    The sun should buy a few million of these fellows. Will help a lot with temperature when sunspots get too cool.
  • at_clucks - Monday, January 4, 2021 - link

    Nooo, as the article puts it, it's "not as bad as it sounds". People just needlessly freak out when they processor FREAKING BOILS WATER on "an open test bed with a chunky copper cooler" and needing 270W at full load. But in reality it's just "more about thermal gradients inside the processor and how easily the thermals can move" so it's all good. I can feel the temps dropping just by reading these comforting words.

    I guess the only thing that can make a current gen Intel CPU look good is another current gen Intel CPU. Here's hoping that they get their act together for the next gen(s) or we'll keep having to read reviews where Intel competes against Intel as the only way to get some praise.
  • at_clucks - Monday, January 4, 2021 - link

    Read the comment above as if there was an edit button to fix all the grammatical... inconsistencies.
  • 1_rick - Monday, January 4, 2021 - link

    It's no worse than the article itself.
  • Smell This - Monday, January 4, 2021 - link

    as the article puts it, it's "not as bad as it sounds"
    _______________________________________

    Sadly, the presentation makes it much worst. Bad enough smoked with the Ryzen 5900x ... there is little to no downward price pressure on an 18-month old AMD 3900x at $499.

    Even worst --- 18 months ago the HEDT Intel 12/24 i9-79xxX was $1,190 and was slobber-knocked by the 3900x . . .
  • GeoffreyA - Wednesday, January 6, 2021 - link

    "Here's hoping that they get their act together"

    Rocket Lake should cover the ground performance-wise, but it's going to be disastrous when it comes to power and heat. Would be nice to see Sunny Cove running on 7 or 5 nm, but I suspect even there, Zen 3 would use less power, owing to its more economical design.
  • powerarmour - Monday, January 4, 2021 - link

    It's a genuinely horrible CPU compared to the competition, Intel have regressed so much on performance-per-watt that even Apple are stealing their lunch now.
  • shelbystripes - Monday, January 4, 2021 - link

    It’s embarrassing (for Intel) how easily Apple justified the Intel-to-ARM switch with the M1. Stagnation is a bitch.
  • Operandi - Monday, January 4, 2021 - link

    Part stagnation part major technological fumbles. Intel definitely played it safe on the architectural level but you could argue that was the right call given the state of the market. What is really killing them is the dropped ball on the fabrication front.

    Regardless though Apple's move to their own designs was going to happen even if Intel was stumbling all over the place, that move has been planned for over a decade.

Log in

Don't have an account? Sign up now