Introduction

Computer game developers always face a dilemma when creating cutting edge games: do you try to target the lowest common denominator in terms of graphics and performance, or do you try to create a truly next-generation game and hope that the hardware is able to support it at launch? When The Elder Scrolls IV: Oblivion was released, we saw the most recent instance of a game that reached beyond the limits of the graphics cards available at launch, and even now 8 months after its release it still is one of the most stressful games available. Part of the problem with pushing too far into next-generation hardware requirements is that developers run the risk of people being disenfranchised with their product, due to poor performance. After all, it doesn't matter how great a game looks if it's basically an unplayable slideshow.

Bethesda was fortunate that they created a game that received enough critical acclaim that many people were willing to live with the low frame rates, or even spend the extra money to get hardware capable of running the game with all of the details turned up. Certainly, the fact that Oblivion is a sequel to a popular franchise helped overcome the performance limitations. Ubisoft is in a somewhat similar situation with their successful line of games in the Splinter Cell series. Recently they released their newest installment of the series, Splinter Cell: Double Agent (SCDA) for the PC, which is essentially a direct port from the Xbox 360 version released several days before. We even see configuration files designed for the 360 included with the PC version.

Splinter Cell: Double agent doesn't break the same type of graphical boundaries that The Elder Scrolls IV: Oblivion did, and unfortunately there seem to be a lot of technical problems with the game. Aspects of the game feel rushed, and even though there is a patch available which addresses some issues (such as anti-aliasing) the game is still rife with bugs and glitches. Luckily though, when working the game itself is very enjoyable and a great addition to the series; we just wish Ubisoft could have waited till they fixed many of the technical issues in the game before they released it (or better yet, address the issues and release the game on schedule in the first place).

One of the major issues we have had during our testing is the exclusion of a usable timedemo feature. Timedemo functionality has been included in previous generations of Splinter Cell, and even the readme file distributed with SCDA makes reference to using its timedemo feature to test the performance of in-game settings. Without the ability to make use of a timedemo, we had to fall back on FRAPS for our performance testing. This does introduce a higher degree of variability to our tests, but our numbers will reflect the performance gamers will see when playing on a system similar to our test configuration. Hopefully Ubisoft will release a patch that exposes the timedemo (and console) features that are inherent in games built around Unreal Engine 2. If they do, we will be there with an update to our testing.

As with our initial Oblivion performance articles, we tackled Splinter Cell: Double Agent's performance tests in multiple parts. We found that the game has a wide range of environments, some of which have higher or lower impacts on performance, so we chose two benchmarks that represent a good balance of CPU and GPU limited scenarios the gamer encounters throughout the different maps. We will also be testing different graphical settings to find out what works the best for the different types of mainstream and high-end graphics cards out there right now.



Benchmarking Splinter Cell: Double Agent

As with previous Splinter Cell games, Double Agent has a variety of environments, but whereas there was a lot of sneaking around in dark areas previously, this newest game has a lot of more open daylight scenarios. This makes for interesting and more difficult gameplay, since for most missions the goal is still to remain undetected. Like in Chaos Theory, Double Agent begins with Sam Fisher, your character, working for the NSA and each mission is about completing objectives for the NSA and gaining trust. As you progress, Sam infiltrates a terrorist organization called the JBA (John Brown's Army), rising through the ranks and gaining trust from them as well.

The first benchmark takes place in the first mission, and is at night (with night vision enabled) where our familiar Sam Fisher is trying to infiltrate a missile silo surrounded by guards.


This benchmark is a short zip-line ride over a portion of the first map near the beginning just before one of the first checkpoints. Sam and his partner enter the enemy base via a cable without alerting anyone, completing an NSA objective and gaining trust. There are a few guards standing and walking around below, but not much else is going on in this scene.

The second benchmark is later in the game, which takes place in broad daylight on a cruise ship which is very open. Like the first benchmark, the test is a zip-line ride from the bridge of the ship down to the front of the boat, looking down to take in the scenery.


The elaborate structures like the crane on the boat cast shadows in the bright daylight, which help make this a more stressful test on the GPU. There is one guard standing beneath who doesn't happen to look up and spot Fisher as he flies overhead. The benchmark takes place near the very end of the mission, when Sam must complete the final objectives and escape the boat.



Settings

We tested Double Agent at two major settings; the first we refer to as high quality, and the other we will call medium quality (though the game doesn't have specific settings that correspond with this). With high quality settings, all the graphics settings are set to "on" and/or to their highest quality level (with the exception of AA and HDR which are mutually exclusive on cards that don't support floating point with AA, such as the GeForce 7 series).


We found that the settings which had some of the largest impacts on performance with Splinter Cell: Double Agent were the different shadow settings, specifically high quality soft shadows and high detail shader. By turning off these effects as well as trilinear filtering, we were able to significantly improve performance, making the game much more playable on lower-end hardware.


We chose these settings for our lower quality tests both because of the improved performance they offer while still allowing for some great visuals. Certain settings like high detail environments and high detail shader might offer better performance when switched to low or off, but we found the sacrifice in graphical quality too great to justify anything but "high" for these options. Luckily, the game isn't the most graphically intensive game out right now, and it happens to run well at highest quality on a wide variety of cards, particularly with ATI hardware.

Without the shadow effects enabled, the game does look a bit less striking, but the performance gains we see with these settings off is a good trade-off, especially for lower end hardware. Also, there is almost no difference visually with these settings on or off in our first benchmark because it takes place at night, and this will be the case for many areas in the game. Of course, if you have a card that can handle the extra stress, the complex shadow effects make levels like the cruise ship look much more stunning.



As we mentioned, because there is no timedemo function like with the previous Splinter Cell: Chaos Theory, we had to use FRAPS to record the frame rate for both of our benchmarks. Even though we used FRAPS rather than a timedemo to record our average frame rates during these scenes, our results were fairly consistent between multiple runs. To be as accurate as possible, we also ran each benchmark multiple times and took the mean result.



Setting Expectations

Luckily, with Double Agent, we don't have a game like Oblivion on our hands that stresses current generation GPUs to their limit and beyond. However, the game is still fairly taxing on your graphics card with all the quality settings turned up, particularly at high resolutions. We found that ATI hardware ran the game a little better and without as many problems as NVIDIA gear, and this isn't a big surprise, considering the game was originally made for the Xbox 360. In fact, the game is apparently a direct transfer of software from the Xbox360 platform to the PC, which might account for a lot of the bugs and the clunky game interface.

It's no secret that Splinter Cell: Double Agent has some problems running, especially on NVIDIA hardware. We found that on our test system, the game would crash to the desktop often when just changing simple settings like HDR and antialiasing. We discovered that by changing these settings while the game resolution was set to 640x480 we were able to avoid the constant crashing, but the game still had to be restarted for some settings to take effect. Constantly restarting the game gets very tiresome because of the long intro/splash screen which cannot be bypassed. This coupled with the long load times (and unload times when quitting a game back to the main menu) is a frustrating aspect of the game menu. We also found that sometimes certain quality settings wouldn't take effect even after restarting the game and/or computer, but then would suddenly take effect when something like the game resolution was changed.

The game was much more stable on ATI hardware, with hardly any of the crashing we saw with the NVIDIA cards. We still had to restart the game and sometimes switch resolutions to get certain quality settings to change, though. Also, CrossFire setups seem to cause a lot of crashing, and on the off chance that the game would work for us with CrossFire enabled it did not improve performance over a single GPU. In fact, both SLI and CrossFire don't seem to be supported by the game at present, and they actually resulted in worse performance than a single card. The game also won't play on our X800 card, and we suspect there will be limited or no support for SM2.0 cards in this game.

Splinter Cell: Double Agent is based on Unreal Engine 2, but graphically it looks better than original UE2 games because Ubisoft added DX9 shaders to handle effects like dynamic shadows, HDR, and a few other items. In this way Ubisoft gives the game a much more current and exciting look despite the older engine. However, even though the graphics look good, they aren't nearly as advanced graphically as what we will see from titles based around Unreal Engine 3.



Test Settings

We tested Double Agent on a variety of cards from both ATI and NVIDIA, all of which support SM3.0. The ATI cards we tested were the X1300 XT, X1650 Pro, X1650 XT, X1950 Pro, X1900 XT 256MB, and finally the X1950 XTX. From NVIDIA, we tested the 7300 GS, 7300 GT, 7600 GS, 7600 GT, 7950 GT and 7900 GTX. Without SLI support the 7950 GX2 was unable to perform as it should given it is effectively single-card SLI; the GX2 saw frame rates lower than our single 7950 GT, thus it was excluded from our tests. Also, NVIDIA's 8800 cards were excluded from tests due to rendering errors during gameplay. While the game did run on our 8800, the Sam Fisher model was warped and twisted into a very strange shape making gameplay pretty much impossible, as the image shows below. Hopefully we will see a patch for the game that will address this, along with the other SLI/CrossFire issues and various bugs soon.



Since Double Agent is still mainly a game of stealth, a very high frame rate is not necessary for a decent gameplay experience. We found that an average frame rate of no less than 15fps was smooth enough to provide good gameplay, but any less and the game starts to get choppy. Even though 15fps was still playable, though, 25fps and higher is definitely preferred. Here is the test system we used for these benchmarks.

Sytem Test Configuration
CPU:	Intel Core 2 Extreme X6800 (2.93GHz/4MB)
Motherboard:	EVGA nForce 680i SLI Intel BadAxe
Chipset:	NVIDIA nForce 680i SLI Intel 975X
Chipset Drivers:	Intel 7.2.2.1007 (Intel) NVIDIA nForce 9.35
Hard Disk:	Seagate 7200.7 160GB SATA
Memory:	Corsair XMS2 DDR2-800 4-4-4-12 (1GB x 2)
Video Card:	Various
Video Drivers:	ATI Catalyst 6.10 NVIDIA ForceWare 96.97 NVIDIA ForceWare 91.47 (G70 SLI)
Desktop Resolution:	2560 x 1600 - 32-bit @ 60Hz
OS:	Windows XP Professional SP2

High-End Cards

Certain things jumped out as soon as we started benchmarking Splinter Cell: Double Agent. In particular, aside from running with less bugs and glitches, the game performed much better on ATI hardware than NVIDIA hardware in all of our tests. Almost all of our ATI cards were able to run the game with fairly playable frame rates at the highest resolution, with the highest quality settings. As a testing side note, we also noticed that ATI saw frame rates that were more consistent than NVIDIA's between multiple runs of the same benchmark and settings. For all the NVIDIA cards, when running a benchmark twice, there was an improvement in performance of 1-2 fps, while with ATI the difference was only a fraction of an fps.

For the performance tests, we divided the test resolutions into three sections for NVIDIA and ATI based on the three levels of hardware: low-end, mid-range, and high end performance. For the lower-end cards, we tested at 640x480, 800x600, and 1024x768 resolutions. For mid-range cards, we tested at 800x600, 1024x768, and 1280x1024 resolutions. On high end cards, we tested at the three highest resolutions: 1024x768, 1280x1024, and 1600x1200. In this way we were able to see some performance overlaps between tests with each of the different types of cards.





Here we see right away the difference in performance between ATI and NVIDIA hardware with Double Agent. ATI's X1950 XTX gets the highest performance of the group, and could have handled much higher resolutions had they been available. The X1950 XTX even sees a little bit of CPU limitation between these resolutions, because of its high level of performance with this game. NVIDIA's 7900 GTX gets similar performance to the X1900 XT 256, and the 7950 GT similar to the X1950 Pro.

For the more stressful cruise ship benchmark, we can see how much performance impact a highly detailed area can have in this game. The cruise ship level is one of the more graphically intensive levels of the game, and this is one of the places where your hardware will really make a difference in gameplay. The 7900 GS manages to run this benchmark smoothly at 1280x1024, but if you want to play the game at 1600x1200 you will have more flexibility with ATI hardware. The performance difference, coupled with the smoother operation and fewer crashes, make ATI hardware a much better solution for running this game than NVIDIA.



Midrange Cards

The midrange or mainstream cards we chose to test were the NVIDIA 7600 GT and 7600 GS, as well as ATI's X1650 XT and X1650 Pro. Since some of these cards had trouble running the game at all the resolutions with high quality enabled (especially with the cruise ship benchmark), we tested these cards on both benchmarks with both the highest quality settings enabled, and again with the lower quality graphics settings (high quality soft shadows, high detail shader, and trilinear filtering set to off).





The difference in performance between the X1650 XT and 7600 GT further illustrates the fact that this game runs better on ATI hardware. These two cards are direct competitors in performance and price, and the X1650 XT sees about a 14% increase in performance over the 7600 GT. At around $140 The 7600 GT is a little cheaper than the X1650 XT right now, but we found a Sapphire X1650 XT for about $150 which would be a much better buy if Splinter Cell: Double Agent is your game of choice. The two cards trade blows in other titles, so the final decision will come down to game preference, but we would give the X1650 XT a slight edge overall.

For our medium quality performance tests, we left out the higher end cards from NVIDIA and ATI which obviously didn't need a performance boost at the sacrifice of image quality. Double Agent's graphics suffer with the shadow effects disabled, in some levels more than others, so high-quality settings should be enabled to get the most out of the game if your hardware can handle it. We also found the game looked and played well on some mainstream cards at lower resolutions like 1024x768 with all the effects enabled, so turning down the resolution a little might be a good solution for less powerful hardware. For those that must play the game at higher resolutions, turning off some of the shadow effects as well as trilinear filtering can significantly improve performance.





The performance differences we see for the cruise ship benchmark when the shadow/shader effects are disabled are much more dramatic than with the first benchmark. This is because in the particular test we chose, there are lots of shadows created by the bright sun shining down on the whole front deck of the boat. This is very different from the first benchmark which takes place under the cover of night with night vision enabled. Without the high quality soft shadows and high detail shader enabled, there is a marked difference in how the deck looks as Sam slides down the cable. It just looks like the weather has changed from bright and sunny to overcast suddenly, which doesn't take that much away from the actual gameplay.



Low End Cards





Since the second benchmark is so much more demanding graphically than the first, some of the lower-end cards had trouble running at anything above 800x600 with the quality settings on highest. For these cards, we only included tests at two resolutions (640x480 and 800x600) instead of three.

Something interesting here is that the game gets better performance out of even the lowest-end ATI card (X1300 XT) than the 7300 GS and GT cards, and even the 7600 GS trails the X1300 XT. On the very low end, the 7300 GT barely manages to run at 800x600, and with these high quality settings, you might still be able to enjoy the game at this resolution. The 7300 GS isn't so lucky, and it had to be excluded from the second test because it couldn't run the game any higher than 640x480 (and even then performance was poor), making it unfit for enabling the high quality settings regardless of resolution.





Switching to medium quality settings, in the first benchmark the 7300 GS gets a little improvement in performance, but still has trouble running above 800x600. Across the cards though, we do see improvements of up to ~40% in performance with NVIDIA and ATI by just disabling trilinear filtering, high quality soft shadows, and high detail shader in the display menu.

Notice that the 7300 GS, while barely being able to run the cruise ship benchmark at 800x600 with the highest quality settings, gets a respectable framerate at 1024x768 with the shadow effects turned off. The performance differences here are extreme, with the medium quality settings easily causing over a 100% improvement in performance over high quality. Turning off these effects will give more flexibility for running the game at higher resolutions, but we still recommend leaving them on and playing at lower resolutions if your card permits it.



NVIDIA AA Performance

Antialiasing was something that we wanted to look at with Double Agent, and we found there were some quirks with AA in the game. The game was released without any in-game AA options, but the 1.01 patch added the setting to the game menu. With NVIDIA hardware, Double Agent is unable to run AA with HDR effects enabled, which is something we saw in previous Splinter Cell games. ATI hardware, on the other hand, has no trouble running the game with both AA and HDR effects enabled, but unlike NVIDIA ATI cards had trouble running AA with HDR disabled. For this reason, we split up our AA tests into separate sections for NVIDIA and ATI. We aren't able to do an apples to apples comparison of how the game performs with AA between ATI and NVIDIA hardware, but we can see what kind of performance impact we will see with AA enabled for both types of cards.

We tested AA performance on the first benchmark, with Sam Fisher sliding down the zip-line at night. We saw that disabling HDR didn't make a very large performance impact on the game, and with NVIDIA hardware the game took a considerable performance hit with AA enabled. The fact that the game currently supports resolutions only up to 1600x1200 makes AA something that could be more useful, at least for those with larger displays and the GPU to handle it.

NVIDIA 4xAA Without HDR
	640x480	800x600	1024x768	1280x1024	1600x1200
NVIDIA GeForce 7300 GS	14.1	9.9
NVIDIA GeForce 7300 GT	23.3	17.4	12
NVIDIA GeForce 7600 GS	30.9	23.5	16.6	11.1
NVIDIA GeForce 7600 GT	46.6	35.4	25.2	17.1
NVIDIA GeForce 7900 GS	58.5	48	35.3	23.5	12.4
NVIDIA GeForce 7950 GT	59.8	57.4	44.3	31.6	23.4
NVIDIA GeForce 7900 GTX	60.2	59.1	49.4	35.5	26.4

With AA enabled and HDR off, the cards see much lower frame rates, with only cards like the 7900 GTX and 7950 GT really being able to run well at 1600x1200. Also, keep in mind that since we used the less demanding benchmark of the two (the other being on the cruise ship) for AA testing, performance in other areas of the game will be even worse with this option enabled. When we compare these results to our non-AA results for the same benchmark, we see that with AA off there is upwards of a 60% performance increase for most of the cards at the same resolutions. AA might still be helpful for cleaning up the jaggies on cards like the 7900 GS or 7600 GT which will still run the game more or less smoothly at 1280x1024 with AA enabled, especially if you don't have a display that supports higher resolutions.

ATI AA Performance

As we mentioned, ATI was able to run with AA and HDR enabled, but had some trouble running AA with HDR off. We saw some graphical problems which made the game unplayable at 1600x1200 resolution on all of the cards. These problems caused improper rendering of the scene making it impossible to see unless heat vision is enabled. Some of the other cards had the same problem at other resolutions as well. Therefore we tested ATI cards with HDR and AA enabled in order to see the kind of impact AA has on performance.

ATI 4xAA With HDR
	640x480	800x600	1024x768	1280x1024	1600x1200
ATI Radeon X1300 XT	32.5	24	16.3	11.2	7.2
ATI Radeon X1650 Pro	36.1	25.7	18.6	12.7	8.4
ATI Radeon X1650 XT	51.9	49.4	43.3	17.2	12.4
ATI Radeon X1900 XT 256	52.5	50.3	46.3	34.5	24.4
ATI Radeon X1950 Pro	53.7	49	40	28.3	21.2
ATI Radeon X1950 XTX	53.7	53.4	52.7	40.9	31.3

As with NVIDIA, turning on AA with these ATI cards does cause a large decrease in performance. The impact on performance here is greater than we saw with NVIDIA, but keep in mind that both HDR and AA are enabled for these tests. Lower performance cards like the X1300 XT just don't have the capacity to handle AA well and we see a large impact in performance here. The X1950 XTX on the other hand handles AA better, and though it still takes a performance hit it has more than enough power to keep the game running smoothly at 1600x1200 with AA enabled.

The fact that HDR seems to cause problems for ATI hardware when disabled while AA is enabled is unfortunate, but the performance difference between HDR on and off on ATI isn't that great. It would be much better to disable high quality soft shadows and/or high detail shader if you are looking to pick up some extra performance while running AA, for ATI as well as NVIDIA hardware.



Final Words

Splinter Cell: Double Agent is a very enjoyable game in spite of its bugs, and it does justice to the Splinter Cell series. While there are areas where the graphics could be improved, the SM3.0 effects make the game look very good. Sam Fisher and the other characters look better than in previous Splinter Cell games, and you can see the attention to detail the developers had with things like beard stubble and the sweat on his skin.

The gameplay is different than in previous Splinter Cells, and it's fun being able to play as a double agent, infiltrating the ranks of a secret terrorist organization. The game also gets harder as you progress, and it can get complicated trying to juggle objectives for both the NSA and JBA, but the greater challenges make the game that much more enjoyable. Playing the game in scenarios where there is broad daylight, which you have to do in much of the game, makes for an interesting challenge as well, especially for those who are familiar with the Splinter Cell series and are more comfortable slinking around in the dark like a cowardly harbinger of death.

Looking at performance with Double Agent, our tests clearly show that ATI cards get better performance compared to their direct-price competition from NVIDIA. There are a lot of potential reasons as to why this is the case, but we suspect it has something to do with a rush from the publisher to get the PC version of this Xbox 360 game out the door. We saw similar issues with The Elder Scrolls IV: Oblivion, which favored ATI hardware more than NVIDIA at the time of its release. Some NVIDIA cards run Double Agent well, but because of the fact that ATI did better over all at running the game than NVIDIA, an ATI part would be the wiser choice if you were looking for a card solely to run this game. Also, as we said earlier, Double Agent seems to run much more stably with ATI's hardware (without bugs, crashes, etc.) as opposed to NVIDIA's.

The fact that the game runs better on ATI hardware than NVIDIA's parts could come from a variety of factors - drivers, the programmers, and/or the specific types of shaders/effects used in the game. We can speculate (and hope) that as we saw with Oblivion, patches and driver updates will possibly fix the bugs and close the gap in performance between ATI and NVIDIA with this game. Even so, the fact that Splinter Cell: Double Agent does not currently support the NVIDIA 8800 series is unacceptable. Also, the problems with SLI and CrossFire are unfortunate, and it would have been nice if support had been provided for multi-GPU solutions. It may be possible that some of the effects used just aren't going to benefit from multi-GPU configurations, but whether or not Ubisoft puts any time in to fixing this, ATI and NVIDIA will almost certainly be looking for a solution.

Overall the game feels rushed, not in the storyline or gameplay necessarily, but in basic things like the interface and overall game stability. There are a lot of hardware incompatibilities which will leave some people, especially 8800 owners, with no current means to play the game after paying up their hard-earned money to buy it. It might be more understandable if the problems were with older-generation cards that were phased out, but on the 8800 - NVIDIA's flagship solution for highest performance - it's just not right.

We aren't sure why this latest addition to the Splinter Cell series was released with so many problems. Even though we've seen similar issues with newly-released games in the past, Double Agent happens to have more problems than usual for a game of this type. At any rate, we hope there will be some patches released soon that will fix these major issues. They should have been addressed before Double Agent's release, but the holiday season can cause some unfortunate effects on things like product launches. In the mean time, hopefully you have a better idea of how GPU performance stacks up in something other than a FPS for a change.