The XP Transition – Feel the Pain

Setup & Methodology

The first step was to select the target platforms. In order to give ourselves the greatest freedom when designing our test scenarios, we decided to forgo the various DOS-based Windows incarnations and focus exclusively on enterprise-caliber platforms based on the Windows NT kernel. This narrowed our field to Windows NT Workstation 4.0, Windows 2000 Professional and Windows XP (Whistler Beta 2 release - see our test note for why we didn't use RC1). By selecting only true 32-bit OS platforms we were able to construct a much more sophisticated simulation scenario than would be possible under the system resources-constrained Win9x architecture.

Next, we chose the application suites where once again our selection list was narrowed for us. Out of the various flavors and versions of Microsoft Office, only Office 2000 and the recently released Office XP include a comprehensive object model suitable for scripting. This, combined with their common application set (Word, Excel, PowerPoint), made a generational study of Office 2000 vs. Office XP the logical choice.

Finally, we had to settle on a test script. In this case, the choice was easy: OfficeBench 2001. Only CSA Research’s OfficeBench 2001 script could provide the kind of application and OS flexibility that our project required – application flexibility in that it works with both versions of Office; and OS flexibility in that it will install on all three of our target operating systems. Add to this the ability to apply concurrent workloads to the test mix, and we had the perfect tool for the job.

Of course, we still needed to decide on a common hardware test bed. In an effort to stay as mainstream as possible, we chose a genuine Intel D850GB motherboard with a healthy serving (256MB) of PC800 RDRAM, ATI Radeon DDR (64MB) video, Intel Pro 100+ Management NIC and driven by a Pentium 4 CPU running at 1.5GHz. It’s not the fastest system on the planet, but it is what most OEMs have been pitching to their enterprise IT customers as the current state of the art in PC design. Throw-in a set of 3 identical 7200RPM UDMA/66 EIDE disks (one for each OS version – alternated between test sessions), and we had what we felt to be a fairly representative PC test bed.  Note that the performance standings wouldn’t have changed had we used an AMD Athlon platform.

To fully explore the issue of generational OS and application performance we decided to run 2 different test scenarios against each of the 6 possible Microsoft Windows and Office combinations, for a total of 12 unique data points. The test scenarios themselves corresponded to two different OfficeBench 2001 loading levels:

Baseline Scenario – Involved booting the PC and running the OfficeBench 2001 Test Script by itself with none of the background Loading Tasks active on the system. In such a configuration, the scenario does little to tax the underlying OS and hardware multitasking capability.

Light Multitasking – Similar to the Baseline scenario except that this time the Loading Tasks are active at their lowest setting. Given the nature of the Load Simulators in OfficeBench 2001 – Database, Workflow and Multimedia – this translates into a moderately active knowledge worker environment where multiple, concurrent applications are running on the system and competing with the Test Script for CPU time and other resources.

In each scenario we began executed a single, “caching” run of the Test Script in order to force the OS to cache the various file images, eliminating application startup time as a factor. We then executed the scenarios in a 10-iteration loop, with OfficeBench 2001 automatically starting/stopping the Loading Tasks and recording the completion times for the individual test runs.

Other key steps included disabling all font smoothing and interface animation, running the disk defragmenter on the hard disk prior to testing and maintaining a constant color depth/display resolution (1024x768x16bpp at 75Hz). All OS installations were configured as Windows 2000 domain clients and connected via a full-duplex, 100Mbps Ethernet backbone.