Introduction

NX protection seems great; it stops viruses dead in their tracks and eliminates those pesky buffer overflows we have been hearing so much about for the last 15 years. Well, maybe not. In fact it seems that NX provides several layers of false security, particularly since it only stops some buffer overflows and whether or not it stops any viruses has yet to be seen yet.

Although AMD and Intel both like to spin their newest NX/XD "Virus Protection" as a new feature on x86 technology, in reality NX behaves more like an emergency patch for an easily exploitable architecture. To really explore how NX benefits us, and where it provides a false sense of security, we must understand one of the scenarios that it supposedly prevents against.

The simplest explanation of a buffer overflow is when a program executes memory it was not intending to. Typically, this is done when a program writes something into memory that it has not properly allocated for; sometimes writing over pieces of the same program already in memory.

For example, let's take a look at the following program code:

int main(int argc, char **argv)
{
           char line[512];
           line[0] = 0;
           gets(line); 
           ...
           return 0;
}

Undoubtedly, even those who have taken only rudimentary programming classes are having a good chuckle with the monstrosity above. However, this code is actually nearly verbatim copied from the original UNIX fingerd program circa 1988. In UNIX (or any other operating system), the entire contents of the program are loaded into memory as machine code. A simple, although not necessarily proper, way to think about the above code is an array of several hundred bytes in memory; 512 bytes allocated for the array of characters, line, on line 3 of the code.

As the program starts flipping through the code in its array (which is actually called a stack), it reaches the function gets(). gets(), a horrible, ancient function, reads input and places it into the line array. But line has only enough memory allocated for 512 characters! It would only take a malicious user a few seconds to realize that writing more than 512 characters to the fingerd program would result in the ability to write data past the end of the line array. And since line exists inside a stack in memory, writing past the end of line actually starts to overwrite critical pieces of the program! If a user were to write 512 characters to fingerd followed by machine code, the user could essentially control any attribute of the entire machine. The example with fingerd above became the first real internet worm exploit, a buffer overflow. Worms like Nimda and Code Red work exactly the same as the original fingerd worm, so unfortunately 16 years later imprudent programming and exploitable hardware are still to blame.

What does it all mean?

It was probably mid 2003 after the effects of Sapphire, Nimda and Code Red that Intel and AMD both started working with Microsoft to patch the x86 architecture itself. The inherent flaw is two fold; for one, malicious code should not write past the end of an array like line from fingerd, and second the computer should not continue to happily execute the machine code on the stack.

For those not familiar with the stack, here is a quick illustration of how memory looks to an x86 based machine. All memory is divided up into several segments; the stack, heap, and code (or text). Segments are then further divided into pages. Generally, a program executes and runs from the "code" segment. The heap stores dynamically loaded memory, static variables, global variables - anything that need exist during the duration of the program's executable life. The stack is the wild west of memory; function parameters, general variables and arrays end up here. Memory on the stack is reused; once we have processed a function whatever junk it left over in memory resides there.

In the fingerd example from the previous page, writing more than 512 characters to the array begins to overwrite components of the program. When a program begins to load an array, a special byte after the array (called a return address) tells the computer to go back to the code segment and continue to run the program. Unfortunately if a malicious input overwrites the return address with an address in the stack, the computer will execute the code at the location designated, rather than the code the return address pointed to.

NX attempts to correct the buffer overflow issue by disallowing the computer from executing memory in specific pages of the stack. Thus if a malicious input overwrites the portion of the stack and the return address with a clever piece of machine code, NX signals the kernel to panic and the program is terminated.

Thus, some issues are addressed with buffer overflows. It makes it much harder for a malicious input to load a payload into memory with any astounding effects, since we cannot execute some pages of memory on the stack. What NX doesn't stop are buffer overflows that do not rely on their own payload. A malicious input can still overwrite the return address and direct exectution to some other code segment that can cause nefarious deeds - such as a function that creates a file before the user is correctly identified as having the rights to do so. The program might also utilize shared libraries and external system functions like execve, which would give the input identical privileges to the user executing the program. A clever return address combined with initialization of some parameters on the stack would result in execve replicating a payload without machine code.

This is not to say that NX protection is not a step in the right direction. In fact, NX/XD is a good first step to locking down the x86 architecture, as long as it's adopted correctly. OpenBSD and the Execshield projects have made the largest progress with implementing non-executable writable pages and other features, if only in software. However, NX does not completely eliminate buffer overflow exploits, and thus far it has only caused more problems than it has solved with Windows SP2.

Special thanks to D.J. Bernstein, who provided the technical background for the majority of this primer.