UNIX, Windows NT, and all the operating systems in their class rely on virtual memory, or paging, in order to provide every process on the system a complete address space of its own. An easier way to protect processes from each other is segmentation: The 4 GB address space of a 32 bit CPU is divided into segments (consisting of a physical base address and a limit), one for each process, and every process may only access their own segment. This is what the 286 did.
The 386 then introduced virtual memory, but segmentation was still possible, either instead of, or on top of the paged virtual address space. Today, no modern operating system for the x86 uses segmentation any more, so for every process, the base for the code and data segments is set to 0, and the limit is set to 0xFFFFFFFF.
The AMD64 architecture, while still being fully compatible in 32 bit mode, retired a lot of legacy functionality in the new 64 bit long mode, including most of segmentation. The CS (code), DS (data 1), ES (data 2) and SS (stack) segment registers are practically gone, and the FS and GS segments still support a base (which can be used in tricks to quickly access data at a constant position, like the TCB), but the limit is no longer enforced. Now operating systems don’t have to save and restore most of these segment registers any more when switching contexts, making these switches faster.
But this broke VMware. While VMware could still virtualize 32 bit operating systems on AMD64 CPUs, they could not virtualize 64 bit operating systems, because they required segment limits.
In a nutshell, this is how VMware works: All user mode code of the guest runs in exactly the environment it expects; VMware makes sure the page mappings of the user mode address spaces are correct. All kernel mode code of the guest will be run in user mode, and again, VMware must layout memory as the guest kernel expects it to be. In both modes of operation, there can be exceptions, like system calls (by guest user mode code) or page table modifications (by guest kernel mode code). These have to be trapped by the virtual machine monitor, and the respective functionality has to be carried out in a modified way, so that they still seem to have the correct effect to the guest, but don’t interfere with the host operating system.
The virtual machine monitor’s trap handler must reside in the guest’s address space, because an exception cannot switch address spaces. So VMware’s trap handler sits at the very top of the every guest’s address space, which is unused by all major operating systems. According to Popek and Goldberg’s definition of virtualization, there must be no way for code inside a virtual machine to escape, and modify the host’s state in any way not directly controlled by the monitor. Therefore, it must be made sure that the guest code cannot write to the trap handler code. VMware does this using segment limits: The limits of all segment registers are set to something like 0xFFFFEFFF to protect the uppermost 4 KB of the address space where the trap handler resides.
With no segment limits any more for 64 bit code, this way to protect the trap handler was impossible. Unable to comply with Popek and Goldberg’s security requirement, VMware chose not to support 64 bit virtualization until AMD reintroduced (optional) segment limits on later models of their Opteron and Athlon 64 CPUs. Intel never implemented 64 bit segment limits on their EM64T/Intel64 CPUs, because their 64 bit processors soon implemented VT/Vanderpool, which also worked around the problem. So this is why VMware requires a certain model and stepping of the AMD CPU line or a VT-enabled Intel CPU in order to support 64 bit virtualization.
Now the question is: Why don’t they protect the uppermost page using the permission bits in the page table? This is how all operating systems protect themselves from user mode processes. If you have an answer on this, or otherwise have thoughts, please comment on this post.