How retiring segmentation in AMD64 long mode broke VMware

UNIX, Windows NT, and all the operating systems in their class rely on virtual memory, or paging, in order to provide every process on the system a complete address space of its own. An easier way to protect processes from each other is segmentation: The 4 GB address space of a 32 bit CPU is divided into segments (consisting of a physical base address and a limit), one for each process, and every process may only access their own segment. This is what the 286 did.

The 386 then introduced virtual memory, but segmentation was still possible, either instead of, or on top of the paged virtual address space. Today, no modern operating system for the x86 uses segmentation any more, so for every process, the base for the code and data segments is set to 0, and the limit is set to 0xFFFFFFFF.

The AMD64 architecture, while still being fully compatible in 32 bit mode, retired a lot of legacy functionality in the new 64 bit long mode, including most of segmentation. The CS (code), DS (data 1), ES (data 2) and SS (stack) segment registers are practically gone, and the FS and GS segments still support a base (which can be used in tricks to quickly access data at a constant position, like the TCB), but the limit is no longer enforced. Now operating systems don’t have to save and restore most of these segment registers any more when switching contexts, making these switches faster.

But this broke VMware. While VMware could still virtualize 32 bit operating systems on AMD64 CPUs, they could not virtualize 64 bit operating systems, because they required segment limits.

In a nutshell, this is how VMware works: All user mode code of the guest runs in exactly the environment it expects; VMware makes sure the page mappings of the user mode address spaces are correct. All kernel mode code of the guest will be run in user mode, and again, VMware must layout memory as the guest kernel expects it to be. In both modes of operation, there can be exceptions, like system calls (by guest user mode code) or page table modifications (by guest kernel mode code). These have to be trapped by the virtual machine monitor, and the respective functionality has to be carried out in a modified way, so that they still seem to have the correct effect to the guest, but don’t interfere with the host operating system.

The virtual machine monitor’s trap handler must reside in the guest’s address space, because an exception cannot switch address spaces. So VMware’s trap handler sits at the very top of the every guest’s address space, which is unused by all major operating systems. According to Popek and Goldberg’s definition of virtualization, there must be no way for code inside a virtual machine to escape, and modify the host’s state in any way not directly controlled by the monitor. Therefore, it must be made sure that the guest code cannot write to the trap handler code. VMware does this using segment limits: The limits of all segment registers are set to something like 0xFFFFEFFF to protect the uppermost 4 KB of the address space where the trap handler resides.

With no segment limits any more for 64 bit code, this way to protect the trap handler was impossible. Unable to comply with Popek and Goldberg’s security requirement, VMware chose not to support 64 bit virtualization until AMD reintroduced (optional) segment limits on later models of their Opteron and Athlon 64 CPUs. Intel never implemented 64 bit segment limits on their EM64T/Intel64 CPUs, because their 64 bit processors soon implemented VT/Vanderpool, which also worked around the problem. So this is why VMware requires a certain model and stepping of the AMD CPU line or a VT-enabled Intel CPU in order to support 64 bit virtualization.

Now the question is: Why don’t they protect the uppermost page using the permission bits in the page table? This is how all operating systems protect themselves from user mode processes. If you have an answer on this, or otherwise have thoughts, please comment on this post. :-)

References: 1, 2, 3, 4, 5

32 thoughts on “How retiring segmentation in AMD64 long mode broke VMware”

  1. Mmmm… good question. My uniformed guess will be that with the segment protection model you have a finer grained protection model, but in the case of the MMU model you only have one bit per PDT: (U)ser/(S)ystem where 3=U and 0,1,2 = S, so it might come to excesive overhead if vmware also needs to babysit the MMU traps to implement yet another protection level.

  2. You wrote: “The virtual machine monitor’s trap handler must reside in the guest’s address space, because an exception cannot switch address spaces.” I still think that this is wrong. I haven’t tried it but I’m quite sure you can place a task gate in the IDT pointing to a TSS which contains cr3 and so can switch address spaces.

  3. I think they prefer segments because they can prevent the OS from reading VMware code, which should ideally be impossible. As far as I know, you can’t do executable-without-read with the MMU.

  4. James Ideal is dead on, though he incorrectly implies that you can solve the issue by interposing on MMU traps. x86 may have 4 rings, but as far as paging is concerned, it only has 0 and 1-3. If you use ring0 to protect your monitor, you have no way of protecting guest kernelspace from guest userspace.

    Interestingly, this means that Xen/x86-64 (since even AMD64 revD’s re-enabling of segmentation doesn’t re-enable %fs’s limit) runs both guest kernel space and guest user space in ring3 (and the monitor in ring0, of course), with two (mostly) disjoint sets of page tables. This makes system calls quite a bit more expensive, but does provide all the various isolation you’d expect from both the OS and the VMM.

  5. Using WinDbg, I dumped CS GDT descriptor (index 1 and index 3) under VMware, the segment size is 4GB.
    How come?

  6. dex: WinDbg uses GetThreadSelectorEntry (NtQueryInformationThread), an operating system call. The kernel reads the GDT to get this information for you, which is the virtual GDT.

    Try using the “lsl” instruction in user mode, which can’t be faked because it doesn’t cause an exception. I don’t have VMware at home so I can’t try this right now.

  7. Ghostwriter, isn’t rings 0-2 or 3 for paging (and not 0,1-3)? Hence, guest OS needs to reside in 3 so as not to affect the VMM.

  8. Pingback: penicillin
  9. Pingback: low cost meridia
  10. A pure trap handler would not work, because some instructions that expose privileged state do not trap on x86. vmware uses binary translation to work around this, but this requires to run translated code in the protected upper area, which is why they don’t use permission bits. The asplos paper you cited as your 4th reference gives a small example of binary translation at work.

  11. Аааану-ка ребяти голоусме!!!

    Призннавайтесь проказники и владельцы сайта ))))

    ЧТО вы будет делать этимм летомп?!

  12. Pingback: Hairy Old Snatch
  13. Pingback: Old Moms And Teens
  14. Hello! Help solve the problem.
    Very often try to enter the site, but says that the password is not correct.
    Regrettably use of remembering. Give like to be?
    Thank you!

  15. Anyone have a reference to programming info on how to take advantage of the support for the 64-bit segment limits?

  16. So uh, i just happen to be running a virtual x86/amd64 machine in vmware, on a host 32bit OS. I use my virtual machines to do alot of tinkering with programs etc, because *nix is obviously the choice OS for this. Anyway, my question is, could this account for a segfault in some assembly code im tinkering with? Like ive read that when writing asm, the .data segment should be read/write. Knowing this, i couldnt figure out this segfault till i found valgrind for some debugging. Im new to assembly(very new even) and i thought for sure i was the one at fault. But then valgrind reports that i dont have proper permissions to modify this memory location(the .data segment, im doing a self modifying string.) So I guess my actual question is, am i at fault, is it something particular with amd64 assembly, or is it something to do with running a 64bit guest OS on a 32bit host OS in vmware?

  17. Was able to get the results i wanted by manually passing the address of .data to ld by compiling, and finding my original string. This still seems strange to me tho, it doesnt seem to matter how i define the .data section in the source it gets screwed up so i have to manually examine the compiled binary and then do the whole process over each time i change things. Any examples i can follow for nasm syntax to see if im messing up the section declaration? Im already into the nasm docs, but im reading slowly over a whole base of knowledge trying to get a good jump into asm. Still kind of new at this tho, so im wondering is this kind of thing the usual? Seems like I might be keeping my headache for a while if it is.

  18. Myria, they have to virtualize LSL just like they have to virtualize MOV AX, CS and similar. (Unless they do something extraordinary tricky, like running Ring-3 with selector like on Ring-0 … is that possible?)

    As far as I know, there is a thing called code scanning (up to branches and then again), although this is not very clear to me. For if code scanning put INT3 opcodes, then there’s again the problem what to do to avoid seeing changed code by the guest…

  19. This makes me to solve some questions:

    1. Why segmentation still exists?
    2. The “segmentation” can be supported by other architecture?
    3. is that possible to occur overhead by using segmentation while translating between virtual address and physical address?

    Thanks for posting Good information.

  20. Retiring the bi-endian-ness flag of the PowerPC, on the PowerPC G5, broke VirtualPC, too.

    Virtual machines are arguably the most brutal piece of software you can run. Nothing taxes more obscure parts of your system than a VM.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.