How retiring segmentation in AMD64 long mode broke VMware

UNIX, Windows NT, and all the operating systems in their class rely on virtual memory, or paging, in order to provide every process on the system a complete address space of its own. An easier way to protect processes from each other is segmentation: The 4 GB address space of a 32 bit CPU is divided into segments (consisting of a physical base address and a limit), one for each process, and every process may only access their own segment. This is what the 286 did.

The 386 then introduced virtual memory, but segmentation was still possible, either instead of, or on top of the paged virtual address space. Today, no modern operating system for the x86 uses segmentation any more, so for every process, the base for the code and data segments is set to 0, and the limit is set to 0xFFFFFFFF.

The AMD64 architecture, while still being fully compatible in 32 bit mode, retired a lot of legacy functionality in the new 64 bit long mode, including most of segmentation. The CS (code), DS (data 1), ES (data 2) and SS (stack) segment registers are practically gone, and the FS and GS segments still support a base (which can be used in tricks to quickly access data at a constant position, like the TCB), but the limit is no longer enforced. Now operating systems don’t have to save and restore most of these segment registers any more when switching contexts, making these switches faster.

But this broke VMware. While VMware could still virtualize 32 bit operating systems on AMD64 CPUs, they could not virtualize 64 bit operating systems, because they required segment limits.

In a nutshell, this is how VMware works: All user mode code of the guest runs in exactly the environment it expects; VMware makes sure the page mappings of the user mode address spaces are correct. All kernel mode code of the guest will be run in user mode, and again, VMware must layout memory as the guest kernel expects it to be. In both modes of operation, there can be exceptions, like system calls (by guest user mode code) or page table modifications (by guest kernel mode code). These have to be trapped by the virtual machine monitor, and the respective functionality has to be carried out in a modified way, so that they still seem to have the correct effect to the guest, but don’t interfere with the host operating system.

The virtual machine monitor’s trap handler must reside in the guest’s address space, because an exception cannot switch address spaces. So VMware’s trap handler sits at the very top of the every guest’s address space, which is unused by all major operating systems. According to Popek and Goldberg’s definition of virtualization, there must be no way for code inside a virtual machine to escape, and modify the host’s state in any way not directly controlled by the monitor. Therefore, it must be made sure that the guest code cannot write to the trap handler code. VMware does this using segment limits: The limits of all segment registers are set to something like 0xFFFFEFFF to protect the uppermost 4 KB of the address space where the trap handler resides.

With no segment limits any more for 64 bit code, this way to protect the trap handler was impossible. Unable to comply with Popek and Goldberg’s security requirement, VMware chose not to support 64 bit virtualization until AMD reintroduced (optional) segment limits on later models of their Opteron and Athlon 64 CPUs. Intel never implemented 64 bit segment limits on their EM64T/Intel64 CPUs, because their 64 bit processors soon implemented VT/Vanderpool, which also worked around the problem. So this is why VMware requires a certain model and stepping of the AMD CPU line or a VT-enabled Intel CPU in order to support 64 bit virtualization.

Now the question is: Why don’t they protect the uppermost page using the permission bits in the page table? This is how all operating systems protect themselves from user mode processes. If you have an answer on this, or otherwise have thoughts, please comment on this post. :-)

References: 1, 2, 3, 4, 5

32 thoughts on “How retiring segmentation in AMD64 long mode broke VMware

  1. James Ideal

    Mmmm… good question. My uniformed guess will be that with the segment protection model you have a finer grained protection model, but in the case of the MMU model you only have one bit per PDT: (U)ser/(S)ystem where 3=U and 0,1,2 = S, so it might come to excesive overhead if vmware also needs to babysit the MMU traps to implement yet another protection level.

    Reply
  2. seppel

    You wrote: “The virtual machine monitor’s trap handler must reside in the guest’s address space, because an exception cannot switch address spaces.” I still think that this is wrong. I haven’t tried it but I’m quite sure you can place a task gate in the IDT pointing to a TSS which contains cr3 and so can switch address spaces.

    Reply
  3. Matthias

    I think they prefer segments because they can prevent the OS from reading VMware code, which should ideally be impossible. As far as I know, you can’t do executable-without-read with the MMU.

    Reply
  4. Ghostwriter

    James Ideal is dead on, though he incorrectly implies that you can solve the issue by interposing on MMU traps. x86 may have 4 rings, but as far as paging is concerned, it only has 0 and 1-3. If you use ring0 to protect your monitor, you have no way of protecting guest kernelspace from guest userspace.

    Interestingly, this means that Xen/x86-64 (since even AMD64 revD’s re-enabling of segmentation doesn’t re-enable %fs’s limit) runs both guest kernel space and guest user space in ring3 (and the monitor in ring0, of course), with two (mostly) disjoint sets of page tables. This makes system calls quite a bit more expensive, but does provide all the various isolation you’d expect from both the OS and the VMM.

    Reply
  5. dex

    Using WinDbg, I dumped CS GDT descriptor (index 1 and index 3) under VMware, the segment size is 4GB.
    How come?

    Reply
  6. Myria

    dex: WinDbg uses GetThreadSelectorEntry (NtQueryInformationThread), an operating system call. The kernel reads the GDT to get this information for you, which is the virtual GDT.

    Try using the “lsl” instruction in user mode, which can’t be faked because it doesn’t cause an exception. I don’t have VMware at home so I can’t try this right now.

    Reply
  7. ZPulse

    Ghostwriter, isn’t rings 0-2 or 3 for paging (and not 0,1-3)? Hence, guest OS needs to reside in 3 so as not to affect the VMM.

    Reply
  8. Pingback: penicillin

  9. Pingback: low cost meridia

  10. walken

    A pure trap handler would not work, because some instructions that expose privileged state do not trap on x86. vmware uses binary translation to work around this, but this requires to run translated code in the protected upper area, which is why they don’t use permission bits. The asplos paper you cited as your 4th reference gives a small example of binary translation at work.

    Reply
  11. LolitochkaBC

    Аааану-ка ребяти голоусме!!!

    Призннавайтесь проказники и владельцы сайта http://www.pagetable.com ))))

    ЧТО вы будет делать этимм летомп?!

    Reply
  12. Pingback: 71b39b1b9e9b64828c198e6252dea268

  13. Pingback: Hairy Old Snatch

  14. Pingback: Old Moms And Teens

  15. AltaGid

    Hello! Help solve the problem.
    Very often try to enter the site, but says that the password is not correct.
    Regrettably use of remembering. Give like to be?
    Thank you!

    Reply
  16. Yuhong Bao

    Anyone have a reference to programming info on how to take advantage of the support for the 64-bit segment limits?

    Reply
  17. Interested

    So uh, i just happen to be running a virtual x86/amd64 machine in vmware, on a host 32bit OS. I use my virtual machines to do alot of tinkering with programs etc, because *nix is obviously the choice OS for this. Anyway, my question is, could this account for a segfault in some assembly code im tinkering with? Like ive read that when writing asm, the .data segment should be read/write. Knowing this, i couldnt figure out this segfault till i found valgrind for some debugging. Im new to assembly(very new even) and i thought for sure i was the one at fault. But then valgrind reports that i dont have proper permissions to modify this memory location(the .data segment, im doing a self modifying string.) So I guess my actual question is, am i at fault, is it something particular with amd64 assembly, or is it something to do with running a 64bit guest OS on a 32bit host OS in vmware?

    Reply
  18. Interested

    Was able to get the results i wanted by manually passing the address of .data to ld by compiling, and finding my original string. This still seems strange to me tho, it doesnt seem to matter how i define the .data section in the source it gets screwed up so i have to manually examine the compiled binary and then do the whole process over each time i change things. Any examples i can follow for nasm syntax to see if im messing up the section declaration? Im already into the nasm docs, but im reading slowly over a whole base of knowledge trying to get a good jump into asm. Still kind of new at this tho, so im wondering is this kind of thing the usual? Seems like I might be keeping my headache for a while if it is.

    Reply
  19. adx

    Myria, they have to virtualize LSL just like they have to virtualize MOV AX, CS and similar. (Unless they do something extraordinary tricky, like running Ring-3 with selector like on Ring-0 … is that possible?)

    As far as I know, there is a thing called code scanning (up to branches and then again), although this is not very clear to me. For if code scanning put INT3 opcodes, then there’s again the problem what to do to avoid seeing changed code by the guest…

    Reply
  20. chaoxifer

    This makes me to solve some questions:

    1. Why segmentation still exists?
    2. The “segmentation” can be supported by other architecture?
    3. is that possible to occur overhead by using segmentation while translating between virtual address and physical address?

    Thanks for posting Good information.

    Reply
  21. MSFT

    Retiring the bi-endian-ness flag of the PowerPC, on the PowerPC G5, broke VirtualPC, too.

    Virtual machines are arguably the most brutal piece of software you can run. Nothing taxes more obscure parts of your system than a VM.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>