Aggressive Tail Call Optimization

In some i386/x86_64 assembly code my coworker was working on, there was a macro like this:

#define ENDFUNC leave 

Having forgotten about the exact implementation of the macro, he then wrote a function that ended like this:


Now this obviously ended the function with


and he got very interesting results. What was happening?

7 thoughts on “Aggressive Tail Call Optimization

  1. 2500

    I think the doubled leave cleans up two stack frames, and this way restores the ESP/EPB pair of the grandparental function. Then ret returns to it. So this ‘just’ skips the rest of the direct calling function.
    What kind of side effects other than the obvious (memleaks and invalid return value) this can have is up to your imagination.

  2. mirabilos

    Yeah, it just returned to the function calling the caller.
    This won’t work on all arches, though, and only with an ABI
    that specifies to use enter/leave or their equivalent.

    After all, LEAVE = MOV SP,BP + RETN

    PS: posting here with Lynx isn’t possible, I had to use a Buntbrause ☹

  3. Wouter van Kleunen

    Leave =
    Set ESP to EBP, then pop EBP

    So esp will become epb
    esp will become esp – 4 bytes
    from the stack again esp is read, which can be any value
    esp will become any value – 4 bytes
    ebp will be popped from this strange stack

  4. mirabilos

    Hmpf, I seem to have remembered the LEAVE expansion wrong
    and stand corrected.

    Nevertheless, it probably will clean up two stack frames and return
    to the grandfather function, but not restore any pushed-away non-
    modifiable registers (such as EBX, ESI, EDI by the ELF ABI).

  5. gwenhwyfaer

    It’s actually relatively well known, at least in some circles; Thinking Forth describes doing something similar, to effect a conditional return from a function. It complicates control flow, but can simplify the appearance of code by containing exceptional conditions – and it’s not a million miles removed from exceptions.


Leave a Reply to Wouter van Kleunen Cancel reply

Your email address will not be published. Required fields are marked *