Aggressive Tail Call Optimization

In some i386/x86_64 assembly code my coworker was working on, there was a macro like this:

#define ENDFUNC leave 
                ret

Having forgotten about the exact implementation of the macro, he then wrote a function that ended like this:

        leave
        ENDFUNC

Now this obviously ended the function with

        leave
        leave
        ret

and he got very interesting results. What was happening?

7 thoughts on “Aggressive Tail Call Optimization”

  1. I think the doubled leave cleans up two stack frames, and this way restores the ESP/EPB pair of the grandparental function. Then ret returns to it. So this ‘just’ skips the rest of the direct calling function.
    What kind of side effects other than the obvious (memleaks and invalid return value) this can have is up to your imagination.

  2. Yeah, it just returned to the function calling the caller.
    This won’t work on all arches, though, and only with an ABI
    that specifies to use enter/leave or their equivalent.

    After all, LEAVE = MOV SP,BP + RETN

    PS: posting here with Lynx isn’t possible, I had to use a Buntbrause ☹

  3. Leave =
    Set ESP to EBP, then pop EBP

    So esp will become epb
    esp will become esp – 4 bytes
    from the stack again esp is read, which can be any value
    esp will become any value – 4 bytes
    ebp will be popped from this strange stack

  4. Hmpf, I seem to have remembered the LEAVE expansion wrong
    and stand corrected.

    Nevertheless, it probably will clean up two stack frames and return
    to the grandfather function, but not restore any pushed-away non-
    modifiable registers (such as EBX, ESI, EDI by the ELF ABI).

  5. It’s actually relatively well known, at least in some circles; Thinking Forth describes doing something similar, to effect a conditional return from a function. It complicates control flow, but can simplify the appearance of code by containing exceptional conditions – and it’s not a million miles removed from exceptions.

Leave a Reply to Wouter van Kleunen Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.