Aggressive Tail Call Optimization

In some i386/x86_64 assembly code my coworker was working on, there was a macro like this:

#define ENDFUNC leave \
                ret

Having forgotten about the exact implementation of the macro, he then wrote a function that ended like this:

        leave
        ENDFUNC

Now this obviously ended the function with

        leave
        leave
        ret

and he got very interesting results. What was happening?

pixelstats trackingpixel

6 Responses to “Aggressive Tail Call Optimization”

  1. 2500 says:

    I think the doubled leave cleans up two stack frames, and this way restores the ESP/EPB pair of the grandparental function. Then ret returns to it. So this ‘just’ skips the rest of the direct calling function.
    What kind of side effects other than the obvious (memleaks and invalid return value) this can have is up to your imagination.

  2. mirabilos says:

    Yeah, it just returned to the function calling the caller.
    This won’t work on all arches, though, and only with an ABI
    that specifies to use enter/leave or their equivalent.

    After all, LEAVE = MOV SP,BP + RETN

    PS: posting here with Lynx isn’t possible, I had to use a Buntbrause ☹

  3. Wouter van Kleunen says:

    Leave =
    Set ESP to EBP, then pop EBP

    So esp will become epb
    esp will become esp – 4 bytes
    from the stack again esp is read, which can be any value
    esp will become any value – 4 bytes
    ebp will be popped from this strange stack

  4. Wouter van Kleunen says:

    That any value is the return address of the function call

  5. mirabilos says:

    Hmpf, I seem to have remembered the LEAVE expansion wrong
    and stand corrected.

    Nevertheless, it probably will clean up two stack frames and return
    to the grandfather function, but not restore any pushed-away non-
    modifiable registers (such as EBX, ESI, EDI by the ELF ABI).

  6. gwenhwyfaer says:

    It’s actually relatively well known, at least in some circles; Thinking Forth describes doing something similar, to effect a conditional return from a function. It complicates control flow, but can simplify the appearance of code by containing exceptional conditions – and it’s not a million miles removed from exceptions.

Leave a Reply

*
To prove you're a person (not a spam script), type the security word shown in the picture. Click on the picture to hear an audio file of the word.
Click to hear an audio file of the anti-spam word