Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Demonstrates gcc 4.2.1 variadic function weirdness for x86_64 architecture
#import <Foundation/Foundation.h>
#import <objc/runtime.h>
static id pear()
return @"pear";
static id banana(id self, SEL _cmd, ...)
return @"banana";
@interface Fruit : NSObject
@implementation Fruit
- (id) apple
return @"apple";
int main (int argc, char const *argv[])
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
FILE *devnull = fopen("/dev/null", "w");
Method apple = class_getInstanceMethod([Fruit class], @selector(apple));
method_setImplementation(apple, banana);
Fruit *fruit = [[[Fruit alloc] init] autorelease];
fprintf(devnull, "(0x100000c3c - 0x100000bac) / 4 = 36");
printf("%s\n", [[fruit apple] UTF8String]);
[pool drain];
return 0;
# $ gcc --version
# i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3)
# Copyright (C) 2007 Free Software Foundation, Inc.
# This is free software; see the source for copying conditions. There is NO
# $ clang --version
# Apple clang version 1.7 (tags/Apple/clang-77) (based on LLVM 2.9svn)
# Target: x86_64-apple-darwin10
# Thread model: posix
gcc -arch x86_64 fruit.m -o fruit-gcc -framework Foundation
clang -arch x86_64 fruit.m -o fruit-clang -framework Foundation
rm -f fruit-gcc fruit-clang
run: all

This comment has been minimized.

Copy link
Owner Author

@0xced 0xced commented Oct 5, 2019

Result with clang: banana
Result with gcc: pear

Explanation: I don't really remember, it was 9 years ago, I should have written it at that time! It has something to do with how variadic functions are called in the x64 ABI. The %rax register is clobbered by the fprintfcall with the value 36 (the length of the string "(0x100000c3c - 0x100000bac) / 4 = 36"). I think banana was at address 0x100000c3c and pear was at address 0x100000bac. I remember adding the __asm__("nop"); so that the pear function would be at an address which is a multiple of 4.


This comment has been minimized.

Copy link

@gparker42 gparker42 commented Oct 5, 2019

The path to gcc's unexpected result is in two parts. First, the ObjC runtime shenanigans lead to a non-variadic call site that calls a variadic implementation, which leaves one register value uninitialized. Second, gcc's codegen performs an out-of-bounds jump when given that uninitialized value, which happens to jump to instructions that return the wrong value without crashing.

On x86_64, a call site for a variadic function (one with ... in the parameter list) sets register %al with the number of XMM registers that hold parameter values. There are eight XMM parameter registers, so this value ought to be 0...8. Non-variadic call sites don't need to set %al.

The call site in question is [fruit apple]. The compiler sees that the method signature for -apple is non-variadic. Thus the generated code for the call [fruit apple] does not set %al. Its value happened to be 36 as returned by fprintf.

At runtime the call site [fruit apple] instead jumps to the implementation of -banana thanks to the earlier method_setImplementation call. -banana is variadic and expects %al to have been set by the caller. It wasn't.

gcc-4.2.1's prologue for -banana used a computed branch to save the XMM registers. It looks something like this:

  jump end - %al * 4;  // each store instruction is 4 bytes long
  end-32: store %xmm7;
  end-28: store %xmm6;
  end-24: store %xmm5;
  end-20: store %xmm4;
  end-16: store %xmm3;
  end-12: store %xmm2;
  end-8:  store %xmm1;
  end-4:  store %xmm0;
  end:    ...

This works fine when %al is within 0...8 as expected, but when %al is 36 it jumps backwards a few dozen bytes. In this case it happened to jump into -pear at an instruction that returned @"pear" successfully.

This behavior is of course extremely fragile. Turning on compiler optimizations probably removes the XMM-saving prologue entirely. Almost any code change will lead to a different value in %al at the call site or emit different instructions at the destination of the wild jump. Generating code that returns the wrong value without crashing was unlikely.

clang-1.7's variadic prologue was different. It simply checks if %al is zero and saves either zero or eight XMM registers. It doesn't care if %al is too big.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment