Notes on calling convention

calling_conventions.md

Common calling conventions

• cdecl
• fastcall
• stdcall

CDECL calling convention.

cdecl stands for "C declaration", it is used by most c compiler in the x86 architecture.

Arguments passing in cdecl calling convention

Arguments in cdecl calling convention are passed on the stack and the return value is stored in EAX register. The argument are passed in right-to-left/reverse order, .i.e The last argument is pushed first.

Code example

Lets consider this c code snippet which is using cdecl calling convention.

int callee(int, int, int);

int caller(void)
{
	return callee(1, 2, 3) + 5;
}

x86 assembly of this c code with the intel syntax.

caller:
	; stack setup
	push ebp ; save old stack frame.
	mov ebp, esp ; initialize new stack frame.
	push 3 ; last argument is pushed to stack.
	push 2 ; last second argument pushed.
	push 1 ; first argument pushed.
	call callee ; callee funciton is called.
	add rsp, 12 ; restoring the stack frame.
	add eax, 5 ; adding 5 to the result of the function call.
	; stack destruction
	mov esp, ebp 
	pop ebp
	ret

As we can see that the caller cleans the stack after the function returns.

add rsp, 12

Fastcall calling convention.

fastcall is also called as Microsoft fastcall, __fastcall, __msfastcall.

Arguments passing in fastcall calling convention

The first two arguments are passed in ECX and EDX registers and remaining arguments are passed to the stack. Unlike the argument passing in cdecl convention, the first arguments in msfastcall are passed in left to right order in the ECX and EDX registers and the rest in right to left order. Stack cleanup in msfastcall is performed by the callee.

Code example

__attribute__((fastcall)) void printnums(int num1, int num2, int num3){
	printf("The numbers you sent are: %d %d %d", num1, num2, num3);
}

int main(){
	printnums(3, 2, 1);
	return 0;
}

The x86 disassembly of the main function looks like this.

main:
	; stack setup
	push ebp
	mov ebp, esp
	push 3 ; immidiate 3(first argument is pushed to the stack)
	mov edx, 0x2 ; immidiate 2(second argument) is copied to edx register.
	mov ecx, 0x1 ; immidiate 1(third argument) is copied to edx register.
	call printnums
	mov eax, 0 ; return 0
	leave
	retn

As we can see, the first two arguments are passed in the left to right order and the third argument is pushed in the stack but there is no stack cleanup, right? As we know that in msfastcall the stack cleanup is performed by the callee. Lets see the disassembly of the callee function to have a look at the stack cleanup also.

printnums:
	; stack setup
	push ebp
	mov ebp, esp
	sub esp, 0x08
	mov [ebp-0x04], ecx    ; in x86, ecx = first argument.
	mov [ebp-0x08], edx    ; arg2
	push [ebp+0x08]        ; arg3 is pushed to stack.
	push [ebp-0x08]        ; arg2 is pushed
	push [ebp-0x04]        ; arg1 is pushed
	push 0x8065d67         ; "The numbers you sent are %d %d %d"
	call printf
	; stack cleanup
	add esp, 0x10
	nop
	leave
	retn 0x04

Note that the printf function is by default cdecl calling convention, so it will be called in the default way it is called. Here we can see the stack cleanup clearly. As the two arguments were passed through the registers and only one parameter was pushed in the stack, the pushed value is being cleared by retn instruction as int is 4 bytes in size in x86 systems.

stdcall calling convention.

This calling convention is pretty straightforward and can be explained easily if you have understood the cdecl and msfastcalll calling conventions.

Arguments passing in stdcall calling convention.

Like cdecl, this calling convention also pushes all the arguments in the stack in right to left order. Like fastcall, the stack cleanup in this calling convention is done by the callee.

Code example

Lets take the c code snippet from the last example with some changes.

__attribute__((stdcall)) void printnums(int num1, int num2){
	printf("The numbers you sent are: %d %d", num1, num2);
}

int main(){
	printnums(3, 2);
	return 0;
}

x86 disassembly of this function:

main:
	push ebp
	mov ebp, esp
	push 0x2
	push 0x3
	call printnums
	mov eax, 0x0
	leave
	retn

We can clearly see that the two arguments are pushed to the stack in the right to left order and no stack cleanup is done. Lets look at the callee disassembly to see the stack cleanup.

printnums:
	push ebp
	mov ebp, esp
	push dword [ebp+0xc]
	push dword [ebp+0x8]
	push 0x807897d7         ; "The numbers you sent are %d %d"
	call printf
	add esp, 0xc
	nop
	leave
	retn 0x8

As we can see that 8 bytes being cleared from the stack, as we used two integer arguments.