Machine Instruction Calling Conventions

instruction categories calling conventions

Machine instructions fit, more or less, into one of three groups: Data movement copying bytes from one place to another mov a,b int x; x = 3; Control flow go someplace other than the next instruction conditional logic if,else,switch; function call/return Arithmetic/logic add,sub,mul, &, |, >> etc.

Addition and logic imul ;integer multiplication imul reg, reg imul mem, reg imul con, reg1, reg2 imul con, mem, reg ;reg2 = con * reg1 ;reg = con*mem

Calling conventions Lot s of people writing code + the need to layer software to keep complexity from being a show stopper = A convention/protocol to follow so we can share code without have to read the other person s code each time to figure out what it is expecting and how it behaves when it returns

Calling conventions Contract between caller and callee functions at the assembly level How and where arguments are passed Responsibility for managing the stack What registers are preserved, what can be overwritten Where return values go Compilers follow the rules by algorithm and implementing code compiler code generation algorithms need to work for all possible cases; You follow the rules by pattern and process your choices need only work for the current program these two could converge, but that would mean extra work, without much, or arguably negative return

Calling conventions Name ------------------ cdecl fastcall stdcall thiscall x86-64 (x64) stack cleanup responsibility -------------------- caller callee callee callee ? (but note syscall vs user space differences) parameter passing mechanism ------------------------------------ on stack in reverse order first 2 in reg, rest on stack in rev order stack in reverse order first in ECX, rest on stack in rev order first 6 in rdi,rsi,rdx,rcx,r8,r9, rest on stack who owns these definitions? x86-64: https://stackoverflow.com/questions/2535989/what-are-the-calling-conventions-for-unix-linux-system-calls-and-user-space-f

Calling conventions cdecl (see-dec-el) Very common, default x86 32bit most places 2 sets of rules, those for caller, those for callee

Calling conventions Caller rules, getting ready to call 1. Save EAX, ECX, EDX 2. Pass parameters by pushing on the stack in reverse order 3. Use call push return address on top of parameters, then jump ;called func gets to modify without restoring Caller rules, after return from call 1. Return value, if any, is found in EAX 2. Remove parameters from the stack 3. Restore EAX, ECX, EDX

Calling conventions, callee rules 1. Push EBP ;push %ebp (base pointer) 2. Copy the value of ESP into EBP ;mov %esp, %ebp (stack pointer) 3. Make space for local variables by decrementing %ESP ;stack grows down 4. Save EBX, EDI, ESI body of function 5. Put the return value in EAX 6. Restore any of EBX, EDI, ESI if modified 7. Deallocate storage ;mov %ebd, %esp (don t subtract) 8. Restore caller s base pointer ;pop %ebp 9. Return ;ret pop ret address from stack and jump

A little more on calling conventions along with compiler (gcc) intersection

posted question: How could I enforce gcc/g++ to not use registers but only stack in x86_64 to pass arguments to functions, I don't care if push/pop or mov/sub way is used. I expect there should be a flag to compiler that could enforce it but I couldn't find it. It seems you can't without hacking GCC's source code. There is no standard x86-64 calling convention that uses inefficient stack args. GCC only knows how to use standard calling conventions, in this case x86-64 SysV and MS Windows fastcall and vectorcall. (e.g. __attribute__((ms_abi)) or vectorcall). Normally nobody wants this; is that true? You can use that for some functions (controlled by __attribute__) even when compiling for Linux, MacOS, *BSD, etc., if that helps. Hard to imagine a use-case for pure stack args.

about __attribute__ #include <stdio.h> int __attribute__ ((stdcall)) f1(int a, int b) { int total; total = a*b; return(total); } ;change calling conventions from default changed from 32bit to 64bit: https://stackoverflow.com/questions/69869519/is-it-possible-to-change-calling-convention-in-gcc-for-x64 x86-32 (x86) - stdcall and cdecl exist (+fastcall) x86-64 (x64) - stdcall and cdecl do not exist if used on Windows they both refer to Microsoft x64 calling convention (ms_abi) int main(int argc, char **argv, char **envp) { int x,y,t; if used on UNIX-like, they both refer to System V AMD64 ABI (sysv_abi) x=5; y=7; t=f1(x,y); printf("t = %d\n", t); } ms_abi and sysv_abi exist (for Windows and UNIX-like respectively)

Notes: compilers sometimes just follow all the steps, regardless of local optimization opportunity to skip parts optimization flags are legion, understanding them all and how they interact can be more difficult than hand optimization (really, ameliorization) object oriented programming leads to a very high % of instructions being branches branches wreak havoc with pipelining there is often hardware support for stack operations it is a messy world, always check for reality watch for cdecl details: https://m.youtube.com/watch?v=im09tdVnYwQ