Stack Memory Allocation in GCC

In the previous installment, we touched upon the System V AMD64 ABI convention for passing parameters. Now, we dive into how GCC manages stack memory and handles function return values. The stack is a fundamental mechanism for x86 processors, equipped with specific instructions like push and pop for saving and restoring data. It relies on the rsp register, the stack pointer, to keep track of its contents. Compilers like GCC leverage this for storing local variables within functions.

Consider a simple C program designed to illustrate these concepts:

#include <stdio.h>

// 1. Return via register (Standard scalar type)
int return_integer() {
    int a = 10;
    return a;
}

// 2. Return via stack (Larger type)
struct Point {
    int x;
    int y;
};

struct Point return_struct() {
    struct Point p = {1, 2};
    return p;
}

int main() {
    int result_int = return_integer();
    printf("Integer return: %d\n", result_int);

    struct Point result_struct = return_struct();
    printf("Struct return: (%d, %d)\n", result_struct.x, result_struct.y);

    return 0;
}

Register-Based Return Values

For standard scalar types, such as int, float, or pointers, GCC follows the System V AMD64 ABI by returning the value in a specific register. Specifically, the rax register is used for integer and pointer return values. For floating-point values, the xmm0 register is employed. This approach is efficient as it avoids the overhead of memory operations.

When the return_integer function is compiled, GCC will generate assembly code that loads the value of the local variable a into the rax register before the function returns. The caller function then reads the result directly from rax.

Let's examine a simplified view of the assembly generated for return_integer:


return_integer:
    push    rbp         ; Standard function prologue
    mov     rbp, rsp
    sub     rsp, 16     ; Allocate space for local variable 'a'

    mov     DWORD PTR [rbp-4], 10 ; Store 10 into 'a'

    mov     eax, DWORD PTR [rbp-4] ; Move the value of 'a' into EAX (lower 32 bits of RAX)

    leave               ; Standard function epilogue (restores RSP and RBP)
    ret                 ; Return from function, EAX holds the return value

The key instruction here is mov eax, DWORD PTR [rbp-4], which copies the value of a (stored on the stack relative to rbp) into the eax register. Upon returning, the calling function will find the integer result in rax.

Stack-Based Return Values for Structures

The situation changes significantly when functions need to return larger data types, such as structures or arrays. The System V AMD64 ABI dictates that for types larger than 64 bits, the caller must allocate space on its stack for the return value. The address of this allocated space is then passed to the called function via the first argument register, rdi.

In our example, the return_struct function returns a struct Point, which consists of two integers (8 bytes total). This size exceeds the typical register-based return. Therefore, the caller (main in this case) will set up a buffer on its stack, and pass the address of this buffer to return_struct.

The return_struct function then copies the structure's data into this caller-provided buffer. The function's return value is effectively the pointer to this buffer, which is still passed back through the rax register (as a pointer).

A simplified view of the assembly for return_struct might look like this:


return_struct:
    push    rbp         ; Standard function prologue
    mov     rbp, rsp
    ; No stack space allocated for local struct 'p' here as it's small

    mov     DWORD PTR [rbp-8], 1 ; Store x = 1
    mov     DWORD PTR [rbp-12], 2 ; Store y = 2

    ; Assume RDI holds the address of the caller's buffer for the return struct
    mov     eax, edi        ; Copy the address from RDI to RAX (where the pointer is returned)
    mov     ecx, DWORD PTR [rbp-8] ; Load x
    mov     [rax], ecx      ; Store x into the caller's buffer at offset 0
    mov     ecx, DWORD PTR [rbp-12] ; Load y
    mov     [rax+4], ecx    ; Store y into the caller's buffer at offset 4

    leave               ; Standard function epilogue
    ret                 ; Return from function, RAX holds the pointer to the buffer

Here, rax is loaded with the address passed in rdi. The local struct members are then copied into the memory pointed to by rax. The caller, main, will then access the returned structure data from its own stack frame.

GCC's Role in Stack Management

GCC's compiler backend is responsible for translating the high-level C code into machine instructions that adhere to the target architecture's ABI. This includes deciding whether a return value fits within a register or requires stack space managed by the caller. The compiler analyzes the size and type of the return value to determine the most efficient mechanism.

The stack pointer (rsp) is crucial. When a function is called, the call instruction pushes the return address onto the stack. If the function needs local variables, it typically subtracts a value from rsp to allocate space. When the function returns, it uses the ret instruction, which pops the return address from the stack and jumps to it. If the function allocated stack space for local variables, it must restore rsp to its original position before returning, often using the leave instruction which is equivalent to mov rsp, rbp followed by pop rbp.

This intricate dance of register usage and stack manipulation ensures that functions can communicate data effectively, whether it's a simple integer or a complex data structure, all while maintaining program state.