Saturday, February 18, 2012

Understanding the VS C++ Compiler's Buffer Security Check

In this post I will show how VS C++ compiler implements the so called stack canary protection against stack buffer overflows.

Buffer Security Check

This technique is used to detect stack buffer overflow in order to prevent execution of malicious code. It simply places arbitrary value (security cookie) between local variables and return pointer.

Most common attacks overwrite memory starting from addresses referenced by local variables with intent to overwrite return pointer (from lower to higher memory addresses - see Stack representation below). If such attack will take place, security cookie will be overwritten as well.

By simply comparing stored and original values just before exiting the function we ensure return pointer integrity.

Stack representation with security cookie in place (__cdecl convention):

(Lower memory addresses)
Local variable #2
Local variable #1
Security cookie
Base pointer
Return address pointer
Function parameter #1
Function parameter #2
(Higher memory addresses)