Alex Lipov @ osom.info: Understanding the VS C++ Compiler's Buffer Security Check

In this post I will show how VS C++ compiler implements the so called stack canary protection against stack buffer overflows.

Buffer Security Check

This technique is used to detect stack buffer overflow in order to prevent execution of malicious code. It simply places arbitrary value (security cookie) between local variables and return pointer.

Most common attacks overwrite memory starting from addresses referenced by local variables with intent to overwrite return pointer (from lower to higher memory addresses - see Stack representation below). If such attack will take place, security cookie will be overwritten as well.

By simply comparing stored and original values just before exiting the function we ensure return pointer integrity.

Stack representation with security cookie in place (__cdecl convention):

(Lower memory addresses)

Local variable #2

Local variable #1

Security cookie

Base pointer

Return address pointer

Function parameter #1

Function parameter #2

(Higher memory addresses)

VS C++ compiler implements this technique for each function with a code that can potentially lead to stack buffer overflow. On function entry, the security cookie is stored just after the base pointer. On function exit, the stored value is validated to make sure it was not overridden.

The relevant switch is called /GS (Buffer Security Check), and it is available since Visual Studio 2002. For enabling/disabling it use /GS- or /GS+, or edit the project properties (enabled by default in Release configuration):

Code Example

Let’s actually see how VS C++ compiler generates the security cookie using the following C program:

#include "string.h"
 
void foo (char *bar)
{
   char  c[12];
   memcpy(c, bar, strlen(bar));  // no bounds checking...
}
 
int main (int argc, char **argv)
{
   foo("my string is too long !!!!! \x10\x10\xC0\x42");
   return 0;
}

Since this program is doing nothing (except of overflowing the stack), the compiler by default will not generate any code. So we’ll have to disable compiler’s optimization and function inlining in project properties:

I will use IDA Pro to disassemble the program. The following code generated for foo() function with /GS switch set to /GS- (disabled):

.text:00401000                 push    ebp
.text:00401001                 mov     ebp, esp
.text:00401003                 sub     esp, 1Ch
.text:00401006                 mov     eax, [ebp+arg_0]
.text:00401009                 mov     [ebp+var_10], eax
.text:0040100C                 mov     ecx, [ebp+var_10]
.text:0040100F                 add     ecx, 1
.text:00401012                 mov     [ebp+var_14], ecx
loc_401015:  
.text:00401015                 mov     edx, [ebp+var_10]
.text:00401018                 mov     al, [edx]
.text:0040101A                 mov     [ebp+var_15], al
.text:0040101D                 add     [ebp+var_10], 1
.text:00401021                 cmp     [ebp+var_15], 0
.text:00401025                 jnz     short loc_401015
.text:00401027                 mov     ecx, [ebp+var_10]
.text:0040102A                 sub     ecx, [ebp+var_14]
.text:0040102D                 mov     [ebp+var_1C], ecx
.text:00401030                 mov     edx, [ebp+var_1C]
.text:00401033                 push    edx             ; size_t
.text:00401034                 mov     eax, [ebp+arg_0]
.text:00401037                 push    eax             ; void *
.text:00401038                 lea     ecx, [ebp+var_C]
.text:0040103B                 push    ecx             ; void *
.text:0040103C                 call    _memcpy
.text:00401041                 add     esp, 0Ch
.text:00401044                 mov     esp, ebp
.text:00401046                 pop     ebp
.text:00401047                 retn

After enabling the Buffer Security Check (/GS+) and recompiling, the foo() function grew by 7 instructions (see marked in red). Those instructions were generated by compiler, since it detected the call to unsafe memcpy API within the function. The memcpy function included in SDL list of banned function calls, and this is definitely one of the resources used by compiler to determine whether to generate security cookies. The 4-bytes long security cookie (which is generated on module load) xored with base pointer and stored in the appropriate location:

.text:00401000                 push    ebp
.text:00401001                 mov     ebp, esp
.text:00401003                 sub     esp, 20h
.text:00401006                 mov     eax, __security_cookie
.text:0040100B                 xor     eax, ebp
.text:0040100D                 mov     [ebp+var_4], eax
.text:00401010                 mov     eax, [ebp+arg_0]
.text:00401013                 mov     [ebp+var_14], eax
.text:00401016                 mov     ecx, [ebp+var_14]
.text:00401019                 add     ecx, 1
.text:0040101C                 mov     [ebp+var_18], ecx
loc_40101F:
.text:0040101F                 mov     edx, [ebp+var_14]
.text:00401022                 mov     al, [edx]
.text:00401024                 mov     [ebp+var_19], al
.text:00401027                 add     [ebp+var_14], 1
.text:0040102B                 cmp     [ebp+var_19], 0
.text:0040102F                 jnz     short loc_40101F
.text:00401031                 mov     ecx, [ebp+var_14]
.text:00401034                 sub     ecx, [ebp+var_18]
.text:00401037                 mov     [ebp+var_20], ecx
.text:0040103A                 mov     edx, [ebp+var_20]
.text:0040103D                 push    edx             ; size_t
.text:0040103E                 mov     eax, [ebp+arg_0]
.text:00401041                 push    eax             ; void *
.text:00401042                 lea     ecx, [ebp+var_10]
.text:00401045                 push    ecx             ; void *
.text:00401046                 call    _memcpy
.text:0040104B                 add     esp, 0Ch
.text:0040104E                 mov     ecx, [ebp+var_4]
.text:00401051                 xor     ecx, ebp
.text:00401053                 call    __security_check_cookie
.text:00401058                 mov     esp, ebp
.text:0040105A                 pop     ebp
.text:0040105B                 retn

And just before the exit, stored security cookie value xored again with base pointer (converting it back to original form) and then compared against the known value. If the values are not equal, __report_gsfailure function called, which will eventually terminate the process.

__security_check_cookie function:

.text:00851074 cmp     ecx, __security_cookie
.text:0085107A jnz     __report_gsfailure
.text:0085107C rep retn

Running the program

Before the memcpy call, the stack looks as shown below. The return address is pointing on correct address. The value of security cookie is 0x39E8565E.