How to trap stack overflow in a Visual C++ application

Article translations Article translations
Article ID: 315937 - View products that this article applies to.
This article was previously published under Q315937
Expand all | Collapse all

On This Page

SUMMARY

A thread that exceeds its stack allocation will raise an exception. This exception can be trapped with the __try and __except keywords in Microsoft Visual C++. However, without correct handling, subsequent stack overflows will raise access violation exceptions. This article shows you how to handle stack overflows more robustly.

Requirements

The following items describe the recommended hardware, software, network infrastructure, skills, and knowledge and service packs that you need.
  • Microsoft Windows NT, Microsoft Windows 2000, or Microsoft Windows XP
Prior knowledge required:
  • You must have a good knowledge of Visual C++ programming, including exception handling.

Creating a Test Program to Demonstrate Stack Overflow

  1. Start Microsoft Visual C++ 6.0, and then create a new Win32 Console Application named StackOverflowDemo. In the Win32 Console Application wizard, select A simple application, and then click Finish. Click OK to create the new application.
  2. In the Workspace window, click the FileView tab. Expand the StackOverflowDemo files node, and then expand the Source Files node. Double-click StackOverflowDemo.cpp to edit this file in the Code window.
  3. In StackOverflowDemo.cpp, add the following #include directives after the existing #include "stdafx.h" directive:
    #include <windows.h>
    #include <stdio.h>
    						
  4. After the #include directives, type the following function. This deliberately causes a stack overflow:
    void StackOverflow(int depth)
    {
        char blockdata[10000];
        printf("Overflow: %d\n", depth);
        StackOverflow(depth+1);
    }
    						
  5. Edit the main function as follows: int main(int argc, char * argv[])
    {
        StackOverflow(0);
        return 0;
    }
    						
  6. Press F5 to run the program in the debugger. The program displays a series of messages to the console. These messages have the form "Overflow: n". Each message indicates a recursive call to the StackOverflow function. Eventually, a message box appears that displays the following error message:
    Unhandled exception in StackOverflowDemo.exe: 0xC00000FD: Stack Overflow.
  7. Press OK to dismiss the dialog box. Depending on your configuration, the IDE may try to find the source for CHKSTK.ASM. If this dialog box appears, press Cancel. Click the Debug menu, and then click Stop Debugging to close the program.

Trapping the Exception with __try and __except (Partial Solution)

In this section, you trap the stack overflow exception using __try and __except. This solution is incomplete. The first stack overflow will raise a stack overflow exception. However, subsequent stack overflows cause access violation exceptions.
  1. Modify the main function as follows: int main(int argc, char * argv[])
    {
        for (;;)
        {    
            __try
            {
                StackOverflow(0);
            }
            __except (EXCEPTION_EXECUTE_HANDLER)
            {
                printf("Exception handler %lX\n", _exception_code()); 
                Sleep(2000);
            }
        }
        return 0;
    }
    						
  2. Press F5 to run the program in the debugger.
  3. The console output proceeds as before, until the stack overflow occurs. The exception handler message is then printed out. The loop makes sure that the StackOverflow function is retried.
  4. The program displays the following output:
    Overflow: 0
    Overflow: 1
    Overflow: 2
    .
    .
    Exception Handler C00000FD
    Overflow: 0
    Overflow: 1
    Overflow: 2
    .
    .
    Exception Handler C0000005
    Overflow: 0
    Overflow: 1
    Overflow: 2
    .
    .
    Exception Handler C0000005
    						
    The first exception is a stack overflow (C00000FD), but subsequent calls to the StackOverflow function cause an access violation exception (C0000005).
  5. Click the Debug menu in Visual C++, and then click Stop Debugging to close the program.

Trapping the Exception with __try and __except (Full Solution)

If a thread in your application causes an EXCEPTION_STACK_OVERFLOW exception, then your thread has left its stack in a damaged state. This is in contrast to other exceptions such as EXCEPTION_ACCESS_VIOLATION or EXCEPTION_INT_DIVIDE_BY_ZERO, where the stack is not damaged. This is because the stack is set to an arbitrarily small value when the program is first loaded. The stack then grows on demand to meet the needs of the thread. This is implemented by placing a page with PAGE_GUARD access at the end of the current stack. When your code causes the stack pointer to point to an address on this page, an exception occurs. The system then does the three following things:
  1. Remove the PAGE_GUARD protection on the guard page, so that the thread can read and write data to the memory.
  2. Allocate a new guard page that is located one page below the last one.
  3. Rerun the instruction that raised the exception.
In this way, the system can grow the stack for your thread automatically. Each thread in a process has a maximum stack size. The stack size is set at compile time by the /STACK:reserve[,commit] linker switch, or by the STACKSIZE statement in the .def file for the project. When this maximum stack size is exceeded, the system does the three following things:
  • Remove the PAGE_GUARD protection on the guard page, as described earlier.
  • Try to allocate a new guard page below the last one. However, this fails because the maximum stack size has been exceeded.
  • Raise an exception, so that the thread can handle it in the exception block.
Notice an important point here: Your stack no longer has a guard page. The next time that your program grows the stack all the way to the end (where there should be a guard page), your program writes beyond the end of the stack and causes an access violation.

You can repair the guard page in the exception handler by using the example shown in this section.

Modify the main function as follows:
int main(int argc, char* argv[])
{
    for (;;)
    {
        __try
        {
            StackOverflow(0);
        }
        __except(EXCEPTION_EXECUTE_HANDLER)
        {
            LPBYTE lpPage;
            static SYSTEM_INFO si;
            static MEMORY_BASIC_INFORMATION mi;
            static DWORD dwOldProtect;

            // Get page size of system
            GetSystemInfo(&si);
            
            // Find SP address
            _asm mov lpPage, esp;

            // Get allocation base of stack
            VirtualQuery(lpPage, &mi, sizeof(mi));

            // Go to page beyond current page
            lpPage = (LPBYTE)(mi.BaseAddress)-si.dwPageSize;

            // Free portion of stack just abandoned
            if (!VirtualFree(mi.AllocationBase,
                            (LPBYTE)lpPage - (LPBYTE)mi.AllocationBase,
                             MEM_DECOMMIT))
				{
                // If we get here, exit
                exit(1);
            }

				// Reintroduce the guard page
				if (!VirtualProtect(lpPage, si.dwPageSize, 
                                PAGE_GUARD | PAGE_READWRITE, 
                                &dwOldProtect))
				{
                exit(1);
            }
				printf("Exception handler %lX\n", _exception_code()); 
            Sleep(2000);
        }
	}
	return 0;
}
				

Verifying That It Works

  1. Press F5 to run the program in the debugger.
  2. Observe the output from your program. Make sure that all of the exceptions are stack overflow exceptions (C00000FD). This output shows that you have successfully replaced the guard page as part of the exception-handling mechanism.
  3. Select the Debug menu in Visual C++, and then click Stop Debugging to close the program.

Troubleshooting

This solution uses in-line assembly statements. This means that the solution is valid only for Intel (x86) versions of Microsoft Windows. It is not suitable for implementations of Windows on non-Intel architectures (MIPS or Alpha).

Properties

Article ID: 315937 - Last Review: June 6, 2005 - Revision: 3.0
APPLIES TO
  • Microsoft Visual C++ 6.0 Standard Edition
  • Microsoft Visual C++ 6.0 Service Pack 5
  • Microsoft Visual C++ 6.0 Professional Edition
  • Microsoft Visual C++ 6.0 Enterprise Edition
Keywords: 
kbhowtomaster KB315937

Give Feedback

 

Contact us for more help

Contact us for more help
Connect with Answer Desk for expert help.
Get more support from smallbusiness.support.microsoft.com