Recursive calls to window manager functions may fail unexpectedly

Symptoms

Recursive calls to window manager functions exported by USER32.DLL may return without performing the requested operation and without setting an error code. This typically occurs in applications with a deeply nested window hierarchy.

Among the symptoms you may experience:

• Applications with a deeply nested window hierarchy fail to resize child windows correctly when the frame window is resized. Windows are moved and/or resized by calling MoveWindow, SetWindowPos or DeferWindowPos.

• Window messages are not propagated to parent windows or to child windows as expected. DefWindowProc may not successfully propagate messages to the parent window or child windows of the window receiving the message.

• Window messages sent to a window by calling SendMessage, SendMessageTimeout or SendMessageCallback are not received by the specified window.
  

Additionally, applications that otherwise function normally may also experience the symptoms described above if WH_CALLWNDPROC or WH_CALLWNDPROCRET window hooks are set on threads in the application that own the windows. Window hooks can be set on a specific thread or on all UI threads by calling the SetWindowsHookEx function.

Cause

This behavior is a result of Windows being unable to grow the kernel stack of the calling thread in order to perform the requested operation.

Due to additional kernel stack handling procedures necessary in the x64 Windows environment, the kernel stack can be consumed at a faster rate than in an x86 Windows environment when making recursive calls to window manager functions exported by USER32.DLL. Although the symptoms described in this article are more likely to occur in x64 Windows platforms, it is possible for recursive calls to consume a thread’s kernel stack on x86 Windows platforms.

Resolution

The following solutions can be used to work around this issue

1. Resize child windows when handling WM_WINDOWPOSCHANGED window messages instead passing the message to DefWindowProc.

2. Asynchronously resize child windows when the parent window is resized instead of resizing child windows while processing the WM_WINDOWPOSCHANGED or the WM_SIZE window message.

3. Redesign the application UI to reduce the nested window depth.

More Information

Portions of the Win32 subsystem are implemented in a kernel-mode device driver (WIN32K.SYS). Calls to functions exported by USER32.DLL to change the state of a window, including its size and position, will call into WIN32K.SYS to perform the requested operation.

Functions that modify the state of a window typically result in window messages being sent to the window being modified, where WIN32K.SYS makes a user-mode callout to call the window procedure of the window being modified.

For example, WIN32K.SYS will send a window a WM_WINDOWPOSCHANGING window message and a WM_WINDOWPOSCHANGED window message when the size and/or position of the window are modified by calling the SetWindowPos function. DefWindowProc will send the specified window a WM_SIZE message when called with a WM_WINDOWPOSCHANGED message and the size of the window has changed. Applications typically resize child windows when the parent window receives a WM_WINDOWPOSCHANGED or WM_SIZE window message, which leads to making recursive calls into WIN32K.SYS for deeply nested window hierarchies.

Applications that otherwise function normally may also experience the symptoms described in this article when WH_CALLWNDPROC or WH_CALLWNDPROCRET hooks are set on threads in the process. This is due to the additional kernel stack space that is consumed when WIN32K.SYS handles calling the hook procedures.

Calling SendMessage to send a window message to a window owned by the calling thread will typically call the window procedure of the window receiving the message without having to call into WIN32K.SYS. However, SendMessage will call into WIN32K.SYS if there are WH_CALLWNDPROC hooks or WH_CALLWNDPROCRET hooks set on the calling thread, as WIN32K.SYS manages the hooks and handles calling hook procedures.

As noted above, DefWindowProc will send the specified window a WM_SIZE message when called with a WM_WINDOWPOSCHANGED message and the size of the window has changed. A WH_CALLWNDPROC hook or a WH_CALLWNDPROCRET will cause the SendMessage call DefWindowProc makes to transition into kernel mode in order to call the hook procedures. Resizing child windows when handling WM_WINDOWPOSCHANGED window messages instead of WM_SIZE window messages will reduce kernel stack usage by eliminating the need for SendMessage to transition into kernel mode in order call hook procedures.

Developers of Windows Forms applications that are encountering this issue should refer to KB article 953934for additional information.