Applications Using Older ATL Components May Experience Conflicts With DEP

Article translations Article translations
Close Close
Article ID: 948468 - View products that this article applies to.
Expand all | Collapse all
Source: Microsoft Support

RAPID PUBLISHING

RAPID PUBLISHING ARTICLES PROVIDE INFORMATION DIRECTLY FROM WITHIN THE MICROSOFT SUPPORT ORGANIZATION. THE INFORMATION CONTAINED HEREIN IS CREATED IN RESPONSE TO EMERGING OR UNIQUE TOPICS, OR IS INTENDED SUPPLEMENT OTHER KNOWLEDGE BASE INFORMATION.

Action

An application that uses components built with ATL version 7.1 or earlier is built with the /NXCOMPAT switch, or is otherwise treated by the OS as being "No eXecute Compatible".  Installing Visual Studio 2008 can cause programs subsequently built with VS 2008 or VS 2005 to enable NX compatibility by default.  Also, if the DEP policy for the system is set to AlwaysOn, applications that incorporate the older thunking code will see this problem.

Result

The program may generate an access violation, due to Data Execution Prevention (DEP) detecting the "thunk" attempt from the older ATL code.

If attempting to add such an ATL control to a form designer in Visual Studio, you may see a misleading error like "Unable to get window handle for the AxMyATLCtrl control, windowless ActiveX controls are not supported".  The inner exception, if viewed, would be more revealing: "Attempted to read or write protected memory. This is often an indication that other memory is corrupt."  However, in this case, corruption is most likely not the problem, but rather the attempt to execute code in NX memory.

Cause

ATL 7.1 and earlier versions were not designed with foreknowledge of the DEP security feature. Newer versions of Windows (e.g., XP SP2, 2003 Server, Vista) provide additional protection against malicious attacks via the DEP feature, but they were also coded to recognize the old ATL "thunk" pattern and allow it to execute when the process is not marked as NX compatible and thunk emulation is enabled. Thunk emulation is the default, except when a process is marked NX compatible, or if the system DEP policy has disabled it.  If the /NXCOMPAT switch is used in the linking of the application, or the application process is marked by some other means as NX Compatible, then thunk emulation will be disabled, and the older ATL components may cause an access violation exception.  When the system DEP policy is set to AlwaysOn, ATL thunk emulation is disabled, regardless of the NXCOMPAT attribute for the process.

Resolution


  • If possible, replace the older components with ones built to support the "No eXecute Compatibility", such as those using ATL 8.0 or newer.  The ATL thunk strategy was devised as a lookup convenience and to avoid using thread-local storage for a window-handle-to-object map, but the thunk emulation required in DEP-aware OS's negates and even reverses any performance improvement.  Newer versions of ATL don't require the thunk emulation because their thunks are created in executable data blocks.
  • The SetProcessDEPPolicy Function, introduced in Vista SP1 and Windows Server 2008, can be a better alternative, unless system DEP policy is AlwaysOn.
  • Rebuild the application without the /NXCOMPAT switch, or use a tool such as EditBin to remove the /NXCOMPAT attribute. Note that system DEP policy can override, and having DEP AlwaysOn will disable ATL thunk emulation, regardless of the attribute. It would also not be advisable to use EditBin on the Visual Studio IDE.
  • If you must continue to use ATL 7.1, modify your ATL headers to use executable data allocations for their thunks, and rebuild the components.  Note that modifying the ATL code in this way will not be supported by Microsoft, and you do so at your own risk.  This is not recommended, but there are known cases where this appears to have been successful.
  • Disable DEP on the machine or for the process.  This is not recommended.

More Information

Background of the ATL Thunk
ATL was designed to be lightweight and efficient. This thunking code is used in CWindowImplRoot, CContainedWindowT, and any class that inherits from them.  ATL subclasses windows it creates to provide some custom behaviors. But since there is no notion of “this” pointer (the pointer to the current instance of a class object) associated with window handles (HWND), ATL needs some kind of mechanism to translate from HWND to “this” pointer.  Being very efficient, instead of having a separate look up table for HWND -> “this” pointers, ATL inserts executable code into data blocks allocated at runtime to inject a "this" pointer not known until runtime.  (See the ATL source in {your VS Install Directory}\VC\atlmfc\include\atlstdthunk.h and {your VS Install Directory}\VC\atlmfc\src\atl\atls\amd64\atlthunk.cpp).

In ATL 8.0 and newer, the allocated memory is marked executable, such as by using the PAGE_EXECUTE_READWRITE parameter when calling VirtualAlloc. This code is going to be changed in a future ATL release (so that pages aren’t both writeable and executable at the same time) and we’d like developers to use that when it becomes available.

Other Related Articles:
Data Execution Prevention (Windows)
A detailed description of the Data Execution Prevention (DEP) feature in Windows XP Service Pack 2, Windows XP Tablet PC Edition 2005, and Windows Server 2003
/NXCOMPAT (Compatible with Data Execution Prevention)
EDITBIN Reference
SetProcessDEPPolicy Function (Windows)

DISCLAIMER

MICROSOFT AND/OR ITS SUPPLIERS MAKE NO REPRESENTATIONS OR WARRANTIES ABOUT THE SUITABILITY, RELIABILITY OR ACCURACY OF THE INFORMATION CONTAINED IN THE DOCUMENTS AND RELATED GRAPHICS PUBLISHED ON THIS WEBSITE (THE “MATERIALS”) FOR ANY PURPOSE. THE MATERIALS MAY INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS AND MAY BE REVISED AT ANY TIME WITHOUT NOTICE.

TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, MICROSOFT AND/OR ITS SUPPLIERS DISCLAIM AND EXCLUDE ALL REPRESENTATIONS, WARRANTIES, AND CONDITIONS WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING BUT NOT LIMITED TO REPRESENTATIONS, WARRANTIES, OR CONDITIONS OF TITLE, NON INFRINGEMENT, SATISFACTORY CONDITION OR QUALITY, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WITH RESPECT TO THE MATERIALS.

Properties

Article ID: 948468 - Last Review: January 29, 2008 - Revision: 1.2
APPLIES TO
  • Microsoft Visual C++ 2005 Express Edition
  • Microsoft Visual C++ 2008 Express Edition
  • Microsoft Visual Studio 2005 Express Edition
  • Microsoft Visual Studio 2005 Professional Edition
  • Microsoft Visual Studio 2005 Standard Edition
  • Microsoft Visual Studio 2005 Team Suite
  • Microsoft Visual Studio 2008 Academic Edition
  • Microsoft Visual Studio 2008 Professional Edition
  • Microsoft Visual Studio 2008 Standard Edition
  • Microsoft Visual Studio Team System 2008 Team Suite
Keywords: 
kbnomt kbrapidpub KB948468

Give Feedback

 

Contact us for more help

Contact us for more help
Connect with Answer Desk for expert help.
Get more support from smallbusiness.support.microsoft.com