Working Set Trimming can negatively impact SQL, Exchange, and Operating System performance under Windows 2003

Symptoms

Severe performance degradation under Windows 2003 due to the Working Set Trimming issue noted in KB956341.

 

If Microsoft SQL Server in use the following error in SQL Server log:

“A significant part of sql server process memory has been paged out. This may result in a performance degradation.” 

Given Microsoft Exchange and other server applications,  a sudden drop off in Process Object/Working Set noted via Performance Monitor

Example:  Excessive Paging on Exchange 2007

Cause

Windows 2003 behavior of paging all process Working Sets to disk in order to satisfy a driver memory request, same requests detailed in KB956341.

This behavior is changed in Windows 2008, please see resolution section below.

Resolution

Recommended:  Migrate to Windows 2008 x64 or 2008 R2 where this issue is sufficiently mitigated by architectural changes in the Windows Memory Manager.  On Windows 2008, consider installing the Dynamic Cache Service to moderate Windows Cache usage as a vector for this problem (service not required on 2008R2).  Note:  Windows 2003 will not be revised with the relevant Windows 2008 and higher changes and the potential root causes under 2003 are varied and hard to detect.

 

Windows 2003 Known Problem Driver Versions and Configuration Reference:

Contiguous Memory usage by Video Adapters - use /basevideo switch and use Device Manager to change display adapter to “Standard VGA Graphics Adapter”

Note:  The above configuration change in the video adapter is particularly important due to other adapters which have boundary restrictions for memory allocations (allocates below 4GB, such as storport.sys).  Such restricted drivers may not be changed for older device compatibility reasons.  Thus, limiting the usage of contiguous memory <4GB in the video adapter is often the best solution to allow even small allocations (we have seen 8MB) increased chance of success and thus not triggering a Working Set trim.  Also, we recommend removing any extraneous Network Adapter or Storage Adapter instances.  Extra adapters of this type will each consume a share of memory in this address range, increasing probability to encounter this problem under Windows 2003.

Disable Scalable Networking Pack

Upgrade from Windows 2003 SP1 Msgina.dll (fixed in Windows 2003 SP2)

Exchange Server Update Recommendations

Hotfix for Large File Copies in Windows 2003 SP1

Update Older HP iLO Driver

 

More Information

Working Set Trimming is the action Windows Memory Manager takes to satisfy physical memory demand under lower resource conditions.  This can have severe performance consequences under Windows 2003 and the consumers of physical memory responsible are variable and often difficult to identify.  Architectural changes in Windows 2008 mitigate this problem which affects SQL and Exchange Server performance.  The Dynamic Cache Service moderates Windows Cache usage for optimum stability on x64 in both Windows 2003 and 2008.  See also KB918483.

Microsoft Internal Support Information

Troubleshoot Windows 2003 by checking the known issues list or employing Xperf etl tracing

Xperf etl Tracing to Catch Working Set Trimming

Identify the requestor of contiguous physical memory under Windows 2003 via Xperf, see KB956341 (despite what article states x86 and x64 tracing will work).

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Thanks to the WPTKit team, the Windows Performance Toolkit Alpha v.4.7 has been made available through  https://connect.microsoft.com.  Request access to this connection via the wstrim@microsoft.com alias. This version will allow a customer to employ Xperf onsite to gather the necessary data without having to run the internal Xperf version via onsite Microsoft presence.  Note that the msi will NOT install on Windows 2003, please install on a Vista or higher machine and copy over the binaries.

Feature Documentation:  http://sharepoint/sites/wpt/Documentation/Identifying%20%20Drivers%20That%20Allocate%20Contiguous%20Memory.docx

Customer Template:  \\ncperf\public\templates\xperf\xperf_to_find_driver_that_is_causing_the_working_set_trimming.oft

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Configure Dynamic Cache Service to mitigate Windows Cache consumption under Windows 2003 (x64).

Configure Lock Pages for SQL Server to avoid SQL Engine from getting paged out.  Note:  Before enabling “Lock Pages in Memory” for SQL, please review the “Important considerations…” section in KB918483.  Failure to adjust Min/Max Server Memory properly may cause other complications such as Event ID 333 and additional troubleshooting which may require KB970054.

 
 
 
Steps to reproduce.

Product Bug Number:
Author ID (email alias):tatec
Writer ID(email alias):tatec
Tech Review ID (email alias):yongrhee
Confirm Article has been Tech Reviewed: Yes/No
Confirm Article released for Publishing: Yes/No
Свойства

Номер статьи: 2001745 — последний просмотр: 23 марта 2010 г. — редакция: 1

Отзывы и предложения