How to monitor and troubleshoot the use of paged pool memory in Exchange Server 2003 or in Exchange 2000 Server

This article describes how to proactively manage and reduce the use of paged pool memory that is used by client connections to an Exchange server. You can reduce the use of paged pool memory by controlling the size and number of access tokens. Hotfix 912480 directly reduces the number of client access tokens that are used by a client when it makes a connection to Microsoft Exchange Server 2003 Service Pack 2 (SP2). The rest of the article describes how to reduce the size of the access token. Additionally, this article describes other methods that you can use to control, distribute, and optimize client connections in the context of access tokens.

A hotfix for Exchange 2003 SP2 is available to optimize the use of client tokens. This hotfix can reduce token memory consumption that is related to MAPI clients by up to one-third. You should apply this hotfix only if you are experiencing paged pool memory depletion issues that are caused by token allocations. For more information about hotfix 912480, click the following article number to view the article in the Microsoft Knowledge Base:

Access tokens

When a Windows account tries to access a secured Windows resource, an access token is constructed. The access token is used to determine whether access should be granted and how much access should be granted. Tokens are built by the server that hosts the resource. The server queries the appropriate domain controllers to obtain the token information.

The access token consists of several pieces of information, most notably the security identifiers (SIDs) for the user account and for the security groups to which the user account belongs. After a user authenticates to a server, the appropriate SIDs that are associated with the user and the user's group memberships are put in an access token. A SID is a string of numbers that uniquely identifies a Windows security principal or security group. For more information, view the "Security Identifiers Technical Reference" document. To view this document, visit the following Microsoft Web site:SIDs are no more secret than logon names. SIDs are unique numeric identifiers that are associated with object names. The SID stays the same for the lifetime of an Active Directory object. Therefore, the SID can be used to conclusively identify an object regardless of whether other object attributes change.

Each secured resource on a server has a discretionary access control list (DACL) associated with it. The DACL lists the SIDs that are allowed or denied access to the resource.

When a user tries to access a secured resource, the list of SIDs in the user's access token is compared to the list of SIDs in the DACL of the resource. If a SID in the token matches a SID in the DACL of the resource, appropriate access is granted. You cannot reliably determine the number of security groups to which a user account belongs by counting the number of groups that are listed on the Member Of property of the user object. This is because of the following four factors:

• Group nesting
  
  In native mode Microsoft Windows 2000 domains or Windows Server 2003 functional mode domains, groups can be more flexibly nested than in mixed-mode domains. When a group is added to a user's token, the SIDs of nested groups are also added.
• Universal group membership
  
  If the user account domain is in mixed mode, universal groups will not be added to the access token. As soon as the domain to which the account belongs is converted to native mode or one of the Windows Server 2003 functional modes, universal group memberships will be added to the token.
• SIDHistory
  
  Accounts that are migrated from Microsoft Windows NT 4.0 domains or other Active Directory domains may have many group memberships in their SIDHistory attributes. For more information about SIDHistory, visit the following Microsoft Web site:
  http://technet.microsoft.com/en-us/library/Bb727125.aspx (http://technet.microsoft.com/en-us/library/Bb727125.aspx)
  SIDHistory is available only for user account domains that are already in Windows 2000 native mode or in Windows Server 2003 functional modes. If the user account domain is in mixed mode, groups from SIDHistory will be ignored. In practice, these groups should not exist.
• Domain local groups
  
  If a secured resource is hosted in a Windows 2000 native mode or Windows Server 2003 functional mode domain, domain local groups in the resource domain to which the user account belongs will be added to the token. For example, suppose that a user in Domain A tries to access a resource in Domain B. In Windows 2000 native mode or in Windows Server 2003 functional modes, all domain local groups in Domain B to which the user belongs will be added to the access token. Domain local groups in Domain A to which the user belongs will not be added to the token that is generated by a server in Domain B. This is because domain local groups from Domain A are irrelevant to Domain B.

Token copies

A user's access token is stored on the server in paged pool kernel memory. At any time, there are likely to be multiple copies of each user's token in memory. For example, if a client maps a share on a Windows Server 2003-based server by using the NET USE command, two copies of the user's token will be held on the server to support this connection.

Each client application that connects to an Exchange server is likely to generate multiple copies of the user token, depending on the application and its configuration.

There is a finite amount of paged pool memory available. Therefore, there is a limit to the number of client connections that a server can maintain at the same time. On a Windows-based server that has more than 1 gigabyte (GB) of physical memory installed, the maximum paged pool memory is about 350 megabytes (MB). This amount may be reduced by memory tuning in favor of other resources that may be in shorter supply.

Memory tuning recommendations for a large-scale Exchange server include the use of the /3GB boot.ini switch. This reduces the maximum paged pool memory to less than 250 MB. In this context, a large-scale Exchange server is one that hosts thousands of mailboxes and that has more than 1 GB of RAM installed.

If you do not use the /3GB switch, it is likely that Exchange Server services will have to be restarted periodically to defragment virtual memory. Trading off paged pool kernel memory for additional application memory is a worthwhile tradeoff. However, this tradeoff means that you must monitor the use of paged pool memory more closely. For more information about memory tuning for Exchange Server, click the following article number to view the article in the Microsoft Knowledge Base: Additionally, view the "Ruling Out Memory-Bound Problems" section of the "Troubleshooting Exchange Server 2003 Performance" white paper. To view this white paper, visit the following Microsoft Web site: Client tokens are usually the biggest single consumer of paged pool memory on an Exchange server. If the average user token is large, paged pool memory consumption is likely to become an important bottleneck for Exchange Server scalability.

How to calculate token size

Access token size in bytes can be estimated by using the following formula:

• User rights include rights such as "Log on locally" or "Access this computer from the network." The only user rights that are added to an access token are those user rights that are configured on the server that hosts a secured resource. Most Exchange Server users are likely to have only two or three user rights on the Exchange server. Administrators may have dozens of user rights. Each user right requires 12 bytes to store it in the token.
• Token overhead includes multiple fields such as the token source, expiration time, and impersonation information. For a typical domain user who has no special access or restrictions, token overhead is likely to be between 400 and 500 bytes. Typically, estimating 500 bytes for both user rights and token overhead is more than sufficient.
• Each group membership adds the group SID to the token together with an additional 16 bytes for associated attributes and information. The maximum possible size for a SID is 68 bytes. However, it is rare for a SID to be this large. In Windows Server 2003 and in earlier versions of Windows, the typical SID for a user or a group is 28 bytes in length. Therefore, each security group to which a user belongs typically adds 44 bytes to the user's token size.

Token memory allocation

If a token is less than 4 kilobytes (KB), the amount of kernel memory that is allocated for it is exactly what is required to hold the token. For example, consider a typical user who belongs to 30 security groups. By using the formula that is mentioned in the "How to calculate token size" section, this user's token will be about 1,820 bytes (44 bytes x 30 groups + 500 overhead bytes = 1,820).

But if a token is even slightly larger than 4 KB (4,096 bytes), the amount of memory that is allocated per copy will jump to exactly 8 KB (8,192 bytes). If a token is even slightly larger than 8 KB, the memory allocation will jump to exactly 12 KB. Therefore, every time that the token sizes crosses one of these critical 4-KB boundaries, there is a sudden jump in the use of paged pool memory.

Generally, a user who belongs to more than 80 security groups will be near or will go over the 4-KB boundary. Therefore, the user will require an 8-KB token. If a user belongs to more than 170 groups, the token is likely to require 12 KB, and so on.

The following example illustrates how important it is to monitor and control average client token size. Consider an Exchange 2003 Service Pack 2 server on which all clients use Microsoft Office Outlook 2003 in cached mode. A typical cached-mode client causes seven or eight copies of its token to be generated on a Windows Server 2003-based computer. If the average client token is exactly 4 KB, each cached-mode client requires up to 32 KB of paged pool memory.

Note The Microsoft Exchange Information Store service hotfix that is described in the "Introduction" section can reduce the number of token copies for each cached-mode user to four or five instead of seven or eight. This hotfix is scheduled to be included in Microsoft Exchange Server 2003 Service Pack 3.

If the server is configured by using the /3GB switch, there will be about 250 MB of paged pool memory allocated on the server. We recommend that the typical paged pool consumption for the server should be no more than 200 MB. You must reserve sufficient memory for spikes in the server load. If paged pool memory consumption is typically more than 220 MB, you should take immediate steps to reduce the load on the server.

Assume that 150 MB of paged pool memory is available for Exchange Server client tokens. If each client token is 4 KB, the server can comfortably support more than 4,500 concurrent Outlook cached-mode users before token use will become a bottleneck. Note that applying hotfix 912480 would increase this maximum to 7,300 cached-mode users. If token size were to jump to 8 KB, the maximum number of clients would be reduced by half, regardless of whether hotfix 912480 was applied.

Note If Outlook 2003 is run in online mode, there will typically be three or four token copies for each client regardless of whether hotfix 912480 was applied.

Symptoms of kernel memory depletion

If kernel memory resources are close to being exhausted, the server becomes slow or refuses additional requests and connections. Applications may fail suddenly. Additionally, attempts to connect to the affected server may return error 1450, "Insufficient System Resources." In extreme cases, the server may display an error message on a blue screen and stop responding.

Additionally, the following events may be logged in the System log:

Event ID: 2019
Source: SRV
Description: The server was unable to allocate from the system nonpaged pool because the pool was empty.

Event ID: 2020
Source: SRV
Description: The server was unable to allocate from the system paged pool because the pool was empty.

Event ID: 2000
Source: SRV
Description: The server's call to a system service failed unexpectedly.

If a paged pool memory shortage is transient, the server will likely recover. Applications can be somewhat resilient to temporary shortages of memory. However, no application can run forever if critical resource requests are not satisfied. If the paged pool memory shortage lasts very long, it is likely to trigger cascading bottlenecks. In such a case, the server will probably have to be restarted to make it functional again.

Under standard load, there should be approximately 50 MB of available paged pool memory. If you have less than 30 megabytes free, you should take immediate steps to reduce the load on the server.

Paged pool memory is allocated statically during Windows startup. The pool cannot be increased without reconfiguring and restarting the server. The amount of available paged pool memory depends on several factors. These factors include boot switches such as /USERVA and /3GB, registry settings, and physical RAM.

How to reduce the size of user access tokens

You can use the following three strategies to reduce token size:

• Reduce the number of security groups to which each user belongs.
• Host Exchange servers in a different domain from the users who connect to the Exchange servers.
  
  This strategy can reduce the size of user tokens by stripping domain local groups for the user account domain from the token that is presented to the Exchange server. This works because domain local groups from one domain are not kept in the token that is generated on a server in a different domain.
• When you can, convert security groups to distribution groups.
  
  Token size is increased by membership in security groups, not distribution groups. Users can belong to thousands of distribution groups that do not affect token size. If a group is not being used to deny or grant access to resources, it should be a distribution group, not a security group.

How to reduce the number of access tokens in memory on the server

As soon as you have reduced the typical token size to the practical minimum, the next step is to manage the number of simultaneous connections that are made to the server. You can manage the number of simultaneous connections by using the following methods:

• Restrict unauthorized clients and applications.
  
  Each client may make multiple connections to the server. Additionally, different clients make different numbers of connections based on a wide variety of factors. You may not even have a full list of all the clients that connect to the server. Users may install Outlook add-ins that make additional connections. Developers may run applications that make many connections or that do not shut down connections when they are finished. Therefore, you should analyze what kinds of clients connect to the server and what kinds of effects they have on kernel memory usage. For more information, see the "How to view token allocation sizes" section.
• Remove the public folder store from the server. Then, direct clients to public folders on a different server.
  
  This action eliminates public folder connections that are made by clients.
• Remove specific public folders that account for many client connections.
  
  Good candidates for removal are the Schedule+ Free/Busy folder and the offline address book. Clients must make additional connections to these folders when they schedule appointments or download the address book.
• Add replicas of heavily accessed public folders to distribute the number of clients who connect to them across multiple servers.
• Install dedicated public folder servers to eliminate all public folder connections from mailbox servers.
• Distribute heavy-connection users evenly across multiple servers. Heavy-connection users are likely to be those who have multiple computers or devices and those who are mobile users.
• Distribute users who have large security tokens across multiple servers.
• Apply hotfix 912480 for token optimization. For more information about hotfix 912480, click the following article number to view the article in the Microsoft Knowledge Base:
  912480  (http://support.microsoft.com/kb/912480/ ) An Exchange Server 2003 server that hosts many Outlook client sessions may run out of paged pool memory

How to monitor paged pool memory on an Exchange server

Generally, you should have 50 MB of free paged pool memory available under typical server load conditions. Additionally, you should have 30 MB free under peak loads.

It is easy to determine how much paged pool memory is currently being used. Windows Task Manager displays paged pool usage in the Kernel Memory area on the Performance tab. You can also monitor the use of paged pool memory over time with the Memory\Pool Paged Bytes counter in Windows System Monitor.

An Exchange server that is configured to use the /3GB boot switch will have a maximum possible paged pool memory size of about 250 MB. Additionally, this server will have a non-paged-pool-memory maximum of 128 MB. Without the /3GB switch, the maximums are 350 MB for paged pool memory and 256 MB for non-paged pool memory.

Therefore, a typical large-scale Exchange server should use no more than 200 MB of paged pool memory under typical conditions. Paged pool memory use of more than 220 MB requires immediate attention.

If you are within these limits, and the server is reporting errors that are related to paged pool memory depletion, it is likely that the initial paged pool memory allocation is less than expected. This can be caused by hardware demands, by device drivers, or by memory tuning that reduces initial paged pool memory allocation even more. Large memory configurations, for example, more than 4 GB of physical RAM, are the most common cause of this problem.

Each byte of physical RAM that is installed in a server requires some kernel memory to address and manage it. The more RAM that is installed, the more kernel address space must be reserved for it. Address space may be borrowed from paged pool memory to satisfy this demand.

We recommend that you do not install more than 4 GB of physical RAM in a server that is dedicated to running Exchange Server 2003. Exchange Server will make efficient use of up to 4 GB of RAM. However, Exchange Server will not take advantage of additional RAM even if it is available. Servers that support the Hot Add Memory feature can also cause significant reductions in the availability of paged pool memory. Even if no more than 4 GB of RAM is installed, address space may be reserved for the theoretical maximum amount of hot-add RAM that could be installed.

You can use a kernel debugger to view the size of initial paged pool memory and other kernel memory allocations.

Important Commands that can be used during a kernel debugging session can cause the system to become unstable or stop. We recommend that you stop all Exchange Server services before you initiate a kernel debugging session and that you restart the server after the session.

Setting up a traditional kernel debugging session for Windows 2000 can be a complex task. This task typically requires an extra computer, specialized cabling, and a server restart.

Alternatively, the LiveKD utility from Sysinternals can be used to start a kernel debugging session from the server console. LiveKD does not require that you restart the server. For more information, click the following article number to view the article in the Microsoft Knowledge Base: For Windows Server 2003, the KD kernel debugger supports debugging directly from the server console without special preparation or hardware. To obtain the Debugging Tools for Windows, visit the following Microsoft Web site:Start the debugger by using the KD.EXE -KL command. Then, run the !vm command to view the maximum paged pool memory. For example, run the following commands:

How to view token allocation sizes

Outlook is not the only client that can connect to an Exchange Server database. Outlook add-ins, desktop search engines that include mail search functionality, instant messaging clients, and custom applications can all make additional connections and cause the generation of additional token copies.

You can verify the effect of a client or application by using the Poolmon.exe utility in a laboratory environment. To do this, follow these steps:

1. Generate an isolated laboratory Exchange organization.
2. Install Poolmon on the Exchange server. For more information about how to configure Poolmon.exe, click the following article number to view the article in the Microsoft Knowledge Base:
   177415  (http://support.microsoft.com/kb/177415/ ) How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel memory leaks
3. Run Poolmon.exe with the /iToke switch (Poolmon /iToke). Note that the /iToke parameter is case sensitive. This will configure Poolmon.exe to display only token allocations. You can also use this command on a production server to view total token allocations in real time.
4. Configure an Active Directory user account that is similar to the typical Exchange Server user in your environment. That is, configure a user account that has an equivalent number of security group memberships, a similar permissions profile, and so on.
5. Log on to Exchange Server as the test user who has the client applications and configurations that you want to test. Wait several minutes after you log on for the client application to completely load and stabilize.
6. Exit the client application when you notice the change in the token bytes in Poolmon.exe. You may have to do this several times to obtain an accurate reading of how many bytes are freed when you exit the client. Other token allocations may be created or destroyed at the same time during the test.

Note If you change the user account, such as by adding or deleting security group memberships, you must log off the account from Windows and then log back on before these changes will be reflected in the access token.

How to audit group memberships

The following script examples contain command-line parameters and instructions at the top of each script. You can paste the scripts into Notepad and then save them as .vbs files. Do not save the files as .txt files.

• The Groups.vbs script prints Exchange Server mailbox user account names and the security groups to which they belong. Additionally, the printout contains a separate column that lists groups from SIDHistory. You can restrict the script to a single Exchange server or obtain a report for multiple Exchange servers by using a wildcard character.
  
  Note You cannot use a wildcard character (*) to access all Exchange servers. You must provide at least a partial server name. For example, you can use a string that is similar the following:
  EXCH-HQ-*
• The Groups_statistics.vbs script provides a text-based histogram view that shows you how many users belong to 50 groups, 60 groups, 70 groups, and so on. This can help you determine the likely average token size for users.

See the "How to calculate token size" and "Token memory allocation" sections for detailed information about token sizes.

Scripts

Groups.vbs

Groups_statistics.vbs

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, regarding the performance or reliability of these products.