High CPU usage when you synchronize a mobile device to an Exchange Server CAS

Symptoms

Consider the following scenario:

• You configure a server to run Microsoft Exchange Server 2003, Microsoft Exchange Server 2007, or Microsoft Exchange Server 2010.

• The server that is running Microsoft Exchange Server has the Client Access Server (CAS) role.

• You try to synchronize a mobile device.

In this scenario, the Exchange Server CAS server may run very slowly and not respond to a Microsoft Exchange ActiveSync request. When this issue occurs, the W3wp.exe process that is running under the MSExchangeSyncAppPool may use 100% of the CPU time.

Note On a server that is running Exchange Server 2003, the processing occurs on the back-end server. Therefore, the W3wp.exe process that runs under the ExchangeApplicationPool application pool on the back-end server may use the CPU time. Additionally, you may experience rapid transaction log growth.

Cause

This issue can occur if the mobile device tries to synchronize many contacts in a single request.

Note This synchronization request is frequently an unexpected POST of all the contacts that are on the mobile device. Exchange Server tries to validate the XML schema for each node for all the contacts. This may cause multiple memory allocations and excessive Microsoft .NET garbage collection.

Status

This issue is currently under investigation by the mobile device manufacturer.

Workaround

To work around this issue, follow these steps:

1. Identify the users and devices that are causing the high CPU usage. To do this, follow these steps:

   1. Collect the Internet Information Services (IIS) logs.

   2. Use the following Log Parser Studio query to identify the problem users and devices:

      /* Find Device/User Info for 100% CPU while syncing contacts */
      Select TOP 5000 TO_TIMESTAMP(TO_DATE(date), TO_TIME(time)) as [Time],
      cs-username as [User],
      cs(user-agent) as DeviceID,
      TO_INT(EXTRACT_PREFIX(EXTRACT_SUFFIX(cs-uri-query, 0, '_RpcC'), 0, '_')) As RPCCount,
      sc-status as Status,
      sc-substatus as SubStatus,
      sc-bytes as [Bytes],
      DIV(sc-bytes, 1024) AS [KBytes], time-taken, DIV(time-taken, 1000) as Sec, cs-uri-query
      FROM '[LOGFILEPATH]'
      WHERE 
      RPCCount > 2000 /* <-- Change RPC count as needed */
      AND cs-uri-query LIKE '%Cmd=Sync%' 
      AND cs-uri-query LIKE '%Ty:Co%'
      ORDER BY [Bytes] DESC

      Note If a request takes longer than the IIS time-out to complete, the request is dropped and is not displayed in the IIS log.
      
      Note Typically, the problem users have RPC counts over 2,000. These RPC counts frequently repeat at set intervals.

2. Inspect the returned data to identify high RPC count patterns for users and then match those users to users who have high contact item counts.

3. Test the issue again. To do this, follow these steps:

   1. Temporarily disable a user from synchronization.

   2. Recycle the MSExchangeSyncAppPool.

4. After you identify the problem users and devices, do one of the following:

   • Disable the synchronization of the Contacts folder on the device.

   • Block the problem devices at the firewall by using the User-Agent string.

   • Block users on a per-user basis.

   • Reduce the number of contacts in the Contact folder for a user.

   • Recycle the ActiveSync application pool every 30 minutes.
     
     Note We do not recommend that you recycle the ActiveSync application pool every 30 minutes in a production environment.

   • If the server is running Exchange Server 2010, change the Throttling Policy that is applied to the user. To do this, run the following cmdlet:

     Set-ThrottlingPolicy <ThrottlingPolicyName> -EASPercentTimeInCAS 30Notes

     • The Throttling Policy can be the Default Policy or a policy that is assigned to the user.

     • The default value for EASPercentTimeInCAS is 100. A lower setting may be required to work around this issue. We recommend a starting value of 30. However, the most effective setting depends on the environment. We recommend that you thoroughly test each setting. If the EASPercentTimeInCAS value is incorrect, the devices that are causing the issue may be affected.

If you cannot identify the user or device that is causing the high CPU usage, collect a set of hang dump files, and then contact Microsoft Commercial Technical Support (CTS). CTS examines the hang dump files and troubleshoots the issue. To collect a set of hang dump files, follow these steps:

1. Download and install the ProcDump utility. For more information about the ProcDump utility, visit the following TechNet website:

   http://technet.microsoft.com/en-us/sysinternals/dd996900

2. Click Start, type cmd in the Search programs and files box, and then press Enter.

3. At the command prompt, type the following command, and then press Enter:

   C:\Windows\System32\InetSrv and run appcmd list wpNote This command retrieves the Process ID (PID) of the MSExchangeSyncAppPool.

4. At the command prompt, type the following command where PID is the Process ID that was returned in step 2:

   procdump -ma [pidOfSyncAppPool] -c 90 -n 3 -s 5 -accepteula c:\temp\andy\w3wp.dmp

5. Repeat steps 1 through 4 at least three times, and then contact CTS.

More Information

When the issue that is described in the "Symptoms" section occurs, the mobile device sends the synchronization request again after several minutes if a response is not received. This can cause multiple threads that are trying to synchronize the same set of contacts. The CAS server may not finish any of the requests.

When a request is dropped because the request takes longer than the IIS time-out to complete, an error message is displayed in the HTTP API error log. This error message resembles the following:

References

For more information on this topic, see the following document from Microsoft TechNet online:
http://blogs.technet.com/b/exchange/archive/2007/09/12/3403903.aspx 

For more information about Log Parser Studio, visit the following TechNet website:

http://gallery.technet.microsoft.com/Log-Parser-Studio-cd458765For more information about throttling policies in Exchange Server 2010, visit the following TechNet website:

http://technet.microsoft.com/en-us/library/dd297964.aspxFor more information about instances where duplicate contacts are created when a user tries to synchronize many contacts, see the following article in the Microsoft Knowledge Base:

2711181  Duplicate contacts are created when you synchronize a mobile device by using Exchange ActiveSyncFor more information about the kinds of errors that are logged by the HTTP API and the format of the HTTP API error logs, see the following article in the Microsoft Knowledge Base:

820729 Error logging in HTTP API