EAS proxy failures result in failure to synchronize mobile devices

Symptoms

Consider this scenario.  Users connect to internal-facing Client Access Servers (CAS) using CAS-to-CAS Proxy from Internet-facing CAS servers.  When mobile devices connect using Exchange ActiveSync (EAS) and create several thousand connections in a twenty-four hour period, each, devices are unable to synchronize. 

In this scenario, you may see the following in the Application Event Log: 

Log Name:      Application
Source:        MSExchange ActiveSync
Date:          mm/dd/yyyy hh:mm:ss PM
Event ID:      1034
Task Category: Configuration
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      CAS.contoso.com
Description:   The proxy request to webserver.contoso.com has timed out.

You may also see many entries like the one below in the IIS logs on the CAS server:

YYYY-MM-DD HH:MM:SS 192.168.0.1 POST /Microsoft-Server-ActiveSync/default.eas User=USER&DeviceId=DEVICEINFO&DeviceType=TYPE&Cmd=Ping&Log=PrxTo:mail.contoso.com_LdapC3_Error:AirSyncPermanentException%3a+_Mbx:mbx.contoso.com_Dc:DCNAME/DC.contoso.com_Budget:(D)Conn%3a1%2cHangingConn%3a0%2cAD%3a%24null%2f%24null%2f0%25%2cCAS%3a%24null%2f%24null%2f0%25%2cAB%3a%24null%2f%24null%2f0%25%2cRPC%3a%24null%2f%24null%2f0%25%2cFC%3a1000%2f0%2cPolicy%3aDefaultThrottlingPolicy%5POLICYNAME%2cNorm%5bResources%3a(DC)DCNAME.contoso.com(Health%3a-1%25%2cHistLoad%3a0)%2c%5d_ 443 DOMAIN\USERNAME 192.168.0.11 DEVICE/USERAGENT 500 0 0 631180

Cause

There are several possible issues that can cause the performance degradation. 

• The devices that make many thousands of connections are impacting system resources on the CAS.

• Extremely lengthy KeepAliveTime settings, or possible timeout mismatches between the CAS servers and hardware such as load balancers or accelerators or firewalls, may exist.

• The MSExchangeSyncAppPool is overwhelmed and unable to effictively process requests.

Resolution

This issue is resolved in Exchange Server 2010 Service Pack 1 Rollup Update 5.  See the following article from the Knowledge Base online:

2556133 A device that uses Exchange ActiveSync cannot access mailboxes in an Exchange Server 2010 environment

This update may resolve the issue completely.  If necessary, there are several additional mitigation steps that may impact this issue.

Address Problem Devices
Work with the device users to update their devices, or block problem devices.  See the following article from the Microsoft Knowlege Base online for more information:

2469722 Unable to connect using Exchange ActiveSync due to Exchange resource consumption

Synchronize KeepAliveTime settings

Synchronize KeepAliveTime settings for the CAS and hardware that Exchange ActiveSync traffic must pass through.  See the following article from the Microsoft Knowledge Base online for more information on setting the KeepAliveTime for Windows servers:

324270 How to harden the TCP/IP stack against denial of service attacks in Windows Server 2003

NOTE The recommendation of five minutes for the KeepAliveTime setting may not be necessary for all organizations.  Longer settings can be used, but should be synchronized across the CAS and hardware devices.  For Exchange ActiveSync purposes, you should consider your Heartbeat Interval in setting the KeepAliveTime.  See the following topics from Microsoft TechNet online for additional information:

Understanding Direct Push
http://technet.microsoft.com/en-us/library/aa997252.aspx

Exchange Best Practices Analyzer: Exchange ActiveSync has detected that the minimum heartbeat interval is set to a value that is higher than the maximum heartbeat interval.
http://technet.microsoft.com/en-us/library/ff360297(EXCHG.140).aspx

Modify the ApplicationHost.Config file
Using the following steps, modify how the MSExchangeSyncAppPool handles thread requests.  This requires that you install the hotfix below on the CAS servers that receive traffic from the Internet or that service proxy requests internally.

2290617 FIX: A hotfix is available to enable the configuration of some ASP.NET properties for each application pool in IIS 7

Note:  This fix is not required for systems running Windows Server 2008 R2.

After the fix is installed, use the following steps:

1. Create a config file on the CAS that will be used to configure the MSExchangeSyncAppPool.  In the example below, the file is named EAS_pool.config.  Place this file in any folder on the CAS.  In the example below, the file is placed in C:\.
   

2. The text of the EAS_pool.config file will be as follows:
   
   <configuration>
     <system.web>
       <applicationPool
           maxConcurrentRequestsPerCPU="5000"
           maxConcurrentThreadsPerCPU="0"
           requestQueueLimit="5000" />
    </system.web>
   </configuration>
   
   

3. In the Applicationhost.config file, add the CLRConfigFile setting in the ApplicationPools element to specify the path of the configuration file for an Integrated mode application pool.  For example, change the existing text of the MSExchangeSyncAppPool in the file to:
   
   <applicationPools>
   <add name="MSExchangeSyncAppPool" queueLength="10000" CLRConfigFile= "C:\EAS_pool.config "/>
   
   

4. Restart the IIS Admin Service or restart the computer.

More Information

For additional information on ASP.NET thread usage in IIS 7.0 and 6.0, see the following MSDN blog topic:

ASP.NET Thread Usage on IIS 7.0 and 6.0
http://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx