Unable to connect using Exchange ActiveSync due to Exchange resource consumption

Article ID: 2469722 - View products that this article applies to.
Expand all | Collapse all

Symptoms

  Initial symptoms will include users not being able to synchronize their devices with Exchange.  Additionally, the following symptoms may occur:
  • In the Application log on the Exchange server, you may see the following events:

Event Source: Server ActiveSync
Event ID: 3007
Description:
Exchange mailbox Server response timeout: Server: [mail.contoso.com] User: [user@contoso.com]. Exchange ActiveSync Server failed to communicate with the Exchange mailbox server in a timely manner. Verify that the Exchange mailbox Server is working correctly and is not overloaded.

Event Source: Server ActiveSync
Event ID: 3014
Description:
The Exchange mailbox Server: [mail.contoso.com] has reached its timeout threshold. The mailbox server will be protected from new requests for [60] seconds.

  • The number of allowed RPC connections to the mailbox server may exceed the recommended limits as well.
    The maximum number of RPC requests that can execute at any given time before the Information Store begins rejecting new connections is 500 for Exchange 2007 and 2010, however the counter should remain below 70 at all times. Anything above this indicates a performance bottleneck. To confirm this behavior, one can check the Performance Monitor counter “MSExchangeIS\RPC Requests” which will show if the server is above recommended limits.

    Please see http://technet.microsoft.com/en-us/library/bb201689%28EXCHG.80%29.aspx

  • In the W3SVC logs, you will see HTTP 409 and 503 responses returned for Microsoft-Server-ActiveSync requests.
    You may also see the error returned “TooManyJobsQueued” in the W3SVC logs.  See the following examples:

    2010-09-09 21:35:35 W3SVC1 10.0.0.10 POST /Microsoft-Server-ActiveSync/default.eas User=<user>&DeviceId=<DeviceID>&DeviceType=<Type>&Cmd=Sync=Ping=<Data> 443 Domain\User 10.0.0.20 <UserAgent> 503 0 0

    2010-09-15 00:00:02 W3SVC1 155.109.199.51 POST /Microsoft-Server-ActiveSync/default.eas Cmd=Ping&User=<User>&DeviceId=<DeviceID>&DeviceType=<Type>&Log=Error:TooManyJobsQueued_ 443 Domain\User 10.0.0.20 <UserAgent> 503 0 0

  • The HTTPErr logs will show Connection_Dropped for /Microsoft-Server-ActiveSync requests.
    You should be able to see that the s-port or Source Port the Connection_Dropped is coming from reaches a fairly high number.  For example:

    2010-09-09 00:27:25 10.0.0.10 64637 10.0.0.20 443 HTTP/1.1 POST /Microsoft-Server-ActiveSync?User=<User>&DeviceId=<DeviceID>&DeviceType=<Type>&Cmd=Ping - 1 Connection_Dropped MSExchangeSyncAppPool
  •  For Exchange 2003, the server will be depleted of Non-Paged Pool (NPP) memory.
    As the connection limit is reached, NPP is consumed and HTTP.sys begins dropping HTTP connections once the server begins to peak 100 megabytes of NPP depending on the server configuration. Once the server reaches closer to 108 megabytes, HTTP.sys will fail all connections. NPP can be checked using Task Manager and viewing Performance and checking the Kernel Memory.   

For more information around Exchange and NPP, see http://technet.microsoft.com/en-us/library/aa996269(EXCHG.80).aspx

For information on using Netstat, see http://technet.microsoft.com/en-us/library/cc940097.aspx

For information on downloading and using TCPView, please see  http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx

 

Cause

This can occur due to mobile device connections and/or issues with the device software.  There are several key causes for the connection failures and causes of memory depletion:

1. The server begins dropping connections, due to the number of requests to the server, or the requests are exceeding the number of allowed Application Pool IIS connections.  Likewise, reaching the TCP port limit on the Exchange server(s) is a cause.

NOTE: More than likely, the servers may reach one of the other limits (Application Pool Queue length on CAS or RPC Requests on Mailbox server) before getting close to TCP Port exhaustion. 

The following article discusses port exhaustion:
http://technet.microsoft.com/en-us/library/cc540453(EXCHG.80).aspx

The following articles mention the use of MaxUserPort and TCP connections in Windows and Exchange

http://technet.microsoft.com/en-us/library/cc940037.aspx

http://technet.microsoft.com/en-us/library/bb397382(EXCHG.80).aspx

2. A mobile device synchronizing using Exchange ActiveSync may simulate the behavior of a DOS attack against a server.  Some examples are listed in the following article:
http://technet.microsoft.com/en-us/library/cc182260.aspx

3. The Application Pool is receiving requests faster than it can handle them.  This is discussed in the following article:
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/9701f9dd-d40b-4d24-a2df-1d1abc13f764.mspx?mfr=true

 

Resolution

To stop these devices from consuming Exchange resources, and to prevent the burden, the suspect devices should be blocked. 

IMPORTANT: There is a known issue with iPhone OS 4.0 defined by user agent 801.293 - http://support.apple.com/kb/TS3398

Any IIS logs that contain this user agent should be flagged, and the owner of that device should update to OS 4.1 or above (preferably, iOS version 4.3.x).  We recommend that these users be blocked in the interim.

Devices can be blocked by disabling the Exchange ActiveSync access for the device owner, turning off the device, or using an Internet Security and Acceleration (ISA) or ForeFront Threat Management Gateway (TMG) filter.

For additional information on blocking devices, see the following post on the Microsoft Exchange Team (EHLO) blog:

http://msexchangeteam.com/archive/2008/09/05/449757.aspx

NOTE: Some devices may continue to attempt to connect to the Exchange server (causing a high number of TCP connections) even when the user is disabled for Exchange ActiveSync use. We recommend turning off the device or block the device using an ISA filter.

Exchange Server 2010

If the user with the suspect device user has an Exchange 2010 mailbox, throttling can be enabled and for all Exchange ActiveSync users to prevent devices from overburdening the Exchange server.  See the following article from Microsoft TechNet online:

 http://technet.microsoft.com/en-us/library/dd297964.aspx

Steps to do this are included in the article, however here are some examples:

$a = Get-ThrottlingPolicy | Where-object {$_.IsDefault –eq $true}

$a | Set-ThrottlingPolicy –EASMaxConcurrency 10

For more information on how to set throttling policies, see the following topic from TechNet online:

 http://technet.microsoft.com/en-us/library/dd298094.aspx

Additional factors to check that influence connections per device

  1. Is there high Item Count in the primary folders (Inbox being the main folder)? When there are several thousand items in the Inbox folder and the device is attempting to sync all these items, you will see a large number of connections/requests from this device in order to download all of these items. The solution for this includes:

    a. Set the device to only synchronize the last day or no more than 3 days depending on the number of items that user receives in a day.
    b. Reduce the item count in the mailbox
  2. Are the user accounts disabled for Exchange ActiveSync? This may increase the number of device connections to the server. If the device is still sending several requests to the server, then turn off the device or put a filter in place on ISA to block the DeviceID.
  3. Are the users exceeding their mailbox quota limit? If so, the device will not be able to sync new items to the mailbox and may continue trying over and over again.
  4. Are there message size limits set in the Organization? If there are no limits, you may have a device that is attempting to synchronize large amounts of data.

    NOTE: In addition to the above, continue to monitor device connections using Log Parser for suspect devices. See More Information for assistance in using Log Parser.

When a suspect device is found, we recommend that customers contact the device vendor for assistance in determining why the device is sending the excessive number of requests to Exchange.

See the section "Steps that will assist administrators while troubleshooting:" in the More Information section for additional actions administrators can take to try to improve performance while troubleshooting. 

More Information

Which devices are suspect?
Usually if a device is sending over 1000 requests per day, we recommend investigating.

If the hits (requests) are above 1500, there could be an issue on the device or environment. The device and the user’s activity should be investigated. 

See the following post from the Exchange Team blog and .PS1 file that can be used with Windows Powershell to parse IIS log files for any version of Exchange Server.

A script to troubleshoot issues with Exchange ActiveSync
http://blogs.technet.com/b/exchange/archive/2012/01/31/a-script-to-troubleshoot-issues-with-exchange-activesync.aspx

ActiveSyncReport script
http://gallery.technet.microsoft.com/scriptcenter/ActiveSyncReport-script-a2417a84

Log Parser can be used without the script, but the script was designed to filter based on criteria specified in the command line used.  Using the script above, e-mail alerts or reports can also be created.

Steps that will assist administrators while troubleshooting:

The following options will not resolve the issue.  They will give you more time to parse through the IIS logs to determine the devices are suspect.

  • Reduce the Keep Alive time value on the Exchange Mailbox servers to send a keep alive every 30 minutes to CAS instead of every 2 hours.

    The Keep Alive Value set to 30 minutes (1800000) in the registry, discussed in the following article in the Microsoft Knowledge Base online:
    http://support.microsoft.com/kb/324270  

    The Registry value information is below:
    Path: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\
    Value name: KeepAliveTime
    Value Type: REG_DWORD-Time in milliseconds
                        Valid Range: 1-0xFFFFFFFF
                        Default: 7,200,000 (two hours)

    This value controls how frequently TCP tries to verify that an idle connection is still intact by sending a keep-alive packet. If the remote computer is still reachable, it acknowledges the keep-alive packet. Keep-alive packets are not sent by default. You can use a program to configure this value on a connection. The recommended value setting is 1,800,000 (equal to 30 minutes). (Decimal) (Requires Server restart)

    Monitor this setting and if the server ports are still being exhausted, then this may need to be lowered to 15 minutes instead of 30 minutes.
  • Increase the IIS Connection Timeout in IIS

    The default and recommended value should be 120 seconds. If it is set at a higher value, then it should be set between the 120 seconds minimum recommended value to the 300 second maximum value.
    See the following topic in TechNet online for more information:

    http://technet.microsoft.com/en-us/library/cc182260.aspx

    This can be done by via the Properties of the Default Web Site by modifying the Connection Timeout value (in seconds).

  • Increase the number of IIS connections (Queue Length) for the ExchangeApplicationPool on the Exchange 2003 Mailbox and Front-end servers:


    Use Internet Information Services Manager to turn off worker process recycling in IIS 6.0
    1.   Start Internet Information Services (IIS) Manager.
    2.   Expand the local computer, expand Application Pools, right-click the ExchangeApplicationPool application pools, and then click Properties.
    3.   Click to clear the Recycle worker processes (in minutes) check box, and then click OK.

    Use Internet Information Services Manager to increase the queue length in IIS 6.0
    1.   Start Internet Information Services (IIS) Manager.
    2.   Expand the local computer, expand Application Pools, right-click the ExchangeApplicationPool application pool, and then click Properties.
    3.   Click the Performance tab, and then modify the value in the Request queue limit box. Replace the default value of 1000 with 4000.
    4.   Click OK.
  • Increase the number of IIS connections (Queue Length) for the MSExchangeSyncAppPool on the Exchange 2007 and Exchange 2010 Client Access Servers

    Use Internet Information Services Manager to turn off worker process recycling in IIS 7.0 and above (This should be disabled by default for the MSExchangeSyncAppPool)

    1. Start Internet Information Services Manager.
    2. Expand the local computer, and then click Application Pools.
    3. In the Application Pools pane, click the appropriate application pool, such as MSExchangeSyncAppPool or the new application pool that you created, and then click Advanced Settings.
    4. In the Recycling section, modify the Regular Time Interval (minutes) value. Replace the default value of 1740 with 0 (zero). A value of zero turns off worker process recycling.
    5. Click OK.
  • Use Internet Information Services Manager to increase the queue length in IIS 7.0 and above

    1. Start Internet Information Services Manager.
    2. Expand the local computer, and then click Application Pools.
    3. In the Application Pools pane, click the appropriate application pool, such as MSExchangeSyncAppPool or the new application pool that you created, and then click Advanced Settings.
    4. In the General section, modify the Queue Length value. Replace the default value of 1000 with 10000.
    5. Click OK

NOTE: If running Exchange 2007 on Windows 2003 (IIS 6), follow the steps above to increase the Request Queue limit for the MSExchangeSyncAppPool.

For more information on why it may be important to change these values for Application Pools and for additional links to “How To” information, see the following topic from TechNet online:

http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/9701f9dd-d40b-4d24-a2df-1d1abc13f764.mspx?mfr=true

Note This is a "FAST PUBLISH" article created directly from within the Microsoft support organization. The information contained herein is provided as-is in response to emerging issues. As a result of the speed in making it available, the materials may include typographical errors and may be revised at any time without notice. See Terms of Use for other considerations.

Properties

Article ID: 2469722 - Last Review: February 24, 2012 - Revision: 11.0
APPLIES TO
  • Microsoft Exchange Server 2003 Enterprise Edition
  • Microsoft Exchange Server 2007 Enterprise Edition
  • Microsoft Exchange Server 2010 Enterprise
Keywords: 
KB2469722

Give Feedback

 

Contact us for more help

Contact us for more help
Connect with Answer Desk for expert help.
Get more support from smallbusiness.support.microsoft.com