Initial symptoms will include users not being able to synchronize their devices with Exchange.Additionally, the following symptoms may occur:
In the Application log on the Exchange server, you may see the following events:
Event Source: Server ActiveSync Event ID: 3007 Description: Exchange mailbox Server response timeout: Server: [mail.contoso.com] User: [firstname.lastname@example.org]. Exchange ActiveSync Server failed to communicate with the Exchange mailbox server in a timely manner. Verify that the Exchange mailbox Server is working correctly and is not overloaded.
Event Source: Server ActiveSync Event ID: 3014 Description: The Exchange mailbox Server: [mail.contoso.com] has reached its timeout threshold. The mailbox server will be protected from new requests for  seconds.
The number of allowed RPC connections to the mailbox server may exceed the recommended limits as well. The maximum number of RPC requests that can execute at any given time before the Information Store begins rejecting new connections is 500 for Exchange 2007 and 2010, however the counter should remain below 70 at all times. Anything above this indicates a performance bottleneck. To confirm this behavior, one can check the Performance Monitor counter “MSExchangeIS\RPC Requests” which will show if the server is above recommended limits.
In the W3SVC logs, you will see HTTP 409 and 503 responses returned for Microsoft-Server-ActiveSync requests. You may also see the error returned “TooManyJobsQueued” in the W3SVC logs.See the following examples:
The HTTPErr logs will show Connection_Dropped for /Microsoft-Server-ActiveSync requests. You should be able to see that the s-port or Source Port the Connection_Dropped is coming from reaches a fairly high number.For example:
For Exchange 2003, the server will be depleted of Non-Paged Pool (NPP) memory. As the connection limit is reached, NPP is consumed and HTTP.sys begins dropping HTTP connections once the server begins to peak 100 megabytes of NPP depending on the server configuration. Once the server reaches closer to 108 megabytes, HTTP.sys will fail all connections. NPP can be checked using Task Manager and viewing Performance and checking the Kernel Memory.
This can occur due to mobile device connections and/or issues with the device software. There are several key causes for the connection failures and causes of memory depletion:
1. The server begins dropping connections, due to the number of requests to the server, or the requests are exceeding the number of allowed Application Pool IIS connections.Likewise, reaching the TCP port limit on the Exchange server(s) is a cause.
NOTE: More than likely, the servers may reach one of the other limits (Application Pool Queue length on CAS or RPC Requests on Mailbox server) before getting close to TCP Port exhaustion.
Any IIS logs that contain this user agent should be flagged, and the owner of that device should update to OS 4.1 or above (preferably, iOS version 4.3.x).We recommend that these users be blocked in the interim.
Devices can be blocked by disabling the Exchange ActiveSync access for the device owner, turning off the device, or using an Internet Security and Acceleration (ISA) or ForeFront Threat Management Gateway (TMG) filter.
For additional information on blocking devices, see the following post on the Microsoft Exchange Team (EHLO) blog:
NOTE: Some devices may continue to attempt to connect to the Exchange server (causing a high number of TCP connections) even when the user is disabled for Exchange ActiveSync use. We recommend turning off the device or block the device using an ISA filter.
Exchange Server 2010
If the user with the suspect device user has an Exchange 2010 mailbox, throttling can be enabled and for all Exchange ActiveSync users to prevent devices from overburdening the Exchange server.See the following article from Microsoft TechNet online:
Additional factors to check that influence connections per device
Is there high Item Count in the primary folders (Inbox being the main folder)? When there are several thousand items in the Inbox folder and the device is attempting to sync all these items, you will see a large number of connections/requests from this device in order to download all of these items. The solution for this includes:
a.Set the device to only synchronize the last day or no more than 3 days depending on the number of items that user receives in a day. b.Reduce the item count in the mailbox
Are the user accounts disabled for Exchange ActiveSync? This may increase the number of device connections to the server. If the device is still sending several requests to the server, then turn off the device or put a filter in place on ISA to block the DeviceID.
Are the users exceeding their mailbox quota limit? If so, the device will not be able to sync new items to the mailbox and may continue trying over and over again.
Are there message size limits set in the Organization? If there are no limits, you may have a device that is attempting to synchronize large amounts of data.
NOTE: In addition to the above, continue to monitor device connections using Log Parser for suspect devices. See More Information for assistance in using Log Parser.
When a suspect device is found, we recommend that customers contact the device vendor for assistance in determining why the device is sending the excessive number of requests to Exchange.
See the section "Steps that will assist administrators while troubleshooting:" in the More Information section for additional actions administrators can take to try to improve performance while troubleshooting.
Which devices are suspect? Usually if a device is sending over 1000 requests per day, we recommend investigating.
If the hits (requests) are above 1500, there could be an issue on the device or environment. The device and the user’s activity should be investigated.
See the following post from the Exchange Team blog and .PS1 file that can be used with Windows Powershell to parse IIS log files for any version of Exchange Server.
Log Parser can be used without the script, but the script was designed to filter based on criteria specified in the command line used. Using the script above, e-mail alerts or reports can also be created.
Steps that will assist administrators while troubleshooting:
The following options will not resolve the issue.They will give you more time to parse through the IIS logs to determine the devices are suspect.
Reduce the Keep Alive time value on the Exchange Mailbox servers to send a keep alive every 30 minutes to CAS instead of every 2 hours.
The Registry value information is below: Path: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\ Value name: KeepAliveTime Value Type: REG_DWORD-Time in milliseconds Valid Range: 1-0xFFFFFFFF Default: 7,200,000 (two hours)
This value controls how frequently TCP tries to verify that an idle connection is still intact by sending a keep-alive packet. If the remote computer is still reachable, it acknowledges the keep-alive packet. Keep-alive packets are not sent by default. You can use a program to configure this value on a connection. The recommended value setting is 1,800,000 (equal to 30 minutes). (Decimal) (Requires Server restart)
Monitor this setting and if the server ports are still being exhausted, then this may need to be lowered to 15 minutes instead of 30 minutes.
Increase the IIS Connection Timeout in IIS
The default and recommended value should be 120 seconds. If it is set at a higher value, then it should be set between the 120 seconds minimum recommended value to the 300 second maximum value. See the following topic in TechNet online for more information:
This can be done by via the Properties of the Default Web Site by modifying the Connection Timeout value (in seconds).
Increase the number of IIS connections (Queue Length) for the ExchangeApplicationPool on the Exchange 2003 Mailbox and Front-end servers:
Use Internet Information Services Manager to turn off worker process recycling in IIS 6.0 1.Start Internet Information Services (IIS) Manager. 2.Expand the local computer, expand Application Pools, right-click the ExchangeApplicationPool application pools, and then click Properties. 3.Click to clear the Recycle worker processes (in minutes) check box, and then click OK.
Use Internet Information Services Manager to increase the queue length in IIS 6.0 1.Start Internet Information Services (IIS) Manager. 2.Expand the local computer, expand Application Pools, right-click the ExchangeApplicationPool application pool, and then click Properties. 3.Click the Performance tab, and then modify the value in the Request queue limit box. Replace the default value of 1000 with 4000. 4.Click OK.
Increase the number of IIS connections (Queue Length) for the MSExchangeSyncAppPool on the Exchange 2007 and Exchange 2010 Client Access Servers
Use Internet Information Services Manager to turn off worker process recycling in IIS 7.0 and above (This should be disabled by default for the MSExchangeSyncAppPool)
1.Start Internet Information Services Manager. 2. Expand the local computer, and then click Application Pools. 3. In the Application Pools pane, click the appropriate application pool, such as MSExchangeSyncAppPool or the new application pool that you created, and then click Advanced Settings. 4.In the Recycling section, modify the Regular Time Interval (minutes) value. Replace the default value of 1740 with 0 (zero). A value of zero turns off worker process recycling. 5.Click OK.
Use Internet Information Services Manager to increase the queue length in IIS 7.0 and above
1.Start Internet Information Services Manager. 2.Expand the local computer, and then click Application Pools. 3.In the Application Pools pane, click the appropriate application pool, such as MSExchangeSyncAppPool or the new application pool that you created, and then click Advanced Settings. 4.In the General section, modify the Queue Length value. Replace the default value of 1000 with 10000. 5.Click OK
NOTE: If running Exchange 2007 on Windows 2003 (IIS 6), follow the steps above to increase the Request Queue limit for the MSExchangeSyncAppPool.
For more information on why it may be important to change these values for Application Pools and for additional links to “How To” information, see the following topic from TechNet online: