Exchange 2007 DB resources fail to shutdown before timeout (SCC), Event 1115, 482, 414, 492

Source: Microsoft Support

RAPID PUBLISHING ARTICLES PROVIDE INFORMATION DIRECTLY FROM WITHIN THE MICROSOFT SUPPORT ORGANIZATION. THE INFORMATION CONTAINED HEREIN IS CREATED IN RESPONSE TO EMERGING OR UNIQUE TOPICS, OR IS INTENDED SUPPLEMENT OTHER KNOWLEDGE BASE INFORMATION.

When attempting to ‘Take Offline’ or ‘Move-CMS’ an Exchange 2007 Single
Copy Cluster (SCC) which is under a heavy load, it is possible that any database that
has not dismounted can shutdown (uncleanly).

The following can be seen in the cluster.log:

Attempting to take the DB offline:
00000d18.00001d60::2006/12/08-09:00:45.839 INFO Microsoft Exchange Database
Instance <SG1/SG1DB1 (EVS1)>: [EXRES] calling EcUnmountDatabase()
The failure event after the timeout has been exceeded:
00000d18.00001b1c::2006/12/08-09:01:25.828 ERR Microsoft Exchange Database Instance
<SG1/SG1DB1 (EVS1)>: [EXRES] EventLogging: Clustered Mailbox Server: EVS1 Physical
Server: <ServerName>Failed to bring the resource SG1/SG1DB1 (EVS1) offline due to a
timeout. Error Code: 1460.
2006/12/08-09:01:25.828 WARN Microsoft Exchange Database Instance <SG1/SG1DB1
(EVS1)>: [EXRES] State change: from 130 (OfflinePending) to 4 (Failed).



You may also see the following events in the application log on the passive node:

Event Type: Warning
Event Source: MSExchangeIS Mailbox Store
Event Category: General
Event ID: 1115
Description:
Error 0xfffffbbe returned from closing database table, called from function
JTAB_BASE::EcCloseTable on table Folders.



You will also see ESE errors indicating commit failures:



Event Type: Error
Event Source: ESE
Event Category: General
Event ID: 104
Description:
MSExchangeIS (4284) <SG>: The database engine stopped the instance (4) with error
(-1090).

Event Type: Error
Event Source: ESE
Event Category: General
Event ID: 482
Description:
MSExchangeIS (4284) SG06: An attempt to write to the file
< log file> at offset 457216 (0x000000000006fa00) for 512
(0x00000200) bytes failed after 0 seconds with system error 21 (0x00000015): "The
device is not ready. ". The write operation will fail with error -1022
(0xfffffc02). If this error persists then the file may be damaged and may need to
be restored from a previous backup.

Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 414
Description:
MSExchangeIS (4284) SG06: Unable to write to section 0 while flushing logfile
<log file> Error -1022 (0xfffffc02).

Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 492
Description:
MSExchangeIS (4284) <SG>: The logfile sequence in <log file> has
been halted due to a fatal error. No further updates are possible for the
databases that use this logfile sequence. Please correct the problem and restart
or restore from backup.

Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Event ID: 471
Description:
MSExchangeIS (4284) <SG>: Unable to rollback operation #141661 on database
<Exchange Database>. Error: -510. All future database updates will be rejected.

This issue occurs because once the database resources reach the timeout value they go into a failed state.  Once this occurs the waiting dependent resources such as physical disk are brought offline by cluster and the errors are seen. 

It is possible to eliminate these events by increasing the timeout on the database resources. Increasing the timeout can give the DBs more time to go down cleanly, avoiding the ESE errors.

The default timeout is 180 seconds.  Increasing the timeout, does increase the time it takes to failover.  The timeout will typically be customer specific. 



Typically, for most clusters a timeout value of 5 minutes (300 seconds) should alleviate the problem.


The value is set with the following command:

cluster res "<Database Resource Name>” /prop PendingTimout=300000

 

This can also be set in Cluster Administrator on the properties of the database resource(s) on the advanced tab, changing pending timeout to the desired value. 

Note:  After applying SP1 there have been some improvements flushing the cache but customers with large SCC clusters can still hit this issue. 

To see the pending timeout on a database resource you can run the following:


Cluster /res <SGName/DBName (CMSName) > /prop

For Example:

C:\>cluster res "First Storage Group/Mailbox Database (SCC-Mail1)" /prop
Listing properties for 'First Storage Group/Mailbox Database (SCC-Mail1)':
T Resource Name Value
-- -------------------- ------------------------------ -----------------------
SR First Storage Group/Mailbox Database (SCC-Mail1) Name
First Storage Group/Mailbox Database (SCC-Mail1)
S First Storage Group/Mailbox Database (SCC-Mail1) Type
Microsoft Exchange Database Instance
S First Storage Group/Mailbox Database (SCC-Mail1) Description

S First Storage Group/Mailbox Database (SCC-Mail1) DebugPrefix

D First Storage Group/Mailbox Database (SCC-Mail1) SeparateMonitor
0 (0x0)
D First Storage Group/Mailbox Database (SCC-Mail1) PersistentState
1 (0x1)
D First Storage Group/Mailbox Database (SCC-Mail1) LooksAlivePollInterval
4294967295 (0xffffffff)
D First Storage Group/Mailbox Database (SCC-Mail1) IsAlivePollInterval
4294967295 (0xffffffff)
D First Storage Group/Mailbox Database (SCC-Mail1) RestartAction
1 (0x1)
D First Storage Group/Mailbox Database (SCC-Mail1) RestartThreshold
1 (0x1)
D First Storage Group/Mailbox Database (SCC-Mail1) RestartPeriod
900000 (0xdbba0)
D First Storage Group/Mailbox Database (SCC-Mail1) RetryPeriodOnFailure
4294967295 (0xffffffff)
D First Storage Group/Mailbox Database (SCC-Mail1) PendingTimeout
180000 (0x2bf20)
D First Storage Group/Mailbox Database (SCC-Mail1) LoadBalStartupInterval
300000 (0x493e0)
D First Storage Group/Mailbox Database (SCC-Mail1) LoadBalSampleInterval
10000 (0x2710)
D First Storage Group/Mailbox Database (SCC-Mail1) LoadBalAnalysisInterval
300000 (0x493e0)
D First Storage Group/Mailbox Database (SCC-Mail1) LoadBalMinProcessorUnits
0 (0x0)
D First Storage Group/Mailbox Database (SCC-Mail1) LoadBalMinMemoryUnits
0 (0x0)



Note: The default timeout is 180 seconds.

MICROSOFT AND/OR ITS SUPPLIERS MAKE NO REPRESENTATIONS OR WARRANTIES ABOUT THE SUITABILITY, RELIABILITY OR ACCURACY OF THE INFORMATION CONTAINED IN THE DOCUMENTS AND RELATED GRAPHICS PUBLISHED ON THIS WEBSITE (THE “MATERIALS”) FOR ANY PURPOSE. THE MATERIALS MAY INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS AND MAY BE REVISED AT ANY TIME WITHOUT NOTICE.

TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, MICROSOFT AND/OR ITS SUPPLIERS DISCLAIM AND EXCLUDE ALL REPRESENTATIONS, WARRANTIES, AND CONDITIONS WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING BUT NOT LIMITED TO REPRESENTATIONS, WARRANTIES, OR CONDITIONS OF TITLE, NON INFRINGEMENT, SATISFACTORY CONDITION OR QUALITY, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WITH RESPECT TO THE MATERIALS.