SharePoint 2010: Crawl process is stuck after SQL Server runs out of disk space

Applies to: SharePoint Server 2010SharePoint 2010 For Internet Sites Enterprise

Summary


Consider the following scenario:

You initiate a full/incremental crawl of the content source(s) in your SharePoint 2010 environment. Once the crawl begins, you find that the Crawl Store database (e.g. something such as 'SSA_CrawlStoreDB') for the Search Service Application (SSA) begins to grow causing the SQL Server to run out of disk space for either the database or transaction log directory. In this situation, the crawl will appear stuck in a 'Crawling' state. If an attempt is made to stop the crawl, the status will appear stuck in a 'Stopping' state. Additionally, in this disk starved state, the SQL Server may experience degraded performance (particularly if the disk space for TempDB is also impacted).

More Information


In this scenario, the crawl gets into a loop where the SQL Server attempts to allocate additional space for the applicable data or log directory. However, because there is insufficient disk space (or the destination does not allow for auto-growth), the space allocation will fail and the transaction for Search will be rolled back. The Search processes then attempt to repeat the failed transaction, which will continue to fail.

Attempting to stop the crawl (e.g. from the Search Admin page) could exacerbate the disk starvation problem (notably in large crawls) because this action could generate a potentially large number of transactions to delete items from the related crawl queue tables and "un-wind" the crawl process. As such, it is best to avoid stopping the crawl.

Important: Unless explicitly advised by a member of the Microsoft SharePoint Development Team or a Microsoft SharePoint Support agent, modifying or manually rolling back SharePoint Search stored procedure(s) or related transaction(s) is not a supported action and could result with databases being in an inconsistent state. Please refer to Support for changes to SharePoint databases for additional details.
If the SQL Server runs out of space during a crawl, use the following steps to resume the crawl processing:
  1. From each server with a SharePoint Crawl Component, stop the 'SharePoint Server Search 14' and 'SharePoint 2010 Timer' services from either the Services snap-in (services.msc) or from a command prompt using 'net stop osearch14' and 'net stop spTimerV4'
  2. From SQL Server Management Studio, detach the affected Crawl Store DB(s)
  3. Move the database and/or log (e.g. the .mdf, .ldf, and .ndf) files for the affected Crawl Store DB(s) to a drive path that provides sufficient space for growth
  4. From SQL Server Management Studio, re-attach the affected Crawl Store DB(s)
  5. From each server with a SharePoint Crawl Component, start the 'SharePoint Server Search 14' and 'SharePoint 2010 Timer' services from either the Services snap-in (services.msc) or from a command prompt using 'net start osearch14' and 'net start spTimerV4'

After restarting the Search and Timer services on all servers running a crawl component as noted above, the crawl(s) should fully recover from the disk space condition and continue with normal processing. However, if the crawl was stopped as a troubleshooting measure, it is advised to let the 'Stopping' process complete as usual (which may require a significant amount of time for large crawls). 

Additional Notes:


 - Taking the Crawl Store DB offline will only impact crawls, and the SSA Admin and Query components should remain unaffected
 - Crawls will make no further progress until the Crawl Store DB(s) is attached 
 - Detaching the Property Store DB would cause queries to fail
 - Detaching the SSA Admin DB would prevent the execution of administrative commands (e.g. crawls could not be started) and could cause failures with
 queries by blocking requests for ACL information from the Query Processor during the query-side security trimming process

For more information, see the following articles:

Move all databases (SharePoint Server 2010)
Detaching and Attaching Databases
How to: Attach a Database (SQL Server Management Studio)
How to: Detach a Database (SQL Server Management Studio)
sp_detach_db (Transact-SQL)
sp_attach_db (Transact-SQL)