SBSL: OS: Http <-> Crypto deadlock causes slow boot and service start failure on SSL-enabled W2K8 computers

Symptoms

This article has been replaced by KB 2379016.

KB
2379016provides a hotfix to resolve this issue on a computer that is running Windows Vista or Windows Server 2008.   We recommend you resolve this issue by installing the hotfix. 

For more information, please see KB 2379016  2379016
You should consider integrating the hotfix into the Windows Server build process if you plan to deploy Windows Server 2008 computers in the affected configuration.

Windows Server 2008 R2 is not affected by the problem described in this KB article.

The following symptoms may occur:

  • Windows Server 2008 hangs after boot at Applying Computer Settings orApplying Security Policy
  • Once the server finishes booting a user attempting to log on may hang at Applying User Settings
  • You may notice that services that are set to a Start Type of "Automatic" may not start 

Certain Services that are set to "Automatic" may start without problems - for example:

  • Dcom Process Launcher
  • Remote Procedure Call
  • Event log
  • Group Policy Client
  • Plug and Play
  • DHCP Client
  • DNS Client
  • Task Scheduler
  • Base Filtering Engine
  • Workstation Service
  • Netlogon

Other services set to "Automatic" may fail - for example:

  • Print Spooler
  • Terminal Services
  • Server service
  • Remote Registry
  • WMI
  • Distributed Transaction Cordinator
  • Any Services related to Applications

Trying to manually start services with a Startup type of "Automatic" may result in an Error 1053 indicating that "The service did not respond to the start or control request in a timely fashion." 

Cause

The problems described in the symptoms section occur because of a lock on the Service Control Manager (SCM) database.  As a result of the lock, none of the services can access the SCM database to initialize their service start requests. To verify that a Windows computer is affected by the problem discussed in this article, run the following command from the command Prompt:

sc querylock

The output below would indicate that the SCM database is locked:

QueryServiceLockstatus - Success
IsLocked : True
LockOwner : .\NT Service Control Manager
LockDuration : 1090 (seconds since acquired)

There is no additional information in the Event Logs beyond those from the Service Control Manager indicating that Service startup requests have timed out. The underlying root cause is a deadlock between the Service Control Manager and HTTP.SYS.

Resolution

To resolve this problem, install the hotfix described in KB article 23790162379016
To work around this issue, go to the "Fix it for me" section. If you’d rather resolve this problem yourself, go to the "Let me fix it myself" section.

Fix it for me

To resolve this problem automatically, click the Fix this problem link. Then clickRun in the File Download dialog box, and follow the steps in this wizard.


Fix this problem
Microsoft Fix it 50564



Note This wizard may be in English only. However, the automatic fix also works for other language versions of Windows.

Note If you are not on the computer that has the problem, you can save the automatic fix to a flash drive or to a CD so that you can run it on the computer that has the problem.

Let me fix it myself

you can modify the behavior of HTTP.SYS to depend on another service being started first.  To do this, perform the following steps:

  1. Open Registry Editor
  2. Navigate to HKLM\SYSTEM\CurrentControlSet\Services\HTTP and create the following Multi-string value:DependOnService
  3. Double click the new DependOnService entry
  4. Type CRYPTSVC in the Value Data field and click OK.
  5. Reboot the server

NOTE: Please ensure that you make a backup of the registry / affected keys before making any changes to your system.

More Information

Beginning with Windows Server 2008, Windows does not wait on all of the Automatic Services startup to load Explorer.exe.  Services may be set to Delayed Automatic Start to increase boot performance.  Please see the following blog posts for more information on Delayed Automatic Start:

Startup Processes and Delayed Automatic Start.
http://blogs.technet.com/b/askperf/archive/2008/02/02/ws2008-startup-processes-and-delayed-automatic-start.aspx?wa=wsignin1.0

More information:

HTTP.SYS / Cryptographic Services / LSASS.EXE deadlock
http://blogs.technet.com/b/mrsnrub/archive/2009/11/19/http-sys-cryptographic-services-lsass-exe-deadlock.aspx

Microsoft Internal Support Information

2010.10.21: A public version of the QFE for this problem is available athttp://support.microsoft.com/default.aspx?scid=kb;EN-US;2379016. The internal location for the public hotfix ishttp://hotfix/search.aspx?search=2379016.

Other symptoms reported by Microsoft CSS

 

Article #

Claim / Symptom

Bemis 2204234

No network connectivity to Windows computers with this problem

Bemis 2229715

Windows Server 2008 computers hang while “configuring updates … stage 3 of 3”.

Bemis 2250653

Unable to RDP to affected Windows Server 2008 computers

Bemis 2266552
Bemis 2419420

Exchange Server fails to start

Bemis 2275910

The FW service on TMG fails to start

Bemis 2305103

Unable to RDP to affected server, IIS Service fails to start, Server manager contents not visible

Bemis 2380212

SQL, SMS and McAfee related services fail to start

Bemis 2271166

Error 504 when trying to establish IM session with more than 2 participants

Bemis 2285569

WMI service fails to start, Server Service fails to start

Bemis 2360149

DCOM service fails to start, Eventviewer service fails to start

 

 <add additional symptoms here>


MSCONFIDENTIAL. DO NOT SHARE OR DISCUSS THIS SECTION OF THE ARTICLE WITH CUSTOMERS, PARTNERS OR OEMS, EVEN IF UNDER NDA.

Article author Prakash Gopinadham notes that of the 50 like cases he is aware of, the affected computer is a web server that has sslbindings configured on a web site (hklm\system\ccs\services\http\parameters\sslbindinginfo). Importing new certificates into the web server also creates a dependency for http to call into cryptsvc and eventually results in the SCM lock.

Windows 7 bug 169415 suggests that this problem was fixed in Windows Server 2008 R2


Call coding:

This issue should be coded to the following node of the Windows server 2008 root cause call code tree:

NETWORKING -> HTTP->HTTP.SYS -> CODE DEFECT-> KB 2004121 DEADLOCK BETWEEN HTTP.SYS AND CRYPTO .

2010.07.21 issue explanation from Tonyga

The service control manager (SCM) is trying to start the HTTP.SYS service.  As a normal part of starting a service the SCM puts a lock on the table that contains the state of services.  We'll call this thread thread1. 

During its startup HTTP.SYS makes a call that requires the cryptsvc.  The cryptsvc has not started yet so this action triggers a call to SCM to start the cryptsvc.  

We'll call the request to start cryptsvc thread2.

Since the service state table is locked by thread1 above thread2 waits for the lock to be released so it can start cryptsvc.

Thus we have a classic deadlock, two threads waiting for each other to finish.

The resolution in the SOX of setting the HTTP service to be dependent on the cryptsvc resolves the issue as the cryptsvc is always started before HTTP.sys

Related content / bugs 

WINSE348548 - Test follow-up for Windows SE Bug 345755: Deadlock between services.exe and http.sys over services!ScServiceStartCriticalSection and http!DriverEntry completion
WINSE 303377  - Backport of Win7 169415: HTTP.SYS: Deadlock between services.exe and http.sys ....<snip>
Windows 7 bug 169415 - HTTP.SYS: Deadlock between services.exe and http.sys over services!ScServiceStartCriticalSection and http!DriverEntry completion
Bemis 2019683 - 2008 R2 Web Server hangs at "Applying Computer Settings" after adding certificate to web site.
Bemis 2158690 - PTSMENBR\PRO\W2K8SRV\x64\EST\v-2rokan\Slow booting when meet "Applying computer settings..."
Bemis 2159616 - SBS 2008 Virtualized \Setup\Unable to boot the server in normal mode. Hangs at applying computer settings.
Bemis 2196875 - PTSMESHS\PRO\W2K8R2SRV\SP2\x64\EST\v-2chpav\Server hung at applying computer settings and is taking a long time
SOX091012700008 - Windows Server 2008/All Automatic Services won’t start upon Boot / Computer hangs at Applying Computer Settings

Sample customer experiences + Bemis SOX consolidation

 

Case #

Symptom

User Action

110012949195052 / Bemis 2019683Windows Server 2008 R2 computer hangs at "applying computer settings. Cause reported as http.sys not releasing lock after certificates are added to web site. Only way users could not on was to boot with network cable disconnected. Case claums that W2K8 R2 computers are impacted by this issue even though W7 bug 169415 claims the problem was fixed in W2K8 R2.

AddedCryptsvc toto DependOnService in HKLM\CurrentControlSet\Services\HTTP" section of registry.

110043051978529 / Bemis 2159303

Windows Server 2008 computer hangs for 30 minutes while “applying computer settings. No repro when booting save mode with networking. Clean boot doesn’t resolve problem.


 None

110060456174145 / Bemis 2196875

Windows Server 2008 R2 Enterprise server hangs while applying computer settings and takes a long time to load the user desktop. The HTTP service will not start and service start requests will time out because there is a lock on the database.

AddedCryptsvc to DependOnService in HKLM\CurrentControlSet\Services\HTTP" section of registry.

110052866557854 / Bemis 2159616

Microsoft Exchange 2007 server hangs while “applying computer settings”.

Configured Cryptsvc as DependOnService under HKLM\CurrentControlSet\Services\HTTP" section of registry.

110071374261017Windows Server 2008 R2 VM guest (or is it W2K8 x64?) running OCS + web server hangs for 2.5 hours while "applying computer settings" during OS startup. Group policy fails to apply. Boot time is normal in "safe mode" and "safe mode with networking".

PSS follows steps in 2004121 even though customer says those steps had no impact (perhaps because a previous version of the article had a bad registry path).

Windows 2008 (Standard) x64 – Services not starting thread on
Social.technet.microsoft.com

Windows Server 2008 Standard x64 is slow to start. Numerous services including DHCP client, TCP/IP NetBIOS helper and Event Viewer fail to start or their service status was listed as “starting” in services.msc when current service status should have been “running”. Server Manager displays error 0x80080005 when trying to expand Roles and Features in Server Manager.

Original poster spent 3 days troubleshooting problem before rebuilding server over 10 hours.

Other posters report working around the problem by:

Disconnecting the network cable and rebooting to gain console access to the affected computer

·        deleting personal certificates used by IIS

·        using nets http context to delete the *:443 sslbinding + reboot

·        dumping all certificates from the Personal  -> Certificates container + reboot.

·        Deleting registry keys under hklm\system\ccs\services\http\parameters\sslbindinginfo (after 1st taking a backup of same)

·        Deleting *.domain.com certificate with expired root certificate.

 <add your case # + unique symptom + unique user action>

 

 



0:000> vertarget
Windows Server 2008/Windows Vista Version 6002 (Service Pack 2) MP (4 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
kernel32.dll version: 6.0.6002.18005 (lh_sp2rtm.090410-1830)
Machine Name:
Debug session time: Wed Oct  7 15:18:41.000 2009 (GMT+6)
System Uptime: 0 days 1:32:38.843
Process Uptime: 0 days 1:32:08.000
  Kernel time: 0 days 0:00:00.000
  User time: 0 days 0:00:00.000

0:000> !locks

CritSec services!ScStartImageCriticalSection+0 at 00000000ff525b20
WaiterWoken        No
LockCount          0
RecursionCount     1
OwningThread       2d4
EntryCount         0
ContentionCount    0
*** Locked

CritSec services!ScServiceStartCriticalSection+0 at 00000000ff525af0
WaiterWoken        No
LockCount          8
RecursionCount     1
OwningThread       2d4
EntryCount         0
ContentionCount    8
*** Locked

There are several threads (8) waiting for the critical section ScServiceStartCriticalSection.
This is owned by thread ID 0x2d4

.  0  Id: 2d0.2d4 Suspend: 0 Teb: 000007ff`fffde000 Unfrozen

0:000> ~0s
ntdll!ZwLoadDriver+0xa:
00000000`77cb7ada c3              ret
0:000> k
Child-SP          RetAddr           Call Site
00000000`0029f6f8 00000000`ff4db36a ntdll!ZwLoadDriver+0xa
00000000`0029f700 00000000`ff4d569d services!ScLoadDeviceDriver+0xaa
00000000`0029f750 00000000`ff4d516c services!ScStartService+0x1bc
00000000`0029f820 00000000`ff4db140 services!ScStartMarkedServices+0x1fd
00000000`0029f8b0 00000000`ff4e2722 services!ScStartServiceAndDependencies+0x3aa
00000000`0029f940 00000000`ff4d9219 services!ScAutoStartServices+0x225
00000000`0029f980 00000000`ff4e0919 services!SvcctrlMain+0xa75
00000000`0029fa10 00000000`ff4e08d6 services!main+0x31
00000000`0029fa40 00000000`77b5be3d services!__mainCRTStartup+0x13d
00000000`0029fa80 00000000`77c96a51 kernel32!BaseThreadInitThunk+0xd
00000000`0029fab0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

The thread is trying to start a service and for that it needs to load a device driver.
Check what the service is and the driver is:

The service is HTTP.sys service
0:000> dt services!service_record  00000000`00ab9300
   +0x000 Prev             : 0x00000000`00ab9220 _SERVICE_RECORD
   +0x008 ServiceName      : 0x00000000`00ab93c0  "HTTP"
   +0x010 DisplayName      : 0x00000000`00ab93c0  "HTTP"

This is the path passed to ZwLoadDriver
0:000> dt ntdll!_unicode_string 00000000`0029f700+30
 "\Registry\Machine\System\CurrentControlSet\Services\HTTP"
   +0x000 Length           : 0x70
   +0x002 MaximumLength    : 0x72
   +0x008 Buffer           : 0x00000000`003d74c0  "\Registry\Machine\System\CurrentControlSet\Services\HTTP"

Steps to reproduce.

Product Bug Number: Windows 7 Bug 169415, WINSE 308851, WINSE 348548
Author ID (email alias): prakashg/tode
Writer ID(email alias): cchameed/timothyn
Tech Review ID (email alias): timothyn
Confirm Article has been Tech Reviewed: Yes
Confirm Article released for Publishing: Yes
Ιδιότητες

Αναγνωριστικό άρθρου: 2004121 - Τελευταία αναθεώρηση: 21 Ιουν 2014 - Αναθεώρηση: 1

Σχόλια