Authentication dialog box appears when the DRMAcquireLicense API is executed

Gælder for: Windows Server 2012 DatacenterWindows Server 2012 DatacenterWindows Server 2012 Standard

Symptoms


Consider the following scenario:
  • You have a Windows system that is hosting Internet Information Services (IIS).
  • IIS is deploying the Rights Management Service (RMS).
  • The DRMAcquireLicense API is executed immediately after one of the following actions occurs:
    • IIS is restarted.
    • The Windows system is restarted.
In this scenario, the authentication dialog box appears.

Cause


By default, to improve the performance of authentication-related actions, Windows authentication in IIS has Kernel Mode Authentication enabled. However, when the DRMAcquireLicense API accesses the following site by using NTLM pre-authentication, the authentication fails:
https://[servername]/_wmcs/licensing/
This occurs because Kernel Mode Authentication does not accept NTLM pre-authentication. Therefore, the DRMAcquireLicense API causes the authentication dialog box to appear.

Resolution


To prevent the dialog box from appearing, following these steps:
  1. Under the Licensing site, start IIS Manager.
  2. Under _wmcs, click the licensing site
  3. Under the features view, select Authentication.
  4. Right-click Windows Authentication, and then click Advanced settings.
  5. On the Advanced Settings window, clear the Enable Kernel Mode Authentication check box, and then click OK.
  6. Right-click Windows Authentication, and then click Provider.
  7. In the Provider window, delete Negotiate, and then click OK.

More Information


The DRMAcquireLicense API accesses the site /_wmcs/licensing site in the IIS server that is hosting RMS in order to obtain the license. If the Service Principal Names (SPN) cannot be found, DRMAcquireLicense tries to authenticate by using NTLM Pre-Authentication.

Because NTLM Pre-Authentication cannot be used for Kernel Mode Authentication, IIS returns HTTP_STATUS_DENIED to the client. Therefore, the authentication dialog box appears in DRMAcquireLicense.

Note When you complete the authentication through the dialog box, the dialog box will not reappear. Instead, the client and IIS will use NTLM authentication afterward.