FIX: A nonweb proxy client in a Forefront Threat Management Gateway (TMG) 2010 environment cannot open certain load-balanced websites when TMG HTTPS inspection is enabled

Symptoms

Consider the following scenario:
  • You use a client that is not a web proxy client such as a firewall client or a SecureNAT client on your corporate network.
  • The client tries to connect to an HTTPS website through a server that is running Microsoft Forefront Threat Management Gateway (TMG) 2010. For example, the client tries to connect to https://contoso.com.
  • HTTPS inspection is enabled on the server that is running TMG 2010.
  • The HTTPS website uses a special Domain Name System (DNS)-based load-balancing algorithm in which the responsible DNS server returns an alternating IP address that has a low "Time to Live" value. In this case, successive queries for the website (contoso.com) result in different IP addresses.

In this scenario, the client may be unable to browse the website. Additionally, the following may be logged in the Application log on the server that is running TMG 2010:
Status 12227 The name on the SSL server certificate supplied by a destination server does not match the name of the host requested.

Cause

This issue occurs because of the special DNS-based load-balancing algorithm.

When a nonweb proxy client opens a website such as https://contoso.com, the client resolves the name itself and tries to establish a Secure Sockets Layer (SSL) connection to the destination IP address such as, for example, IP-1.

When HTTPS inspection is enabled, TMG 2010 establishes a connection on behalf of the client and tries to validate the server certificate before the client connection is enabled. One of the many checks that are performed (for example, validity and revocation) verifies that the correct website is being connected.

The verification is performed as follows:
  1. TMG 2010 reads the Subject Name and Subject Alternate Name fields of the retrieved certificate, such as Contoso.com in this case.
  2. TMG 2010 tries to resolve the certificate by using DNS.
  3. TMG 2010 checks whether the result matches the destination IP address that the client used when the connection was established.

Because of how load-balancing is resolved for the website, name resolution in TMG 2010 yields a different IP address, IP-2. Therefore, because the two IP addresses are not the same (IP-1 versus IP-2), TMG 2010 denies the connection.

Resolution

To resolve this problem, install the hotfix package that is described in the following Microsoft Knowledge Base article:
2735208 Rollup 3 for Forefront Threat Management Gateway (TMG) 2010 Service Pack 2
Note After you install this fix, you can configure an exclusion Domain Name Set for HTTPS-inspection IP address validation. These are domain names for which the address validation part of the HTTPS-inspection is suppressed. However, other validation steps are still performed.

The following script configures the exclusion Domain Name Set to be the one that is named in the beginning of the script. The script also creates the exclusion Domain Name Set if it does not already exist. By using the regular TMG management tools such as the administrative console and scripting, the administrator can populate the DomainNameSet variable as appropriate.


' The domain name set for the exclusion list
const strDomainNameSetName = _
"HTTPS-inspection IP address validation exception"
const strDomainNameSetDescription = "HTTPS sites whose certificate's domain-name matches this set are excluded from the validation that the domain-name indeed resolves to the IP address that the Web client originally connected to"


Const strVpsGUID = "{143F5698-103B-12D4-FF34-1F34767DEABC}"
Const strVpsPropertyName = "HTTPSiIpAddressValidationExclusionDNSet"
Const Error_FileNotFound = &H80070002

Set objArray = CreateObject("FPC.Root").GetContainingArray()
Set objDNSet = OpenDNSet( _
objArray.RuleElements.DomainNameSets, _
strDomainNameSetName, _
strDomainNameSetDescription _
)
Set objVPSet = OpenVPSet(objArray.ArrayPolicy, strVpsGUID)
objVPSet.Value(strVpsPropertyName) = objDNSet.PersistentName
objArray.Save

function OpenDNSet(objDNSets, strDNSetName, strDNSetDescription)
On Error Resume Next
Set objDNSet = objDNSets.Item(strDNSetName)
' Save the Err properties in case it needs to be re-raised
errNumber = Err.Number
errSource = Err.Source
errDescription = Err.Description
errHelpFile = Err.HelpFile
errHelpContext = Err.HelpContext

On Error GoTo 0

if errNumber = Error_FileNotFound Then
Set objDNSet = objDNSets.Add(strDNSetName)
objDNSet.Description = strDNSetDescription
Elseif errNumber < 0 Then
' An error other than "file not found" occured -- re-raise the error,
' this time not under "On Error Resume Next"
Err.Raise errNumber, errSource, errDescription, errHelpFile, errHelpContext
End If

Set OpenDNSet = objDNSet
end function

function OpenVPSet(objParent, strVpsGUID)
Set objVPSets = objParent.VendorParametersSets
On Error Resume Next
Set OpenVPSet = objVPSets.Item(strVpsGUID)
' Save the Err properties in case it needs to be re-raised
errNumber = Err.Number
errSource = Err.Source
errDescription = Err.Description
errHelpFile = Err.HelpFile
errHelpContext = Err.HelpContext

On Error GoTo 0

if errNumber = Error_FileNotFound Then
Set OpenVPSet = objVPSets.Add(strVpsGUID)
Elseif errNumber < 0 Then
' An error other than "file not found" occured -- re-raise the error,
' this time not under "On Error Resume Next"
Err.Raise errNumber, errSource, errDescription, errHelpFile, errHelpContext
End If
end function

Workaround

To work around this issue, configure the client to act as a web proxy client. In this case, name resolution occurs only on the TMG server. Or, exclude the affected client from HTTPS inspection, or exclude the problematic destination website, as described on the following Microsoft TechNet website:

Status

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

References

For more information about software update terminology, click the following article number to view the article in the Microsoft Knowledge Base:
824684 Description of the standard terminology that is used to describe Microsoft software updates

Properties

Article ID: 2780594 - Last Review: Jan 10, 2013 - Revision: 1

Feedback