ISA Server 2006 or Forefront TMG 2010 resets the connection to the SMTP client when the SMTP server closes its connection to ISA Server

Symptoms

Consider the following scenario:

• In Microsoft Internet Security and Acceleration (ISA) Server 2006 or Forefront TMG 2010, you enable SMTP filtering.

• An SMTP client communicates with an SMTP server through ISA Server 2006 or Forefront TMG 2010.

• The SMTP client uses one of the following verbs in its conversation with the SMTP server:

  • TLS

  • STARTTLS

  • X-AnonymousTLS

In this scenario, ISA Server 2006 or Forefront TMG 2010 resets the connection to the SMTP client when the SMTP server closes its connection to ISA Server or Forefront TMG 2010 with a TCP FIN packet. This behavior can cause some specific SMTP client applications to report that message delivery failed even though messages are sent correctly.

In ISA logging, you see the following result code for this connection:

0x80074e24 FWX_E_CONNECTION_KILLED

Cause

By design, the connection to the SMTP client is reset when the SMTP server closes its connection to ISA Server or Forefront TMG 2010. This is because ISA Server or Forefront TMG 2010 cannot inspect the SMTP traffic for the SMTP QUIT command when the traffic is encrypted. However, this behavior can cause specific SMTP client applications to report errors.

Resolution

A hotfix is now available to avoid incorrect error reports. This hotfix allows the default behavior to be changed so that ISA Server returns a TCP FIN packet to the SMTP client instead of a RESET packet when the SMTP server closes the TCP connection with a TCP FIN packet. To change the default behavior after you install this hotfix, you have to run a script on the ISA Server computer.

To apply this hotfix, follow these steps:

1. Download and then install the hotfix package that is mentioned in the following Microsoft Knowledge Base article:
   

   959357 Description of the ISA Server 2006 hotfix package: October 29, 2008
   
   Note: If you are running Forefront TMG 2010, copy the script from step 2 into a Notepad file and save it. Open command prompt with elevated privilege and then run the script (ie: cscript smtp.vbs).

2. Start Notepad, and then copy the following script into a Notepad file.

   Const SE_VPS_GUID = "{143F5698-103B-12D4-FF34-1F34767DEabc}"
   Const SE_VPS_NAME = "GracefulPassthroughServerShutdown"
   Const SE_VPS_VALUE = true
   
   Sub SetValue()
   
       ' Create the root obect.
       Dim root  ' The FPCLib.FPC root object
       Set root = CreateObject("FPC.Root")
   
       'Declare the other objects needed.
       Dim array       ' An FPCArray object
       Dim VendorSets  ' An FPCVendorParametersSets collection
       Dim VendorSet   ' An FPCVendorParametersSet object
   
       ' Get references to the array object
       ' and the network rules collection.
       Set array = root.GetContainingArray
       Set VendorSets = array.VendorParametersSets
   
       On Error Resume Next
       Set VendorSet = VendorSets.Item( SE_VPS_GUID )
   
       If Err.Number <> 0 Then
           Err.Clear
   
           ' Add the item
           Set VendorSet = VendorSets.Add( SE_VPS_GUID )
           CheckError
           WScript.Echo "New VendorSet added... " & VendorSet.Name
   
       Else
           WScript.Echo "Existing VendorSet found... value- " &  VendorSet.Value(SE_VPS_NAME)
       End If
   
       if VendorSet.Value(SE_VPS_NAME) <> SE_VPS_VALUE Then
   
           Err.Clear
           VendorSet.Value(SE_VPS_NAME) = SE_VPS_VALUE
   
           If Err.Number <> 0 Then
               CheckError
           Else
               VendorSets.Save false, true
               CheckError
   
               If Err.Number = 0 Then
                   WScript.Echo "Done with " & SE_VPS_NAME & ", saved!"
               End If
           End If
       Else
           WScript.Echo "Done with " & SE_VPS_NAME & ", no change!"
       End If
   
   End Sub
   
   Sub CheckError()
   
       If Err.Number <> 0 Then
           WScript.Echo "An error occurred: 0x" & Hex(Err.Number) & " " & Err.Description
           Err.Clear
       End If
   
   End Sub
   
   SetValue
   

3. Save the file as a Microsoft Visual Basic script file by using the .vbs file name extension.

4. Double-click the .vbs file to run the script.

Note If you want to revert to the default behavior, change the following line of the script:

Const SE_VPS_VALUE = true

Change this line to the following:

Const SE_VPS_VALUE = false

Then, rerun the script on the ISA Server or Forefront TMG computer.

Status

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

References

For more information about software update terminology, click the following article number to view the article in the Microsoft Knowledge Base:

824684 Description of the standard terminology that is used to describe Microsoft software updates