ISA Server 2006 or Forefront TMG 2010 resets the connection to the SMTP client when the SMTP server closes its connection to ISA Server

Article translations Article translations
Article ID: 959312 - View products that this article applies to.
Expand all | Collapse all

SYMPTOMS

Consider the following scenario:
  • In Microsoft Internet Security and Acceleration (ISA) Server 2006 or Forefront TMG 2010, you enable SMTP filtering.
  • An SMTP client communicates with an SMTP server through ISA Server 2006 or Forefront TMG 2010.
  • The SMTP client uses one of the following verbs in its conversation with the SMTP server:
    • TLS
    • STARTTLS
    • X-AnonymousTLS
In this scenario, ISA Server 2006 or Forefront TMG 2010 resets the connection to the SMTP client when the SMTP server closes its connection to ISA Server or Forefront TMG 2010 with a TCP FIN packet. This behavior can cause some specific SMTP client applications to report that message delivery failed even though messages are sent correctly.

In ISA logging, you see the following result code for this connection:
0x80074e24 FWX_E_CONNECTION_KILLED

CAUSE

By design, the connection to the SMTP client is reset when the SMTP server closes its connection to ISA Server or Forefront TMG 2010. This is because ISA Server or Forefront TMG 2010 cannot inspect the SMTP traffic for the SMTP QUIT command when the traffic is encrypted. However, this behavior can cause specific SMTP client applications to report errors.

RESOLUTION

A hotfix is now available to avoid incorrect error reports. This hotfix allows the default behavior to be changed so that ISA Server returns a TCP FIN packet to the SMTP client instead of a RESET packet when the SMTP server closes the TCP connection with a TCP FIN packet. To change the default behavior after you install this hotfix, you have to run a script on the ISA Server computer.

To apply this hotfix, follow these steps:
  1. Download and then install the hotfix package that is mentioned in the following Microsoft Knowledge Base article:
    959357 Description of the ISA Server 2006 hotfix package: October 29, 2008
    Note: If you are running Forefront TMG 2010, copy the script from step 2 into a Notepad file and save it. Open command prompt with elevated privilege and then run the script (ie: cscript smtp.vbs).
  2. Start Notepad, and then copy the following script into a Notepad file.
    Const SE_VPS_GUID = "{143F5698-103B-12D4-FF34-1F34767DEabc}"
    Const SE_VPS_NAME = "GracefulPassthroughServerShutdown"
    Const SE_VPS_VALUE = true
    
    Sub SetValue()
    
        ' Create the root obect.
        Dim root  ' The FPCLib.FPC root object
        Set root = CreateObject("FPC.Root")
    
        'Declare the other objects needed.
        Dim array       ' An FPCArray object
        Dim VendorSets  ' An FPCVendorParametersSets collection
        Dim VendorSet   ' An FPCVendorParametersSet object
    
        ' Get references to the array object
        ' and the network rules collection.
        Set array = root.GetContainingArray
        Set VendorSets = array.VendorParametersSets
    
        On Error Resume Next
        Set VendorSet = VendorSets.Item( SE_VPS_GUID )
    
        If Err.Number <> 0 Then
            Err.Clear
    
            ' Add the item
            Set VendorSet = VendorSets.Add( SE_VPS_GUID )
            CheckError
            WScript.Echo "New VendorSet added... " & VendorSet.Name
    
        Else
            WScript.Echo "Existing VendorSet found... value- " &  VendorSet.Value(SE_VPS_NAME)
        End If
    
        if VendorSet.Value(SE_VPS_NAME) <> SE_VPS_VALUE Then
    
            Err.Clear
            VendorSet.Value(SE_VPS_NAME) = SE_VPS_VALUE
    
            If Err.Number <> 0 Then
                CheckError
            Else
                VendorSets.Save false, true
                CheckError
    
                If Err.Number = 0 Then
                    WScript.Echo "Done with " & SE_VPS_NAME & ", saved!"
                End If
            End If
        Else
            WScript.Echo "Done with " & SE_VPS_NAME & ", no change!"
        End If
    
    End Sub
    
    Sub CheckError()
    
        If Err.Number <> 0 Then
            WScript.Echo "An error occurred: 0x" & Hex(Err.Number) & " " & Err.Description
            Err.Clear
        End If
    
    End Sub
    
    SetValue
    
  3. Save the file as a Microsoft Visual Basic script file by using the .vbs file name extension.
  4. Double-click the .vbs file to run the script.
Note If you want to revert to the default behavior, change the following line of the script:
Const SE_VPS_VALUE = true

Change this line to the following:
Const SE_VPS_VALUE = false

Then, rerun the script on the ISA Server or Forefront TMG computer.

STATUS

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

REFERENCES

For more information about software update terminology, click the following article number to view the article in the Microsoft Knowledge Base:
824684 Description of the standard terminology that is used to describe Microsoft software updates

Properties

Article ID: 959312 - Last Review: February 12, 2011 - Revision: 3.0
APPLIES TO
  • Microsoft Internet Security and Acceleration Server 2006 Service Pack 1, when used with:
    • Microsoft Internet Security and Acceleration Server 2006 Standard Edition
    • Microsoft Internet Security and Acceleration Server 2006 Enterprise Edition
  • Microsoft Forefront Threat Management Gateway 2010 Enterprise
  • Microsoft Forefront Threat Management Gateway 2010 Service Pack 1
  • Microsoft Forefront Threat Management Gateway 2010 Standard
Keywords: 
kbexpertiseinter kbfix kbsurveynew kbqfe KB959312

Give Feedback

 

Contact us for more help

Contact us for more help
Connect with Answer Desk for expert help.
Get more support from smallbusiness.support.microsoft.com