In this scenario, the SMTP traffic between the Edge Transport server and the internal Hub Transport server may still be blocked even though you have added the X-AnonymousTLS verb and the X-EXPs verb to the SMTP filter settings in ISA Server.
This problem occurs when Exchange 2007 uses the proprietary verb X-AnonymousTLS to switch to Transport Layer Security (TLS) encryption. The SMTP filter in ISA Server 2006 is not aware of the usage of this verb. Therefore, the SMTP filter inspects the traffic, even though it is encrypted. Intermittantly, the SMTP filter detects malformed traffic and ends the session.
To resolve this problem, apply the hotfix that is mentioned in the following Microsoft Knowledge Base article:
959357 Description of the ISA Server 2006 hotfix package: October 29, 2008
Note You must still manually add the X-AnonymousTLS verb and the X-EXPs verb to the SMTP filter settings in ISA Server. For more information, visit the following Microsoft TechNet Web site:
Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.
After you apply this hotfix, the SMTP filter in ISA Server 2006 uses passthrough mode for the X-AnonymousTLS verb, and the filter does not inspect traffic. This is identical to how the TLS verb and the STARTTLS verb are treated.
For more information about software update terminology, click the following article number to view the article in the Microsoft Knowledge Base:
824684 Description of the standard terminology that is used to describe Microsoft software updates