Select the product you need help with
- Internet Explorer
- Windows Phone
- More products
How to troubleshoot a memory leak or an out-of-memory exception in the BizTalk Server process
Article ID: 918643 - View products that this article applies to.
Memory leaks are a common issue. You may have to try several steps to find the specific cause of a memory leak or an out-of-memory (OOM) exception in Microsoft BizTalk Server. This article discusses important things to consider when you are evaluating memory usage and possible memory-related issues. These considerations include the following:
This article describes how to troubleshoot a memory leak or an out-of-memory exception in the BizTalk Server process of Microsoft BizTalk Server.
The BizTalk Server process may be experiencing a memory leak when memory usage in Microsoft Windows Task Manager consumes more than 50 percent of the physical RAM. A memory leak may cause an out-of-memory exception when memory usage increases until the process runs out of system memory or until the process stops functioning.
When this problem occurs, a warning message that resembles the following message is logged in the event log:
Event Type: Warning
Physical RAM and memory usageBecause it may be expected behavior for a process to use about half the physical RAM, use the memory usage as a guideline. For example, if the BizTalk Server has 4 gigabytes (GB) of RAM, and the BizTalk Server process uses about 500 megabytes (MB) of RAM, there may not be leak. If the BizTalk Server process uses about 1 GB of RAM, there might be a memory leak or a high memory situation. The memory consumption may be caused by a long-running stored procedure or orchestration. Make sure that you know how much memory the BizTalk host typically uses to determine whether a memory leak or high memory condition is occurring.
Large messagesWhen BizTalk Server processes large messages, the system seems to have a memory leak. However, the messages may be using a large amount of memory. For more information about large messages, visit the following Microsoft Developer Network (MSDN) websites:
http://msdn.microsoft.com/en-us/library/aa560481(BTS.10).aspxAlso, consider that high memory usage may be expected if BizTalk Server is processing large messages. You may want to upgrade your hardware to meet the performance requirements of BizTalk Server in your environment.
How long it takes to reproduce the memory leakMemory leaks can occur immediately or they may accumulate over time. Both scenarios are common.
Use of the /3GB switch on 32-bit computersTypically, a process can access 2 GB of virtual address space. The /3GB switch is an option for systems that require more addressable memory. This option may improve memory usage for processing messages. However, the /3GB switch allows for only 1 GB of addressable memory for kernel mode operations. Additionally, this switch may increase the risk of running out of pool memory.
For more information about the /3GB switch, visit the following Microsoft Developer Network (MSDN) website:
http://msdn.microsoft.com/en-us/library/ms791558.aspxWhen the /3GB switch is enabled on a 32-bit version of Windows, the process can access 3 GB of virtual address space if the process is large-address aware. A process is large-address aware when the executable has the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the image header. Because the BizTalk process is large-address aware, BizTalk will benefit from the /3GB switch.
If a 32-bit BizTalk host instance is running on a 64-bit version of Windows (AMD64), the BizTalk process benefits from the 4 GB memory address space because BizTalk is large-address aware. Therefore, moving your high memory applications to a 64-bit server may be the best solution.
A 64-bit BizTalk process on a 64-bit version of Windows (AMD64) has 8 TB of addressable memory.
You should also consider the virtual bytes and the private bytes used by the process. A BizTalk host instance (which is a .NET Framework application) may receive an out of memory error before the Virtual Bytes value reaches 2 GB. This can occur even though the maximum memory addressable by a process on a 32-bit version of Windows (without the /3GB switch) is 2 GB. For an explanation of why this can occur, visit the following Microsoft Developer Network (MSDN) websites:
http://blogs.msdn.com/tess/archive/2005/11/25/496898.aspxThe /3GB switch also increases the maximum private bytes of the BizTalk process from 800 MB to 1800 MB. For more information about .NET Framework application performance with the /3GB switch enabled, visit the following Microsoft Developer Network (MSDN) website:
http://msdn2.microsoft.com/en-us/library/ms998583.aspxThe following table summarizes this information and includes the practical limits for virtual bytes and private bytes.
For more information about the addressable memory for 32-bit vs. 64-bit Windows, visit the following Microsoft Developer Network (MSDN) website:
Collapse this tableExpand this table
http://msdn.microsoft.com/en-us/library/aa366778.aspxThe following table lists PAE and 3GB supportability for different versions of BizTalk Server.
If you must enable the /3GB switch to meet the performance requirements of a computer that is running BizTalk Server, you may want to consider adding servers to the BizTalk group. This enables you to scale out the memory-intensive host instances.
Collapse this tableExpand this table
BizTalk components that run inside an Internet Information Services (IIS) process may also benefit when the /3GB switch is enabled.
The /3GB switch is not supported on computers that are running Windows SharePoint Services 2.0 or later versions or SharePoint Portal Server 2003 SP2 or later versions. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
(http://support.microsoft.com/kb/933560/ )The Windows Server 2003 /3GB switch is not supported in Windows SharePoint Services 2.0 or in later versions or in SharePoint Portal Server 2003 Service Pack 2 or in later versions
Use of custom componentsIf you use custom components, such as pipelines or service components, you must know what these components do. You must also know the potential effect of these components on memory usage. A common memory problem occurs when a component is transforming a document. The transform operation is a memory-intensive operation. When a document is transformed, BizTalk Server passes the message stream to the Microsoft .NET Framework XslTransform class within the BizTalk process.
Another common issue occurs when there is intensive string manipulation. Intensive string manipulation can consume lots of memory. For more information about ways to improve performance, visit the following Microsoft Developer Network (MSDN) website:
Version of the .NET FrameworkThe Microsoft .NET Framework 2.0 and the .NET Framework 1.1 have different memory behavior. Therefore, you may see varying results between them. If you are using the .NET Framework, confirm that the latest .NET Framework Service Pack 1 is installed. These service packs address several known memory issues. For more information, click the following article numbers:
(http://support.microsoft.com/default.aspx?scid=kb;EN-US;945757)Problems that are fixed in the .NET Framework 2.0 Service Pack 1
(http://support.microsoft.com/kb/867460/ )List of bugs that are fixed in the .NET Framework 1.1 Service Pack 1
Number of processorsThe common language runtime (CLR) has the following garbage collectors (GCs):
If the computer that is running BizTalk Server is a single processor system, the .NET Framework uses the Workstation version of the execution engine. This is the default behavior. The Workstation garbage collector allocation algorithm is not designed for scaling or for maximum throughput. This garbage collector uses concurrent garbage collector methods. These methods are designed for applications that have complex user interfaces. Such applications may require more aggressive garbage collection.
Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
322756Sometimes, it may be appropriate to run the Workstation version of the execution engine on a multiprocessor system. You can use the following registry key to switch to the Workstation version of the execution engine.
(http://support.microsoft.com/kb/322756/ )How to back up and restore the registry in Windows
BizTalk 2006 and later versionsCreate the following CRL Hosting String registry key with the corresponding values:
BizTalk 2004Create the following CRL Hosting String registry key with the corresponding values:
For more information, visit the following Microsoft Developer Network (MSDN) websites:
Common causes and resolutions
Process memory usage and Physical memory usage throttling thresholdsThe Process memory usage and Physical memory usage throttling thresholds can be changed in BizTalk Server 2006 and in later versions.
Dehydration throttling thresholdsThe default memory dehydration thresholds may cause too much dehydration when orchestrations are run on a 64-bit host. For more information about this issue, see the Dehydration Default Properties topic on the following Microsoft Developer Network (MSDN) website:
http://msdn.microsoft.com/en-us/library/aa560586.aspxNote 64-bit hosts are supported in BizTalk Server 2006 and later versions.
On equivalent hardware in a 32-bit host instance, observed dehydration is nominal when the same orchestrations are run by using the default memory dehydration throttling thresholds.
Because 64-bit architecture provides expanded memory address space (16 TB instead of 4 GB), 64-bit host instances are allocated significantly more memory than 32-bit host instances. This can cause the default memory throttling thresholds to be exceeded.
To work around this behavior, change the VirtualMemoryThrottlingCriteria and PrivateMemoryThrottlingCriteria values in the BTSNTSvc64.exe.config file. Use the Process\Virtual Bytes and the Process\Private Bytes Performance Monitor counters to determine the largest amount of memory that is being allocated by an orchestration instance.
If the \Process\Private Bytes Performance Monitor counter value is 435689400 bytes (415 MB), set the OptimalUsage value for PrivateMemoryThrottlingCriteria to 457 MB (435689400 * 1.10 = 479258340 bytes). Set the MaximalUsage value for PrivateMemoryThrottlingCriteria to 594 MB (479258340 * 1.30 = 623035842).
For this example, the following values would be specified in the BTSNTSvc64.exe.config file to reduce throttling.
These values would then be represented in the BTSNTSvc64.exe.config file as follows:
Collapse this tableExpand this table
To determine which host instance is running the orchestration, you can match the ID Process from the \BizTalk:Messaging\ID Process and \Process\ID Process Performance Monitor counters. Then, check the Average value displayed for the corresponding \Process\Virtual Bytes and \Process\Private Bytes Performance Monitor counters.
<xlangs> <Configuration> <Dehydration> <VirtualMemoryThrottlingCriteria OptimalUsage="6069" MaximalUsage="7889" IsActive="true" /> <PrivateMemoryThrottlingCriteria OptimalUsage="457" MaximalUsage="594" IsActive="true" /> </Dehydration> </Configuration> </xlangs>
Note The high dehydration may cause a significant decrease in performance when the BizTalkMsgBoxDb database is running on SQL Server 2008.
BizTalk Server Service Packs and Cumulative UpdatesBizTalk Server service packs and cumulative updates include the latest fixes. These include those that affect known System.OutOfMemoryException issues.
(http://support.microsoft.com/default.aspx?scid=kb;en-US;2281783)Service Pack and Cumulative Update list for BizTalk Server 2006 R2
Microsoft BizTalk Server 2004 Service Pack 2
HeapDeCommitFreeBlockThresholdBy default, theHeapDeCommitFreeBlockThreshold registry key value is 0. A value of 0 means that the heap manager decommits each 4-kilobyte (KB) page that becomes available. Decommit operations can cause virtual memory fragmentation. The size of the HeapDeCommitFreeBlockThreshold setting in the heap manager will depend on the kind of work that the system is doing. A size of 0x00040000 is a recommended starting value.
Consider the following information before you change the value of the
For more information about the HeapDeCommitFreeBlockThreshold registry key, click the following article number to view the article in the Microsoft Knowledge Base:
Value name: HeapDeCommitFreeBlockThreshold
Value type: REG_DWORD
Value data: 0x00040000 (This is the recommended starting value.)
Value default: not present
(http://support.microsoft.com/kb/315407/ )The "HeapDecommitFreeBlockThreshold" registry key
Transform operationsWhen BizTalk Server performs XML transform operations on fairly large messages in a receive port, in a send port, or in XLANG, XSL transforms load the whole message in memory..
To resolve this issue, use one of the following methods:
Most of the default BizTalk functoids are implemented as inline script. These items can cause System.Byte objects to collect in memory. To minimize memory consumption, we recommend that you put any map that uses these functoids into a small assembly. Then, reference that assembly. Use the following chart to determine which functoids use inline script and which functoids do not use inline script.
In the second column, “Yes” means that this functoid is implemented as inline script, and that it will cause System.Byte objects to collect in memory. “No” means that this functoid is not implemented as inline script, and that it will not cause System.Byte objects to collect in memory.
BizTalk Server 2006 and later versions significantly improve memory management for large documents. To do this, BizTalk Server implements a configurable message size threshold for loading documents into memory during transform operations. The default message size threshold is 1 MB. For more information about the TransformThreshold setting, visit the following Microsoft Developer Network (MSDN) website:
Collapse this tableExpand this table
Large attribute values and large element valuesWhen BizTalk Server executes a receive pipeline or a send pipeline on an XML document, the payload is processed in memory if the document contains one or more of the following entities:
Custom pipeline componentsYou are using a custom pipeline component that loads the whole stream into memory. All the components that are included with BizTalk Server, except transforms, support streaming. These components do not use as much memory during streaming. However, custom pipeline components may not support streaming.
Streaming under heavy stressSend hosts run out of memory when they operate under heavy stress. BizTalk Server send pipelines and send adapters support streaming. In streaming, each component loads a small fragment of the stream into memory. Because each message includes other data structures, together with a message context that can be big or small, this behavior affects the behavior of BizTalk Server under heavy stress.
The behavior of BizTalk Server is affected because the engine loads a preconfigured number of messages. The number of messages that the engine loads is based on the values that appear in the LowWaterMark field and the HighWaterMark field of the Adm_serviceClass table. The Adm_serviceClass table is in the BizTalk Management Database. These values control the number of messages that BizTalk Server processes or sends at the same time.
The HighWaterMark value is the total number of messages that the engine processes at the same time. The default value is 200 messages per CPU. Therefore, on an 8-processor server, the send host will try to process 1,600 messages (200*8) at the same time. If you assume that each message is 50 KB, the messages equal 80 MB (1,600*50=80,000KB).
To resolve this issue, you can change the HighWaterMark value and the LowWaterMark value in the database. The values that you use depend on the size of the messages.
For more information about common causes of an out-of-memory condition, see the "Memory Growth in BizTalk Messaging" section at the following Microsoft website:
http://blogs.msdn.com/biztalkperformanceFor BizTalk Server 2006 and later versions, you can change the default host throttling settings. For more information about how to change the default host throttling settings, visit the following Microsoft Developer Network (MSDN) website:
Try to simplify the issueIf you have identified a memory leak, try to determine the cause by removing custom components or by simplifying a map. Also, try to reproduce the issue by using a simple orchestration or a simple solution. Typically, you should create separate receive hosts for receive adapters. You should also create separate send hosts for send adapters. When you use this method, each adapter can run in a separate process. Therefore, if your BizTalk Server process experiences an out-of-memory condition, you will know which components are involved.
Troubleshooting stepsTo troubleshoot an out-of-memory condition, use the Debug Diagnostics tool to monitor memory allocations over time. The Debug Diagnostics tool can create and analyze a memory leak dump file (.dmp). When you troubleshoot memory leaks, the goal is to attach Leaktrack.dll before the high memory condition reproduces to capture memory growth over time. Leaktrack.dll is included with the Debug Diagnostics tool.
How to use Performance Monitor logging
Select the data to logTo select the data to log, use the method that is appropriate for your operating system:
Obtain the dump fileTo obtain the dump file, use one of the following methods:
Stop Performance Monitor loggingIf you are capturing a memory dump and Performance Monitor data, stop Performance Monitor logging about two minutes after the memory dump is created.
Analyze the dump fileTo help determine the cause of a memory leak, you can use the Debug Diagnostics tool to analyze the dump file. To do this, follow these steps:
If you use custom DLLs, you can add the symbol path of the custom .pdb files for analysis. To do this, follow these steps:
http://support.microsoft.com/contactus/?ws=supportBefore you contact Customer Support Services, compress the dump file, the Performance Monitor log, the analysis report file, and the updated event logs (.evt files). You may have to send these files to a BizTalk Server support engineer.
Article ID: 918643 - Last Review: June 13, 2012 - Revision: 12.0