- Physical RAM
- Large message processing
- Use of the /3GB switch
- Use of custom components
- Which version of the Microsoft .NET Framework the system is running
- The number of processors
When this problem occurs, a warning message that resembles the following message is logged in the event log:
Physical RAM and memory usageBecause it may be expected behavior for a process to use about half the physical RAM, use the memory usage as a guideline. For example, if the BizTalk Server has 4 gigabytes (GB) of RAM, and the BizTalk Server process uses about 500 megabytes (MB) of RAM, there may not be leak. If the BizTalk Server process uses about 1 GB of RAM, there might be a memory leak or a high memory situation. The memory consumption may be caused by a long-running stored procedure or orchestration. Make sure that you know how much memory the BizTalk host typically uses to determine whether a memory leak or high memory condition is occurring.
Large messagesWhen BizTalk Server processes large messages, the system seems to have a memory leak. However, the messages may be using a large amount of memory. For more information about large messages, visit the following Microsoft Developer Network (MSDN) websites:
How long it takes to reproduce the memory leakMemory leaks can occur immediately or they may accumulate over time. Both scenarios are common.
Use of the /3GB switch on 32-bit computersTypically, a process can access 2 GB of virtual address space. The /3GB switch is an option for systems that require more addressable memory. This option may improve memory usage for processing messages. However, the /3GB switch allows for only 1 GB of addressable memory for kernel mode operations. Additionally, this switch may increase the risk of running out of pool memory.
For more information about the /3GB switch, visit the following Microsoft Developer Network (MSDN) website: /3GB switch is enabled on a 32-bit version of Windows, the process can access 3 GB of virtual address space if the process is large-address aware. A process is large-address aware when the executable has the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the image header. Because the BizTalk process is large-address aware, BizTalk will benefit from the /3GB switch.
If a 32-bit BizTalk host instance is running on a 64-bit version of Windows (AMD64), the BizTalk process benefits from the 4 GB memory address space because BizTalk is large-address aware. Therefore, moving your high memory applications to a 64-bit server may be the best solution.
A 64-bit BizTalk process on a 64-bit version of Windows (AMD64) has 8 TB of addressable memory.
You should also consider the virtual bytes and the private bytes used by the process. A BizTalk host instance (which is a .NET Framework application) may receive an out of memory error before the Virtual Bytes value reaches 2 GB. This can occur even though the maximum memory addressable by a process on a 32-bit version of Windows (without the /3GB switch) is 2 GB. For an explanation of why this can occur, visit the following Microsoft Developer Network (MSDN) websites: /3GB switch also increases the maximum private bytes of the BizTalk process from 800 MB to 1800 MB. For more information about .NET Framework application performance with the /3GB switch enabled, visit the following Microsoft Developer Network (MSDN) website:
The following table summarizes this information and includes the practical limits for virtual bytes and private bytes.
|Process||Windows||Addressable memory (with a large address-aware process)||Practical limit for virtual bytes||Practical limit for private bytes|
|32-bit||32-bit||2 GB||1400 MB||800 MB|
|32-bit||32-bit with /3GB||3 GB||2400 MB||1800 MB|
|32-bit||64-bit||4 GB||3400 MB||2800 MB|
|64-bit||64-bit||8 TB||Not applicable||Not applicable|
|BizTalk Server 2004||Yes||No|
|BizTalk Server 2006||Yes||Yes|
|BizTalk Server 2006 R2||Yes||Yes|
|BizTalk Server 2009||Yes||Yes|
BizTalk components that run inside an Internet Information Services (IIS) process may also benefit when the /3GB switch is enabled.
The /3GB switch is not supported on computers that are running Windows SharePoint Services 2.0 or later versions or SharePoint Portal Server 2003 SP2 or later versions. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
Use of custom componentsIf you use custom components, such as pipelines or service components, you must know what these components do. You must also know the potential effect of these components on memory usage. A common memory problem occurs when a component is transforming a document. The transform operation is a memory-intensive operation. When a document is transformed, BizTalk Server passes the message stream to the Microsoft .NET Framework XslTransform class within the BizTalk process.
Another common issue occurs when there is intensive string manipulation. Intensive string manipulation can consume lots of memory. For more information about ways to improve performance, visit the following Microsoft Developer Network (MSDN) website:
Version of the .NET FrameworkThe Microsoft .NET Framework 2.0 and the .NET Framework 1.1 have different memory behavior. Therefore, you may see varying results between them. If you are using the .NET Framework, confirm that the latest .NET Framework Service Pack 1 is installed. These service packs address several known memory issues. For more information, click the following article numbers:
945757 Problems that are fixed in the .NET Framework 2.0 Service Pack 1
867460 List of bugs that are fixed in the .NET Framework 1.1 Service Pack 1
Number of processorsThe common language runtime (CLR) has the following garbage collectors (GCs):
- Workstation (Mscorwks.dll)
- Server (Mscorsvr.dll)
If the computer that is running BizTalk Server is a single processor system, the .NET Framework uses the Workstation version of the execution engine. This is the default behavior. The Workstation garbage collector allocation algorithm is not designed for scaling or for maximum throughput. This garbage collector uses concurrent garbage collector methods. These methods are designed for applications that have complex user interfaces. Such applications may require more aggressive garbage collection.
Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
BizTalk 2006 and later versionsCreate the following CRL Hosting String registry key with the corresponding values:
BizTalk 2004Create the following CRL Hosting String registry key with the corresponding values:
For more information, visit the following Microsoft Developer Network (MSDN) websites:
Common causes and resolutions
Process memory usage and Physical memory usage throttling thresholdsThe Process memory usage and Physical memory usage throttling thresholds can be changed in BizTalk Server 2006 and in later versions.
- By default, the Process memory usage throttling threshold is set to 25. If this value is exceeded and the BizTalk process memory usage is more than 300 MB, a throttling condition may occur. On a 32-bit server, you can increase the Process memory usage value to 50. On a 64-bit server, you can increase this value to 100. This allows for more memory consumption by the BizTalk process before throttling occurs.
- The Physical memory usage throttling threshold has a default value of 0. This threshold measures total system memory. Therefore, if a value other than 0 is configured, a throttling condition can occur if a non-BizTalk process is using high memory.
Dehydration throttling thresholdsThe default memory dehydration thresholds may cause too much dehydration when orchestrations are run on a 64-bit host. For more information about this issue, see the Dehydration Default Properties topic on the following Microsoft Developer Network (MSDN) website: Note 64-bit hosts are supported in BizTalk Server 2006 and later versions.
On equivalent hardware in a 32-bit host instance, observed dehydration is nominal when the same orchestrations are run by using the default memory dehydration throttling thresholds.
Because 64-bit architecture provides expanded memory address space (16 TB instead of 4 GB), 64-bit host instances are allocated significantly more memory than 32-bit host instances. This can cause the default memory throttling thresholds to be exceeded.
To work around this behavior, change the VirtualMemoryThrottlingCriteria and PrivateMemoryThrottlingCriteria values in the BTSNTSvc64.exe.config file. Use the Process\Virtual Bytes and the Process\Private Bytes Performance Monitor counters to determine the largest amount of memory that is being allocated by an orchestration instance.
- Set the OptimalUsage value for both properties based on the following: VirtualMemoryThrottlingCriteria: \Process\Virtual Bytes value + 10%
PrivateMemoryThrottlingCriteria: \Process\Private Bytes value + 10%
- Set MaximalUsage for both properties to the OptimalUsage value + 30%
If the \Process\Private Bytes Performance Monitor counter value is 435689400 bytes (415 MB), set the OptimalUsage value for PrivateMemoryThrottlingCriteria to 457 MB (435689400 * 1.10 = 479258340 bytes). Set the MaximalUsage value for PrivateMemoryThrottlingCriteria to 594 MB (479258340 * 1.30 = 623035842).
For this example, the following values would be specified in the BTSNTSvc64.exe.config file to reduce throttling.
|Performance Monitor counter||Memory allocated||OptimalUsage||MaximalUsage|
|\Process\Virtual Bytes||5784787695 bytes (5517 MB)||6069||7889|
|\Process\Private Bytes||435689400 bytes (415 MB)||457||594|
<xlangs>To determine which host instance is running the orchestration, you can match the ID Process from the \BizTalk:Messaging\ID Process and \Process\ID Process Performance Monitor counters. Then, check the Average value displayed for the corresponding \Process\Virtual Bytes and \Process\Private Bytes Performance Monitor counters.
<VirtualMemoryThrottlingCriteria OptimalUsage="6069" MaximalUsage="7889" IsActive="true" />
<PrivateMemoryThrottlingCriteria OptimalUsage="457" MaximalUsage="594" IsActive="true" />
Note The high dehydration may cause a significant decrease in performance when the BizTalkMsgBoxDb database is running on SQL Server 2008.
BizTalk Server Service Packs and Cumulative UpdatesBizTalk Server service packs and cumulative updates include the latest fixes. These include those that affect known System.OutOfMemoryException issues.
2281783 Service Pack and Cumulative Update list for BizTalk Server 2006 R2
Microsoft BizTalk Server 2004 Service Pack 2
HeapDeCommitFreeBlockThresholdBy default, theHeapDeCommitFreeBlockThreshold registry key value is 0. A value of 0 means that the heap manager decommits each 4-kilobyte (KB) page that becomes available. Decommit operations can cause virtual memory fragmentation. The size of the HeapDeCommitFreeBlockThreshold setting in the heap manager will depend on the kind of work that the system is doing. A size of 0x00040000 is a recommended starting value.
Consider the following information before you change the value of the HeapDeCommitFreeBlockThreshold registry key:
- This change only applies to memory fragmentation problems.
- This change is system-wide. Therefore, most processes will use more memory on startup.
- Only consider this change for systems that have BizTalk Server as their primary mission.
Value name: HeapDeCommitFreeBlockThreshold
Value type: REG_DWORD
Value data: 0x00040000 (This is the recommended starting value.)
Value default: not present
Transform operationsWhen BizTalk Server performs XML transform operations on fairly large messages in a receive port, in a send port, or in XLANG, XSL transforms load the whole message in memory..
To resolve this issue, use one of the following methods:
- Decrease the number of messages that BizTalk Server processes at the same time.
- Reduce the size of the XML message that is being transformed.
Most of the default BizTalk functoids are implemented as inline script. These items can cause System.Byte objects to collect in memory. To minimize memory consumption, we recommend that you put any map that uses these functoids into a small assembly. Then, reference that assembly. Use the following chart to determine which functoids use inline script and which functoids do not use inline script.
In the second column, “Yes” means that this functoid is implemented as inline script, and that it will cause System.Byte objects to collect in memory. “No” means that this functoid is not implemented as inline script, and that it will not cause System.Byte objects to collect in memory.
|All String Functoids||Yes|
|All Mathematical Functoids||Yes|
|All Logical Functoids except IsNil||Yes|
|Logical IsNil Functoid||No|
|All Date/Time Functoids||Yes|
|All Conversion Functoids||Yes|
|All Scientific Functoids||Yes|
|All Cumulative Functoids||Yes|
|All Database Functoids||No|
|Advanced Functoids||Inline script?|
|Value Mapping Flattening Functoid||No|
|Table Extractor Functoid||No|
|Table Looping Functoid||No|
|Scripting Functoid with Inline C#||Yes|
|Scripting Functoid with Inline JScript.NET||Yes|
|Scripting Functoid with Inline Visual Basic .NET||Yes|
|Scripting Functoid with Inline XSLT||No|
|Scripting Functoid with Inline XSLT Call Template||No|
|Scripting Functoid calling External Assembly||No|
|Nil Value Functoid||No|
|Value Mapping Functoid||No|
|Mass Copy Functoid||No|
|Record Count Functoid||No|
Large attribute values and large element valuesWhen BizTalk Server executes a receive pipeline or a send pipeline on an XML document, the payload is processed in memory if the document contains one or more of the following entities:
- Large attribute values
- Large element values
- Large attribute or element tags
Custom pipeline componentsYou are using a custom pipeline component that loads the whole stream into memory. All the components that are included with BizTalk Server, except transforms, support streaming. These components do not use as much memory during streaming. However, custom pipeline components may not support streaming.
Streaming under heavy stressSend hosts run out of memory when they operate under heavy stress. BizTalk Server send pipelines and send adapters support streaming. In streaming, each component loads a small fragment of the stream into memory. Because each message includes other data structures, together with a message context that can be big or small, this behavior affects the behavior of BizTalk Server under heavy stress.
The behavior of BizTalk Server is affected because the engine loads a preconfigured number of messages. The number of messages that the engine loads is based on the values that appear in the LowWaterMark field and the HighWaterMark field of the Adm_serviceClass table. The Adm_serviceClass table is in the BizTalk Management Database. These values control the number of messages that BizTalk Server processes or sends at the same time.
The HighWaterMark value is the total number of messages that the engine processes at the same time. The default value is 200 messages per CPU. Therefore, on an 8-processor server, the send host will try to process 1,600 messages (200*8) at the same time. If you assume that each message is 50 KB, the messages equal 80 MB (1,600*50=80,000KB).
To resolve this issue, you can change the HighWaterMark value and the LowWaterMark value in the database. The values that you use depend on the size of the messages.
For more information about common causes of an out-of-memory condition, see the "Memory Growth in BizTalk Messaging" section at the following Microsoft website:
Try to simplify the issueIf you have identified a memory leak, try to determine the cause by removing custom components or by simplifying a map. Also, try to reproduce the issue by using a simple orchestration or a simple solution. Typically, you should create separate receive hosts for receive adapters. You should also create separate send hosts for send adapters. When you use this method, each adapter can run in a separate process. Therefore, if your BizTalk Server process experiences an out-of-memory condition, you will know which components are involved.
Troubleshooting stepsTo troubleshoot an out-of-memory condition, use the Debug Diagnostics tool to monitor memory allocations over time. The Debug Diagnostics tool can create and analyze a memory leak dump file (.dmp). When you troubleshoot memory leaks, the goal is to attach Leaktrack.dll before the high memory condition reproduces to capture memory growth over time. Leaktrack.dll is included with the Debug Diagnostics tool.
- Install the Debug Diagnostics Tool.
The following file is available for download from the Microsoft Download Center:
Download the Debug Diagnostic Tool package now.
For more information about how to download Microsoft support files, click the following article number to view the article in the Microsoft Knowledge Base:119591 How to obtain Microsoft support files from online servicesMicrosoft scanned this file for viruses. Microsoft used the most current virus-detection software that was available on the date that the file was posted. The file is stored on security-enhanced servers that help prevent any unauthorized changes to the file.
- Use Performance Monitor to collect data about system performance. This data may provide important indicators about the efficiency of your BizTalk Server environment. The goal is to capture process performance over time. Therefore, enable Performance Monitor logging before the memory leak occurs.
How to use Performance Monitor logging
Select the data to logTo select the data to log, use the method that is appropriate for your operating system:
- For Windows Server 2008 and Windows Server 2008 R2
- In Administrative Tools, open Reliability and Performance Monitor.
- Right-click Performance Monitor, click New and then click Data Collector Set.
- In the Name box, type a descriptive name, and then click Next.
- Note the Root directory, and then click Next.
- Click Start this data collector set now, and then click Finish.
- Expand Data Collector Sets, expand User Defined and then select your file.
- Right-click System Monitor Log, and then click Properties.
- Click Add on the Performance Counters tab. Select the following objects, and then click Add after you select each object:
- .Net CLR Exceptions
- .Net CLR Memory
- XLANG/s Orchestrations
- SQLServer:General Statistics
- SQLServer:Memory Manager
- Click OK.
- Change the Sample Interval value box to 5 seconds.
Note The Sample Interval value and the time to start to monitor are subjective. These values depend on when the memory leak is reproduced. Because the log file can be large, specify an interval in which you can obtain the information that you must have without overwhelming the server.
- Click OK.
- For Windows Server 2003 or for Windows XP
- Expand Performance Logs and Alerts.
- Right-click Counter Logs, and then click New Log Settings. The New Log Settings dialog box appears.
- In the Name box, type a descriptive name, and then click OK.
- Note the log file location. (You can also click the
Log Files tab, and then click Configure to change the log file location.)
- Click Add Counters.
- Select All counters and All instances.
- In the Performance object list, select the following objects. Click Add after you select each object.
- .Net CLR Exceptions
- .Net CLR Memory
- XLANG/s Orchestrations
- SQLServer:General Statistics
- SQLServer:Memory Manager
- Click Close.
- Change the value in Data Sampling Interval to 5 seconds.
Note The Data Sampling Interval value and the time to start to monitor are subjective. These values depend on when the memory leak is reproduced. Because the log file can be large, specify an interval in which you can obtain the information that you must have without overwhelming the server.
- Click OK.
Obtain the dump fileTo obtain the dump file, use one of the following methods:
- Method 1: Automatic
Creating a Memory and Handle Leak rule with DebugDiag is the recommended approach to capture a memory dump. The Memory and Handle Leak rule automatically attaches Leaktrack.dll. This is used to track memory allocations. To create the Memory and Handle Leak rule, follow these steps:
- Start Debug Diagnostics Tool 1.1.
- Select Memory and Handle Leak, and then click Next.
- Select the Btsntsvc.exe process, and then click Next.
- On the Configure Leak Rule page, follow these steps:
- Click to select the Start memory tracking immediately when rule is activated check box. Otherwise, you can specify a warm-up time before LeakTrack.dll is injected in the BTSNTSvc.exe process.
- Click Configure, and then do the following:
- Confirm that Auto-create a crash rule is selected. By selecting this option, a memory dump will be created automatically if the BTSNTSvc.exe process stops.
- Click to select the Generate a userdump when virtual bytes reach check box, and keep the default value of 1024.
- Click to select the and each additional check box, and keep the default of 200.
- Click Save & Close.
- Click Next.
- On the Select Dump Location And Rule Name page, click Next.
Note You can also change the path of the dump file in the Userdump Location box on this page.
- Click Finish to make the rule active now.
- Method 2: Manual
You can also manually attach Leaktrack.dll and manually obtain the memory dump file. This enables you to control when the memory dump is created. To do this, follow these steps:
- Start Debug Diagnostics Tool 1.1.
- Click the Processes tab.
- Right-click the Btsntsvc.exe process, and then click
Monitor For Leaks.
- In the Debug Diagnostics Tool dialog box, click Yes, and then click OK.
- Start Debug Diagnostics Tool 1.1.
- Select Crash, and then click Next.
- Select A specific process, and then click Next.
- Select the same Btsntsvc.exe process, and then click Next.
- On the Advanced Configuration (Optional) page, click Next.
- In the Select Dump Location And Rule Name (Optional) dialog box, click Next.
- Select Activate the rule now, and then click Finish.
Stop Performance Monitor loggingIf you are capturing a memory dump and Performance Monitor data, stop Performance Monitor logging about two minutes after the memory dump is created.
Analyze the dump fileTo help determine the cause of a memory leak, you can use the Debug Diagnostics tool to analyze the dump file. To do this, follow these steps:
- Click the Advanced Analysistab.
- Click Add Data Files, and then locate the .dmp file.
- Select the Memory Pressure Analysisscript, and then click Start Analysis.
If you use custom DLLs, you can add the symbol path of the custom .pdb files for analysis. To do this, follow these steps:
- Open the Debug Diagnostics tool.
- On the Tools menu, click Options and Settings.
- In the Symbol Search Path For Debuggingbox, type the symbol path.