Provide Feedback on this Broadcast

Note This document is based on the original spoken WebCast transcript. It has been edited for clarity.

Wade Mascia: Welcome, everyone. This is the ASP.NET Threading WebCast. My name is Wade Mascia. For the past two and a half years I've worked in developer support here at Microsoft on what is essentially the middleware team. This team supports quite a few middleware technologies like Transaction Server, COM+, ASP/COM applications, and ASP.NET Web services.

I think that's plenty to know about me. Let's get started.

These are the objectives (slide 2) we are going to hit over the next 45 minutes or so. By the end of the presentation you should have learned how threading affects things like performance, scalability, and even security in your ASP.NET application; how ASP.NET uses different types of threads to execute your requests; and most importantly, how to configure ASP.NET for maximum performance. Also, we'll take a quick look at how COM uses different threads to invoke objects, and then we'll take a look at how ASP.NET and COM come together, and how to avoid some common issues. That's what we're going to cover.

This is the agenda. First, we'll take a look at why we care (slide 3) about threading anyway. We'll look at ASP.NET by itself, and take a mile-high look at how IIS and ASP.NET interact. And then we'll go into detail, we'll take an in-depth look at the threading scheme within the ASP.NET worker process. Then we'll do a quick overview on COM threading. And then finally we'll put it all together and talk about some of the common ASP.NET and COM issues.

The first thing is, why do we care? Mostly because of the three buzzwords you see on this slide (slide 4) here: scalability, performance, and security. Later, in the configuration section, we'll talk about how thread pooling affects scalability and performance and how to size a thread pool to adjust to your particular application.

In a few places we'll talk about thread switches, how they hurt performance and also how they can cause some security headaches if you're using impersonation. Finally, we'll talk about the different types of threads, both within ASP.NET and within COM. We'll talk about the implications that these threads have on performance and scalability.

Here's the mile-high view that I promised. Basically, what this shows (slide 5) is the life of an ASP.NET request. It shows how it will go through IIS and then wind up in the ASPNET worker process. The way it starts is, as with any HTTP request, it comes into port 80 on the server (typically port 80), and that's the port that the IIS service is listening to, that is, Inetinfo.exe on your server. It starts off here at this Isatq.dll, which is where the IIS thread pool is. Essentially, that's the first thread within IIS to take a shot at the request.

Just like any other IIS request, the first thing it does is look at the file extension to see if there is an ISAPI extension registered for that particular file extension, so for anything like .aspx or .asmx, for example, it's going to go to the ISAPI extension that is registered for ASP.NET. That's the Aspnet_isapi.dll. Just like any other ISAPI extension, this is an unmanaged legacy, meaning a non-.NET .dll, and there's an entry point, HTTPExtensionProc; it's a standard entry point that any ISAPI extension would expose.

For the most part, this just forwards requests off to the worker process. It's also responsible for some other things, like helping with monitoring or keeping track of what requests have been sent out to the worker process. If it needs to shut down the worker process for some reason, it can. It can know that some requests were not processed, and instead of losing them, it can forward them off to the next worker process.

One thing to note here is that back in legacy ASP, Asp.dll was also an ISAPI extension, but it would run in a different process, depending on if you used low, medium, or high application protection. It might run in various Dllhost processes, and you would see various COM+ applications created for that purpose.

In this case though, this ISAPI extension is registered as an in-process ISAPI extension. There's a metabase entry for that. So this .dll also runs directly within Inetinfo, and the application protection setting (low, medium, or high) that you pick for your virtual directory doesn't come into play at all for ASP.NET requests.

Eventually IIS decides, based on the file extension, to use this ISAPI extension. And the ISAPI extension forwards the request off to the ASP.NET worker process. That is through the asynchronous name type. That's the way IIS and ASP.NET will communicate with each other. There are actually a couple pipes, depending on the direction. For example, when ASP.NET sends a response back to IIS, or for various other reasons, it might use some other named pipes. But this is the main one in that, as a request is inbound, there's an asynchronous named pipe here.

Now we'll just look at the worker process itself (slide 6). You see that little arrow on the left. Every single request coming into the worker process comes over that named pipe. ASP.NET just makes use of the standard common language runtime thread pool to service these requests. The way it closes that off is it calls the ThreadPool.BindHandle method, so it binds the named pipe to the ThreadPool object; that's the System.Threading.ThreadPool object.

From then on, every request that comes over the named pipe will be serviced by an I/O thread from that common language runtime thread pool. That's the same thread pool that you can use from any .NET application. This is the starting point for every single request. It will come into the named pipe. It will be queued up within this I/O completion port (IOCP), and it will be serviced by an I/O thread off the common language runtime thread pool.

That part is always the same, and then from here there are a few differences. This slide shows you what happens under low loads. We'll talk about what is low load, what is high load, and exactly what those terms mean. Let's just say that for now, under low load, there's not much going on. It's this I/O thread that will take the request directly and start to process it. It calls into essentially what is the ASP.NET runtime, things like HttpHandlers and modules, for example, and it begins to process the request.

At the end of that chain it will eventually call into your custom code, your .aspx or .asmx code. Maybe you've compiled it to a code-behind .dll, that kind of thing. That's the simplest scenario. Everything starts with an I/O thread. If there's not much happening on the server, there's not much load, it will be that I/O thread that will directly process the request. And it goes through the ASP.NET runtime. At the end of that chain, we'll eventually call into your code.

The next thing to consider is the high load (slide 7), which basically means that the I/O threads are mostly busy processing other requests. So at some point ASP.NET decides that there are too many I/O threads processing other requests: "I don't want to process any more requests on I/O threads, because I'm running out." These I/O threads are used internally by ASP.NET for other purposes, other than just processing your requests — for sending responses back and other things, it uses I/O threads.

It just takes the request and it queues it up in this internal queue object within the ASP.NET runtime. Then, after that's queued up, the I/O thread will ask for a worker thread, and then the I/O thread will be returned to its pool. The I/O thread is not processing any requests at this point. It only queues it up and it puts in a request for a worker thread. The way that happens is just like you would do in your code with Threadpool.QueueUserWorkItem.

The ThreadPool object gets that request to the QueueUserWorkItem call, and whenever there's a worker thread available, the worker thread will be used to call into the callback function. And that's the ASP.NET code. So ASP.NET will have that worker thread process the request. It will take it into the ASP.NET runtime, just as the I/O thread would have under low load. It's the same scenario from there. Eventually, at the end of that chain, your code is invoked.

That's the first example we see of a thread switch. In this case, the request starts off on the I/O thread; it is dispatched to a queue, and it calls into queue as a work item. When there's a worker thread available, that second thread comes in and gets the request.

One final thing to consider: regardless of how the request gets into the ASP.NET runtime, either by worker thread or an I/O thread, depending on the load, after it's in the runtime and we go through all those handlers and modules, etc., one of the things that is checked is whether or not you're using the AspCompat attribute (slide 8). Of course, that only applies to .aspx pages, not Web services.

If you are using this attribute, then it means we need to process the request on some kind of STA thread. So what we do is make use of the COM+ STA thread pool that is already written for us. So ASP.NET will basically create a COM+ activity and ask COM+ to process this activity. COM+ is responsible for managing the STA thread pool. It's the same thread pool that was used in Dllhost to process your apartment-threaded objects, for example.

When this STA thread is available, COM+ will call into ASP.NET, and then it's the same story. ASP.NET will take it back into the runtime code and finish off the request, eventually calling into your component.

The one new thing you see on this slide is that System.Web.dll, in that little purple box, there's a little token; that's an impersonation token. After all this is finished, and we have the final thread that we know is going to be calling into your code — it may an I/O thread, worker thread, or a COM+ STA thread — this is the point where impersonation is handled.

ASP.NET will look into your Web.config file. If you're using the Identity tag with impersonate=true, then some kind of impersonation token will be ascribed to this thread. Exactly what user that is depends on exactly what you're doing.

Obviously, one option is to put the user name and password directly in the config file. In that case, ASP.NET impersonates a logged-on user, it gets a token for that identity, and that's the token that you would use.

The other option is to leave that user name and password blank, and in that case it will just use the same impersonation token that IIS has used. IIS always impersonates somebody when processing requests, even if it's just a .jpg image or an .htm file or whatever. Based on the IIS authentication options that you have, Anonymous, Basic, or Integrated, for example, IIS is always going to impersonate some identity. That way the NTFS checks and registry access checks are done, so it can control who has access to the resources.

If you leave this name blank, it will just inherit whatever IIS is using, and that just boils down to the authentication option that you pick. Either you're using the Anonymous user, or you're impersonating the actual end user who is sitting on the browser making a request.

That's the story, as far as how ASP.NET chooses threads to execute your requests.

One more thing that's slightly interesting within ASP.NET, as far as threading is concerned, is the way it synchronizes application and session data (slide 9). This slide basically shows two scenarios. You have one scenario at the top where two threads are trying to access the application object at the same time, and then you have a second scenario at the bottom where two requests from the same user are trying to run at the same time.

Let's look at the top scenario. This shows you Session A and Session B, meaning two different users have put in requests concurrently. They're running together within the server process, and they both want to access some application-scoped piece of data. You've taken some object or some string or some piece of data and you've put it in application scope in the application collection.

Internally, within ASP.NET code, and this is identical to the way classic ASP used to work, there is a public method that even you can call, Application.Lock. Internally, within in the code, when any one of these items is retrieved out of the collection, it will call the Application.Lock. In ASP.NET, that results in this Threading.Monitor object, which is used to synchronize that data.

So when a request comes in to fetch an item out of the collection, it essentially puts a lock around that so that only one request can access that piece of data at any given time. If you need to make multiple requests and you need to wrap a whole bunch of them into a serialized session, you can explicitly call lock and unlock before and after you're done with that.

The second scenario is two concurrent requests from the same user. It's kind of hard to pull this off for the most part; you don't see a lot of this, especially in the Web services world. For one thing, you're typically not using sessions, and, if you are, the client would need to spawn multiple threads and explicitly pass the session around and this kind of thing. It's not very likely.

But in the Web application world, with an .aspx page, one easy way to do this is to have frames on a page, and then the browser will send in a separate request concurrently for each frame. If different frames are requesting different ASP pages, then in that case a very likely scenario is that you'll have multiple requests coming in concurrently for the same session.

What happens in that case, the way ASP and ASP.NET protect the session data, is they just queue those concurrent requests from the same session up on the same thread. The thread can only process one request at a time, so it processes the first request while the second one just waits. Even if the first request doesn't access any session data, they're still queued up, so there's no chance that two requests from the same session will be concurrently accessing any data that's stored in session scope. If you're not using any session data then you should disable sessions to avoid that bottleneck.

Those are a couple bottlenecks to look out for. If you find that requests are taking longer than they should — or within a particular piece in the code, when you're accessing application data, this seems to be taking longer than it should, this would only occur on your load when you have multiple requests — but if you see something like that, keep this in mind, and take a look at what variables you're trying to access. You may run into this kind of synchronization here.

So, the configuration (slide 10), this is definitely a big issue.

Basically I have four groups of items that you may want to tweak in the configuration file. The first thing here is maxIoThreads and maxWorkerThreads. These essentially will just set the upper limit for the size of each of those thread pools. The default is 25, and it's a per-processor setting. So you'll have a maximum of 25 threads in each pool per processor. If you have a dual-processor machine, and this is set to 25, you have a maximum of 50 in each pool; you could potentially have 100 threads in the process, at least for I/O and worker threads; you'll have others.

That's the description, and so why would we want to tweak that? One common scenario is if you have a lot of long blocking calls, meaning you're just waiting on some back-end resource. Maybe you're calling into stored procedures that take quite a bit of time; maybe you're calling to some back-end DCOM server, another remoting server, any kind of back-end resource, meaning it's not in the ASPNET worker process. Maybe it's even on another machine.

When you do that, whatever thread it is that is executing your request, the I/O thread or the worker thread, is going to sit there and block until the response comes back. Under load, that's an easy way that you can just run out of threads, and the ASP.NET worker process is really just kind of sitting there waiting, and you have a lot more CPU that you could be using up on that server, especially if the back-end process is on a different server.

That's one common scenario where you would want to increase that number. Obviously it would depend on your application as to how high you would want to increase that. But you could increase it up to the maximum of 100.

Essentially, the thing you want to look for is CPU, because there is a downside to increasing the number of threads here. If these requests are not waiting on some backend resource, but they're chugging through some loops and doing some calculations directly within the ASPNET worker process, and you find that you're not getting the response that you want, you're not getting so many requests per second, and you want to try and increase performance, well, increasing the number of threads at that point is only going to hurt you. If the CPU utilization is high, and you throw more requests in to the picture, then all you're going to do is give the CPU more work. Now it has to swap between more threads. It takes time, each time it does a context switch from one thread to another. So the key thing to look for is the CPU utilization.

Under low CPU utilization, if you find that you have a bottleneck, where requests are just taking a long time and then they start to queue up — you're running out of threads — you can safely increase the number of threads in that scenario.

But if you have high CPU utilization, increasing the number of threads is only going to hurt you at that point. Decreasing the threads really doesn't help much either. It will help a little bit, because the CPU doesn't have to spend too much time swapping between threads, but it's not going to increase the performance significantly. If that is the case with high CPU utilization, you have to scale out. You have to load balance across multiple servers, or you can scale out in terms of number of CPUs, go eight-way on the box, or something like that.

Increase for long blocking calls under low CPU utilization. Under high CPU utilization, don't increase that number. You're going to need to scale out with either processors or servers, that kind of thing.

Next, we have the minFreeThreads and the minLocalRequestFreeThreads. First of all, just know that the minLocalRequestFreeThreads applies to the case where a request is coming directly from the local machine. Typically it's not a great idea to have that kind of architecture, but it may warrant itself for the application. If you do have that kind of thing, where maybe an .aspx page calls to a Web service, and the Web service is on the same machine, then that's where you would have something like the minLocalRequestFreeThreads taking effect. For external requests, it's just the minFreeThreads.

This is a number that ASP.NET will reserve out of the thread pool for uses other than processing your requests. This is how the high load versus low load is determined. Remember we said that ASP.NET makes a decision that there's not really enough I/O threads left, so it's not going to use an I/O thread for this request. This is a number that it uses to make that determination. It will always leave this number of threads left over in the pool for other uses.

Like we said, the runtime will use some I/O threads for its own use, and there's a potential that you could use worker threads for your own use, also. So if you are using worker threads, either explicitly, say you're calling into QueueUserWorkItem, or implicitly, if you're making outbound Web service calls, if you're doing any kind of asynchronous operation, like you're making an asynchronous Web service call, an asynchronous remoting call, or an asynchronous file I/O, there are a lot of ways you can make use of the .NET thread pool under the hood.

If you are using any worker threads from your requests, you're going to want to make sure that you have enough left over, or else you can get into a couple different situations. The worst kind of circumstance you can get into, if all the worker threads are busy processing your requests, and then all of your requests are trying to wait for worker threads — say they're all calling userWorkItem or they're all making an outbound Web service call, an asynchronous call or something like that — if all the worker threads are busy processing requests, but then they're all just stuck there waiting for new worker threads to become available, then the whole process will deadlock, and none of the requests will unwind. Eventually the process will be recycled, in that case.

The other scenario you can have is if it's not quite so severe, but most of the worker threads are busy processing requests, and a lot of them are waiting for worker threads to become available, you can be left with not enough worker threads to quickly service all these requests, so they'll start to queue up at the thread pool level. So, as you try to do these asynchronous operations, you'll see a lot of blocking, and it will take a lot of time. The bottom line is if you're making use of worker threads, you want to increase this number.

There is another trick you can use here — you'll see the default setting for this is minLocalRequestsFreeThreads as four, while minFreeThreads is eight. The greater the difference between these two numbers, the more priority you're giving to local requests. Essentially what you're saying is, "I only need four free threads left over, if it's a local request." That means it's more likely, if you have local requests and external requests coming in, and you're running out of threads, that ASP.NET is going to be processing the local requests, and it's going to be queuing up the external requests. So that's one way you can give priority.

If you do have a scenario where you have local requests coming in, such as an .aspx page calling a Web service, or even Web service to Web service, and you need to let that complete before processing external requests, then this is one way to do it. You have a big difference between those two numbers. The reason you want to do that is because when an external request comes in, it will take one of these I/O threads or worker threads.

Now, if ASP.NET continues to let all the external requests come in and hog the worker threads, those requests are just going to block, because they're making local requests and there are no more worker threads available. That's the reason you give priority to the local request, so that it's an existing request that is executing and making a local request, and you want to let that finish, so it goes back to the client and frees up the worker thread that is sitting there blocking.

One more thing, maxconnection here, that just limits the number of outbound connections you can have to any given server. The default number here is two, and it applies to all outbound servers. It does not apply to the local host; you can have an infinite number of connections to the local host. This will cause a bottleneck. If you're making outbound Web service calls to some remote server, then you'll notice that only two of them can go off the box at one time. What you'll have is a bunch of requests queuing up on your server, while there are only two requests executing on the downstream server. So keep that in mind for outbound requests.

You want to increase that number slowly, but you'll probably want to increase that if you see that these outbound Web service calls are bottlenecking on you.

After you tweak these settings as best you can, there are still some cases where requests are just going to queue up under load. It may be something extravagant, like maybe you're calling to a stored procedure that just takes an hour, and that's the Web service that you have. Well, you know that that thread is not going to be available for another hour, and if you have n number of threads, and n number of requests come in, then all the threads are going to be used up. Requests are going to start to queue up at that point.

If that's what you expect for your application, if you have some scenario like that, then what you can do is just increase this appRequestQueueLimit, because after this number of requests queues up, then ASP.NET decides there's some problem, and it sends back a "server too busy" error. So if you plan to have these requests queue up, and you know that's going to be the case, then you can increase that appRequestQueueLimit.

The last thing I'd point you to, there is a Recommendations.htm file that comes with the supplemental files. There's a good discussion in there about a lot of things, like calling to a Web service and the different socket threads that are used, and how you might want to configure your maxThreads and your maxconnection based on how many outbound Web service calls you have.

Also, it highlights an issue with I/O threads. You can read the HTLM file, but the bottom line there is that only one I/O thread per processor can be busy at any given time. So as a request comes in, if that many I/O threads are busy (meaning they're executing code, not blocking, waiting on something else, but say they're stuck in a loop or something like that), then no more requests will be put onto I/O threads at that point. That's just built into the I/O completion port APIs that were created long ago for NT, long before ASP.NET. And that's just the default behavior of a completion port.

The intent is to have the CPU only busy on one thread at a time to avoid the contact switching, but it can cause a pretty bad bottleneck. If you happen to have some long-running tight-loop, high CPU — I've seen this happen creating .pdf files, a lot of different things like that — if you happen to have a high-intensive CPU request like that, that's going to take a long time, if that runs on an I/O thread, it's going to cause a bottleneck, and no other request will be able to get in at that time. That .htm file talks about that issue, it talks about how to move that request over to a worker thread. And it also talks about a recent hotfix that addresses the issue for the 1.0 runtime. But there's nothing for the 1.1 runtime.

I would definitely recommend looking into that. If you have some of these high-CPU requests, you're going to need to know what hotfix you're on, what version of the framework you're on, and this kind of thing.

On to COM (slide 11). I'm going to try to keep the COM stuff really short, because it's intended to be just a review, and I want to get to your questions. This is just a look at a generic COM process with different apartments. It will have multiple STA apartments; it shows you STA 0, 1, and 2. It will have only one multi-threaded (MTA) apartment and it will have only one thread-neutral apartment (TNA). Then, also, you can see the number of threads that reside within each apartment. An STA apartment will have only one thread. It's a single-threaded apartment, it has only a single thread to process all the objects that reside within that apartment.

A multi-threaded apartment can have any number of threads that can initialize to the multi-threaded apartment, and any one of those threads can be used to invoke the objects that are residing in the MTA.

Then there is no particular thread associated with the thread-neutral apartment. Any one of the threads in the process can enter and leave the thread-neutral apartment without a thread switch. That's basically COM 101.

The next thing we'll look at is what happens when you trade objects of different threading models and so forth. This slide (slide 12) and the next slide are good for reference. You can take a look when you have a scenario and you want to know where the object is going; what thread is going to be used? It's probably not worth going through every single scenario, but I'll just explain the way this slide is laid out.

They're color-coded, so you'll notice that in the STA 0, the STA 1, and the MTA we have some clients, and they're all different colors. Then we have a number of objects that are marked with different threading models. You have single apartment, both, neutral, and free. What this shows you is that the purple client creates an apartment-threaded object. You see the purple apartment-threaded object; that shows you where that object will reside. In that case it will reside directly in that STA.

The scenario with the red arrow, that would be a thread switch. That is an MTA client creating an apartment-threaded object. That is basically what happens if you create something like a Visual Basic® 6.0 object from an .aspx page or from a Web service without using AspCompat in your .aspx page. That's the scenario you want to watch out for. There are some other problems, not just with the thread switch; there are some other issues that we'll talk about, too.

But this is the reason for AspCompat. This way your client is not running on the MTA; it's running on an STA thread. Then, in that scenario, all the apartment-threaded objects are invoked directly on that thread.

The next side (slide 13) is just another look at the same story; it's just in a spreadsheet view. That red arrow is the same scenario. The client is on the local MTA and it creates an apartment-threaded object. Where is the object created? It's created in some STA. And that STA actually has a name, called the host STA. Those two slides, it's not really worth going through every single scenario; it's better to use this just for reference. If you need to come back at some point to try gain some understanding, it's a good lookup table for you.

Let's try and put it together, ASP.NET and COM (slide 14). As we just saw on the last slide, where an object resides or, more correctly, what thread will be used to invoke an object, all depends on the treading model of the object and the apartment that the client is in.

Let's start with an STA client. How do we get STA clients in ASP.NET? Well, there are only two ways that you can do it. One way is if you have an .aspx page with the AspCompat attribute. If you have aspcompat = true, then your .aspx page will running on an STA thread. The events only — at the end of the presentation there are some Knowledge Base articles that I'm going to refer to. There is one article that talks about this. It's only the events that actually run on an STA thread. Things like the constructor for the page object and a few other things will fire on an MTA thread. So it's only your events, like page load, those kinds of things, those are going to fire on an STA thread. You don't want to use any COM objects from the constructor or other places where it's not an event. Definitely take a look at that article, because it explains this in good detail.

The other way you can get STA threads in ASP.NET is if you create your own custom threads. Another article that I'll point you to, one that I've written, is about hunting and the impersonation token across threads. If you can't avoid the thread switch, then you can pass the token yourself, so that way you don't lose the identity. The identity doesn't change between threads. That article shows you how to create a thread and how to initialize an STA. In .NET, all you have to do is set the apartment type to STA.

It's worth talking about the site switch for a second, too. Back at the top, when I was talking about why we care, I said that thread switches can cause security problems. So let's talk about that. If you're impersonating, then in your Web.config you have impersonate=true. So the thread that is executing your request has some kind of impersonation token ascribed to it. So it's essentially executing under some identity that you want to impersonate.

Now if, for any reason, that thread makes a thread switch to a second thread to either invoke a COM object or do any number of things, the impersonation token stays with the first thread. So the second thread has no impersonation token; it just runs under the process identity, which, by default, is ASP.NET.

There are a lot of ways that this can cause headaches. Because of the fact that the ASP.NET account is a low-permissions account for better security, what normally happens in that case is you just get an access denied where it doesn't make sense to get an access denied. For example, you have an .aspx page that calls a Visual Basic 6.0 .dll, which goes to a SQL database using Windows NT® security. So it relies on the thread identity of that Visual Basic 6.0 object, and if there was a thread switch, you've lost the impersonation token. So it's going to try to go to the database under the ASP.NET account, which will fail. That's one important reason why we want to avoid the thread switches.

The other thing is just the fact that there's a performance hit as the CPU is swapped out between thread 1 and tread 2 and back again, potentially marshaling data or copying data across, for example. When your client is an STA, the bullet there says it's a thread switch to free-threaded objects, so there are some things like the free-threaded XML DOM objects. Any other custom ATL object you may have that is set to free-threaded, you don't want to use those from an STA thread. If you are using both Visual Basic 6.0 and free-threaded objects, then you're probably going to want to do that on different pages, one marked as AspCompat and one not.

So what runs on an MTA thread in ASP.NET? Everything else. By default, I/O threads and worker threads, if you're not using AspCompat, are the only threads you get, and those are MTA. If you create a custom thread and don't explicitly initialize into the STA, and you're running on an MTA thread, it's basically everything else, except for the above.

The important thing here, like we just said, is the thread switch to apartment-threaded objects. This final bullet is one thing I mentioned about the host STA. When an MTA client creates an apartment-threaded object, we saw in the slides above that it will go to some STA. The object has to be in some STA, because it's marked apartment-threaded. But beyond that, it will go to the host STA, and that means that all apartment-threaded objects created from any MTA thread will all share the host STA, so they're all sharing one thread.

If you have a bunch of requests coming in to an .aspx page or a Web service that invokes a Visual Basic 6.0 .dll, then all instances of this .dll will share the same thread, so all your requests are serializing, and they all stack up and all have to wait.

The answer there, if you have any .aspx page, is you use AspCompat to avoid that. If you have a Web service, you don't have that attribute. One thing you can do is put the apartment-threaded object into COM+ and make sure synchronization is required, which is the default setting.

In that case, it will use the COM+ STA thread pool. It can be in a library application and it will use the same STA thread pool that your AspCompat pages are using, also. It doesn't have to be in a server application to go cross-process. At least in that case, all the multiple instances of the Visual Basic objects are going to be spread across the thread pool, and they won't all get stuck serializing on each other.

There's a KB article on that, {303375, "INFO: XML Web Services and Apartment Objects"}. And it's required that you put the apartment-threaded object into at least the library COM+ application with synchronization turned on, so that you make use of the thread pool.

Now let's talk a little bit about some things that have changed between legacy ASP and ASP.NET (slide 15). If you're upgrading your application, there are a few ways that you can run into some problems that are kind of difficult to troubleshoot.

We've already talked about creating apartment-threaded objects. Creating a single-threaded object in a server process is just a very bad idea. Regardless of whether they're created from an STA thread or an MTA thread, a single-threaded object always has to reside in the main STA, which is the first STA thread that is initialized in the process.

In Visual Basic, when you compile your .dll, you have the option of apartment or single. If you use single, there is no way around all these objects. They'll wind up sharing this main STA and serializing, just like we talked about, with the apartment-threaded objects in the host STA. So avoid that at all cost.

One thing that has changed is in classic ASP, if you stored a non-agile object — and there's a good lengthy discussion about what's non-agile, but let's just say apartment-threaded objects for now — when you would place an apartment-threaded object into Session scope, it used to lock that session down to that particular STA thread. The reason it did that was because of the rules of COM. The object must always be invoked on the STA thread on which it was created. It essentially resides in that apartment for its life.

What classic ASP would do is make sure that the user who corresponded to that session always came back to that same thread, so it would always have direct access to that COM object. ASP.NET is much less COM-sensitive, and it doesn't do that for you. There are a lot of weird issues that can arise from that, but the bottom line is the next request from that same session, that same user, is not guaranteed to be on that same STA thread. In fact, it probably won't be. It will be on a different STA thread, just because of the odds.

When you call into that COM component then, from that other thread, you're making a thread switch, you have a performance hit, and you have loss of impersonation token, all those things. That's one thing to watch out for, apartment-threaded objects and session scope. You're not going to lock down that session anymore.

The next thing is non-agile objects in Application scope. It used to be that you would get an exception in this case. If you used CreateObject or Server.CreateObject and you added this object into the application collection, then you would get this exception that said you can't do this. And it was for a good reason, because you don't want an object like that at application scope, where all requests are going to be accessing it. Because, again, it's only one thread that can service any instance of an apartment-threaded object. So all requests that are accessing this object would be serialized on the one thread. It used to be that you would get an exception preventing you from that. Now you can hang, so be careful of that.

This other thing, AspTrackThreadingModel, that is a metabase setting, but it only applies to classic ASP. That would basically change how ASP determined different objects to be agile or non-agile. It only applies to both threaded objects and whether or not they aggregate the free-threaded marshaler, for example. That no longer applies to ASP.NET, and this concept of agile or non-agile doesn't apply either, so the metabase setting won't have any effect here.

Lastly, there is no global interface table, it's just an implementation detail that you don't really need to worry about. It used to be, in classic ASP, that in some cases we would put some objects in a global interface table and use that to marshal pointers. That doesn't happen anymore.

So that's the last slide before we talk about the supplemental utility. On the Knowledge Base article that advertises this WebCast (820913), there is a link to download this thread pool utility (slide 16). This basically shows you, in real time, what the thread pool looks like: the I/O thread and the worker threads in the server process. The way it works is there is a Global.asax and an .asmx file that goes into the server. Then there's a Client.exe, and you have all the source code for this.

They basically communicate over sockets. I wrote this, so I used the event's .Fire for you in the Global.asax when a response starts and ends, for example. I use those notifications to know when to send updates down to the client. It results in a graph like that will show you, on the client, and in real time, how many threads are available on the server.

Basically, the black threads are those minFreeThreads that we talked about. These are the threads that ASP.NET was reserving. It's not going to use it for your requests; it's going to be left over for other purposes. The red requests show how many threads are actually in use. And then everything else that's left over is green.

In this case, you can see in the bottom left that there are 25 maximum threads in the I/O pool. The minimum free is set to four, because it's on the local host, so it's using four. The minLocalRequestFreeThreads is set to four, so that's the black area that is four threads. Then it shows that there are currently four I/O threads in use. The other thing that this shows you is that ASP.NET makes use of the I/O threads first, and after all that green area is used up and it all becomes red, only then will it jump over to the working threads and start to make use of the working threads.

There is a ReadMe.txt that comes with the utility that tells you how to set it up and how to use it. Have fun with that. You can plug it into any application you want. You can drop that .asax into any other virtual directory where you have some ASP.NET Web application, and you want to monitor for that application. You're monitoring the one and only thread pool for the process, but you get the notifications based on when requests come in and out of that particular application.

There's another thing in there that will let you just get updates at regular interval, so that you can put it into any virtual directory, and if you send updates to the client at a regular interval, then at that interval you would see the thread pool usage for the whole process. This is a good way to know, if you're running out of threads, if you set the pool sizes too high, for example. It also shows you, as you do various things, how many threads it's actually using. So have fun with that tool.

The last slide (slide 17) is just the Knowledge Base articles that I've already mentioned. I did link you to that "XML Web Services and Apartment Objects" article. The only other thing is this PAG book, prescriptive architecture guidelines (Production Debugging for .NET Framework Applications). That is a very large book and it's available for free in .pdf format, and I contributed to that. It talks a lot about how to debug the ASP.NET applications. There's a section on ASP.NET, which is very helpful. It talks a lot about thread pooling, the different types of threads, and lot of the topics that we talked about today, so it would be a good resource for you.

That concludes the presentation. If we have time for questions, we can do that.

Otto Cate: Thank you very much for the presentation. Before we jump into the Q&A, I'd like to share a couple of quick program notes with our listeners. If any of the details on the PowerPoint® slides were difficult to view in your browser today, or you'd like to simply have a copy of the slides, they'll be available for download from that Web site, support.microsoft.com/webcasts/, within 24 hours of the end of this live session.

We'll also have the on-demand streaming media available, as well as the utility download that Wade mentioned. That will show up in the supplemental reading section. We also plan to have a full transcript available within three weeks.

The Q&A portion of the Support WebCast is intended to encourage further discussion of the topics that we have addressed today. In addition, one-on-one product support issues are outside the scope of what we're able to address. So if you do find that you need some more complex technical assistance, feel free to submit an incident on the Web, or contact Product Support Services directly, and speak to a support professional.

With that, let's get into the Q&A.

Wade: Well, there are definitely multiple worker threads. The first items are the maxIoThreads and maxWorkerThreads. These control the upper bound of the size of the thread pool and, by default, they are 25 each, per processor. So certainly there are multiple worker threads. They're created dynamically as needed. You'll start off with only one or a few, depending on if you're talking worker threads or I/O threads, but they're created dynamically, as needed, by the thread pool object. It's not something that ASP.NET manages; it's the built-in CLR thread pool that manages when it's time to create threads and so forth. But certainly there are multiple ones.

The only thing, administratively, that you need to do is configure the upper limit of the thread pool, and everything else is done for you.

Otto: For max threads, is it true that there's a limit of 100 threads? For instance, if you have a system with 16 CPUs, will the maximum be 100 for all 16 CPUs? Or is it going to be 100 for each of the 16 CPUs, giving you a total of 1,600?

Wade: It's a per-processor setting, so it would be for each. If you had 100, then it would actually be more than 1,600, because you have 100 I/O threads and 100 worker threads, potentially, for each processor. But keep in mind, that's only an upper limit, and you would only reach that if you needed to, meaning that all the other 3,000 threads are blocked, and you need 200 more. At that point, there is probably something wrong, and 100 is a very high number.

Anyway, it's true that it's per processor, but it's only a maximum, and it only gets there dynamically, if needed.

Otto: If I have two processors in a server, how many Aspnet_wp.exe files will I be seeing in Task Manager?

Wade: Well, excluding Web gardening and any stuff like this, and excluding IIS 6.0, if you have two processors, then by default you're just going to have one worker process.

Otto: The next question: If you have an ASP.NET page calling a Visual Basic .NET component, and the Visual Basic .NET component has global class-level variables, do you have to protect those variables from multiple threads accessing components charging their values in between calls to the component? And in this case, is impersonation is set to true?

Wade: I don't have the benefit of seeing the whole question, but because you use the word "components," it sounds like you said there was an .aspx page calling a global ASP.NET object. I didn't hear any reference to a COM object. If the answer to that is yes, then all of the .NET code, whether you invoke a separate assembly or anything like that, will always happen on the same thread. There won't be any thread switch there. So there's no concern about losing an impersonation token and so forth, if it's not a COM object.

The other question about synchronizing to access, it really depends on what type of data you have. But certainly, if it's a global object and you have multiple requests that are executing concurrently, and they're accessing this object, then there is the potential that different users would corrupt the data, so you would want to put some kind of lock around that data, which is not thread-safe.

Yes, that's true. You're going to be responsible to lock whatever portions of the code you need to lock.

Otto: When users download large files, 2 MB for instance, through our ASP.NET application, it seems to fail after 15 minutes. In the event log, I see the message that Aspnet_wp.exe PID 676 was recycled because it was suspected to be in a deadlocked state. It did not send any responses for pending requests in the last 180 seconds. Is that something that you've seen before, and do you possibly have some information on that?

Wade: It should happen a lot quicker than 15 minutes. The default setting is 180 seconds. I'm trying to remember the name of the setting. In your Machine.config, there is a setting — I think it's responseDeadlockInterval — and it controls at what point ASP.NET will give up and think that your request is deadlocked, and it's going to shut down the process. So if you have requests that you know are going to take much longer than this, then you should increase that interval in the Machine.config.

Otto: How is the ASP.NET threading architecture changed with the IIS 6.0 rearchitecture? Or does it work the same outside of how the request gets to Aspnet_wp.exe?

Wade: The internal threading architecture is the same. The high-level view is the one that might change, because potentially you're going to have multiple worker processes floating around, depending on how you've pooled your applications together, what version of the framework you're using, this kind of stuff.

There might be another presentation specifically about that, but the short answer is the internals don't change, but the externals, as far as how many worker processes you have and which requests are sent to which worker process, which pool, and this kind of stuff, definitely changes with IIS 6.0.

Otto: Under high load, does I/O threads process any requests other than forwarding requests to the worker process thread pool?

Wade: Under high load, meaning that ASP.NET checks this value for minLocalRequestsFreeThreads or minFreeThreads, whichever is applicable, depending on if it's an external request or a local request, if it determines that it's under high load, meaning it doesn't have enough I/O threads, then the I/O thread will always dispatch the request off to a worker thread and return to the pool.

Also, in the Recommendations.htm file I referenced, I mentioned that there is a hotfix for version 1.0 that changes this behavior. What happens is no I/O threads are used to execute requests with that hotfix. The reason for that is because of the high CPU issue we talked about. Like I said, it's important to know exactly what version you're running because of that fix. Take a look at that in the Recommendations.htm file, which leads you to the article that talks about that fix.

Otto: When accessing COM+ components from an .aspx page, using DCOM, does ASP.NET use local thread or worker process thread?

Wade: I'm not sure what a local thread is, but when you say DCOM, that would mean that the COM+ is a server application, meaning it's running in its own process. At that point it really doesn't matter what the threading model is; the threading, as far as the COM object is concerned, is handled within the COM+ process. You're already guaranteed that there is a thread switch, because you're making a process switch. So the identity of the components will run under whatever the server application is configured to run under, and that kind of thing. It's a much different scenario.

But the thread that is used to execute your request is determined by all the things we talked about before: high load or low load, are you using AspCompat, and so on. In that scenario, even if it is an apartment-threaded object, you would not want to use AspCompat there because, like I said, you're calling across processes, so you're not gaining anything by running your .aspx page on an STA thread. Because, still, the call has to go off to the remote .dll host process and execute the component there. You're just losing a little bit of performance as ASP.NET has to make all the thread switches to get you to the STA thread. So you wouldn't use AspCompat in that scenario, and it wouldn't affect what thread is used to process your request.

Otto: I've created a Web service. When it runs, I notice in Task Manager that the memory usage for Aspnet_wp.exe goes up accordingly. When the process is complete, I notice the memory usage does not drop, though. Is that normal behavior, or is the issue here that I'm possibly not cleaning up the resources? All the code for the Web service is .NET-based.

Wade: It's not really related to threading, but that's basically the nature of garbage collection, be it .NET, Java, or whatever. The concept of garbage collection is basically clean up as needed. So even though you release your references properly and so forth, the garbage collector will just mark them for cleanup, and then later on, when it feels memory pressure, meaning it feels like it needs to clean up at this point, and there's not enough memory, then it will go through and clean up the memory.

It's a different world in a garbage collector; you can't look at the memory usage on a second-by-second basis. You would have to wait for the garbage collector to kick in.

Otto: Is the maxconnection applicable in case of IIS taking to the database server? Does that mean it can make only the max number of connections to the database specified by max connections? Does that make sense?

Wade: Yes, it makes sense. No, it doesn't apply to database connections. It would apply to Web requests. There are a couple ways you can do that. The most common way that you would be making an outbound Web request is probably if you're making an outbound Web service call. You could also potentially be using some classes like WebRequest or HttpWebRequest; maybe you need to fetch some HTML off of a remote server and do something based on that file, or go fetch a remote XML file. If you use the objects like that, then in that case the maxconnection applies. But not to something like a database connection.

Otto: Under high load, are requests queued twice, and how is performance affected?

Wade: It depends on exactly what you mean by queued, but if you say that every request, even under low load, is queued once in the I/O completion port, and then taken off the I/O completion port by an I/O thread, if you count that as one queue, then it's true that it would be a second queue as the I/O thread dumps that request off into the ASP.NET internal queue and then waits for a worker thread to service it.

The performance is certainly affected. I don't have any numbers off the top of my head. As long as there are worker threads available, then it's not a tremendous performance hit to queue it up, call queues or work items, and wait for a worker thread.

It's not something that you're going to notice because, remember, you're sending HTML over the wire to a remote Web server, especially if it's a Web service request — the client is serializing, and the server deserializes and back again the other way — so compared to things like that, it's not something that you're going to notice within the total response time.

Otto: We have a Web app developer who is causing assertion failures, which causes a pop-up on the server console and the worker process to hang until we actually reply to the pop-up. Do you know if there is any way that we can prevent that?

Wade: An assertion failure would only happen in a debug build, so the first thing to recommend is that he send a release build — that's probably a pretty easy fix — and address whatever problem it is that is causing the assertion failure.

Otto: Great. Well, with that, it does appear that we've answered all the questions that were submitted to the queue today, so I'm going to wrap up the session.

If you have any feedback about the shows or WebCast program as a whole, or even some suggestions for future topics, feel free to e-mail us at supweb@microsoft.com. We would certainly love to hear from you.

We hope that everyone has the opportunity to tune in again soon. Thanks for joining us today, and have a great day.