|
Provide Feedback on this Broadcast
Microsoft Corporation
Microsoft Internet Information Services 6.0 Architecture:
How Default Settings Can Affect Existing Web Applications
July 30, 2003
Note This document is based on the original spoken WebCast transcript. It has been edited for clarity.
Dean Cron: My name is Dean Cron. I'll be taking you through this presentation today. Again, we're going to cover existing Web applications in the IIS 6.0 architecture. What I've done in the design of this presentation was go through a lot of the cases that PSS has seen since IIS 6.0 released in Windows Server™ 2003, and I looked to see what some of the recurring problems were, as far as what settings customers needed to modify for their Web applications to run IIS 6.0 properly. So we're going to take a look through those. I added a couple of additional issues that we don't currently have too much documentation on, other than in the IIS help, to kind of make people aware of things that might affect the performance of their applications and issues like that.
Let's take a look at the agenda (slide 2). We're going to start off with an IIS architecture review. I'll review the IIS 5.0 architecture, followed by the IIS 6.0 architecture, just so you have a general idea of what we're looking at going into this. Then we'll move into security settings that can affect existing Web applications. We'll talk a little bit about process recycling and health monitoring within IIS 6.0. We'll talk about a few additional settings, including some ASP and ASP.NET considerations that you should take into account when migrating applications to IIS 6.0. We'll cover links to a few key troubleshooting tools for IIS 6.0, and some additional resources, like documentation and things like that, that you can use when you're deploying IIS 6.0.
Now as we move on to the IIS 5.0 architecture review (slide 3), as you'll see here, this probably looks fairly familiar to a lot of you. Most of our services in IIS 5.0 lived in an Inetinfo.exe process, FTP, SMTP, and NNTP, as well as any in-process Web sites and ISAPI filters as well. Metabase, also in IIS 5.0, was hosted with Inetinfo. You can have the kernel-mode/user-mode transition there just to show what it did look like, because we're going to show that in the next slide, the IIS 6.0 review.
On the right slide of the slide, you'll see COM+, and in the middle you'll see the Dllhost marked pooled, that would be the medium out-of-process Web sites. Then you have a separate Dllhost process spun up for any high isolated Web sites. In addition to this, in IIS 4.0 and 5.0 we used Winsock to communicate through Tcpip.sys for any client/server communications in IIS. That's something to keep in the back of your mind as we move into the IIS 6.0 architecture review.
Now looking at the IIS 6.0 architecture (slide 4), one thing to note is that when we are running IIS 6.0 in worker process isolation mode, there are two isolation modes in IIS 6.0. Worker process isolation mode is mentioned, and there is IIS 5.0 isolation mode, which more reflects the architecture of IIS 5.0. Now the only time that you should need to use the IIS 5.0 mode would be if you have an application that isn't compatible with worker process isolation mode for whatever reason. During the course of the presentation, we'll cover some of those reasons.
So if we look at the IIS 6.0 architecture, here we're only covering the Web side of it. FTP, NNTP, and SMTP still live on Inetinfo, so there's no real change there. As far as the Web side goes, Inetinfo is used to host the metabase. W3SVC, which is hosted in an SVC host process, is responsible for configuration and process management in IIS. You'll see examples of three different application pools running their own W3WP processes.
We no longer have the concept of high, medium, and low isolated applications in worker process isolation mode in IIS 6.0. Everything technically runs in what would be considered low isolation, even though each application pool is separated by its own worker process. The reason we consider that low isolation is because the server core is now multi-instance in IIS 6.0, and now runs in each separate W3WP, as opposed to the single-instance server core that just ran in Inetinfo in IIS 4.0 and IIS 5.0.
As you'll see here, in the three app pools we have examples of, one is hosting ASP.NET ISAPI, the other two are both hosting any other ISAPI extensions; ASP, for example. Anytime that you have an ISAPI filter or an ISAPI extension in use in IIS 6.0, when you're in worker process isolation mode, those filters or extensions will be loaded in each individual W3WP. You will not have filters loaded within Inetinfo anymore.
Down below this, going back to the kernel mode and user mode portion we discussed a little bit on the previous slide, you'll see HTTPAPI in user mode and HTTP.sys in kernel mode. HTTP.sys is a new kernel-mode driver within Windows Server 2003. We hook into that for all HTTP communications between the client and the server. We no longer use Winsock on the WWW side. Now on the FTP, SMTP, and NNTP side, we still use normal Winsock communications. We do not make use of HTTP.sys with those three services. This driver is specific to the WWW side for IIS.
We've finished the architecture review, now we'll get into some security considerations when migrating applications to IIS 6.0 (slide 5). We'll start off with the Web service extensions (slide 6). In the IIS management console you may have seen a Web Service Extensions node as a subnode of the master level. In here you define which ISAPI and CGI extensions are allowed to run in IIS 6.0. If you have an ISAPI extension or CGI extension that is not defined and marked to Allowed in this node, any Web application you have that requires the execution of that extension will return a 404 to the incoming client.
In addition to modifying this in the UI, you can add your own custom extensions. You can enable any of the existing ones, such as ASP, ASP.NET, or WebDAV, any of those. In addition to modifying this in the UI, you can also modify this directly in the metabase. These settings are held within the WebSvcExtRestrictionList property. IISEXT.VBS is located in the System32 directory on Windows Server 2003. It's one of our new scripts, and you can use this to add, modify, or view extensions from the command line. You can also use this with a setup application. If you have a custom application with its own setup package, you could also use IISEXT.VBS to programmatically add or modify the status of an extension.
The KB articles listed there give examples of using IISEXT.VBS properly, when doing programmatic addition or modification of Web service extensions.
Now, as mentioned before, if we have a Web service extension, say ASP is set to prohibit it in the Web Service Extensions node, any requests for an ASP page by any client will result in the client seeing a generic 404, file not found. By default, on a clean install, IIS will log a 404.2 in the IIS hit logs. Now that protocol sub-status, here it is .2, is a new feature in IIS 6.0. We log it to allow the admin easier troubleshooting on issues like this. There are multiple reasons a 404 can be returned, and as long as protocol sub-status logging is enabled, all the administrator would need to do is look in the IIS hit log to see why a customer received a 404, for example. We'll get into the protocol of sub-status a little bit later as well, because there are a couple of other things I'm going to cover that can result in a client seeing a 404.
Here's a screen shot of the IIS Manager (slide 7), showing the Web Service Extensions node. Now note, as I mentioned in the previous slide, the Web service extension settings are global for the WWW service. These cannot be set at the site level, they are global only. If you need to deny an extension at a certain level, say ASP, then what you would do is remove that script map from that Web site or virtual directory that you do not want ASP served from. Again, that's just one example. You can add or remove any of these as needed. You can allow or prohibit them.
We also have, as you can see at the top there, the All Unknown ISAPI Extensions and All Unknown CGI Extensions settings. It's really recommended that these be used as more of a troubleshooting step. It's really not a good security practice to leave either one of those set to Allowed, because then if an ISAPI extension was loaded on the machine that the administrator was unaware of, that could be executed at will if the appropriate of these two settings was set to Allowed. So again, just use that as a troubleshooting step. If you set it to Allowed and your application starts working, you need to figure out what the file name is for that ISAPI or CGI extension that you need to have in here, and set it to Allowed.
Now we move into the known extensions feature in IIS 6.0 (slide 8). As it mentions, if a file extension is not mapped in the IIS or operating system MIME map, the file is not served. Now what that means is in IIS 4.0 and IIS 5.0 we had wildcard mapping — if you were to, in IIS 5.0, click that MIME types button, and you would get this user interface by right-clicking the master node, the server name, then bringing up Properties. You can also set this as low as the file level, and anywhere in between.
So if you were to click this MIME types button in IIS 5.0, you would more than likely see a .* mapping in there, meaning that any extension not showing up, not explicitly defined in the MIME map, would still be served by IIS, regardless. In IIS 6.0, that wildcard mapping was removed. So any file extension, again, not defined in either the IIS MIME map or the system MIME map is going to result in a generic 404 being returned to the client. In this case, if protocol sub-status logging is enabled, IIS is going to log a 404.3 to the hit logs. The .3 is specific to the known extensions feature. If you look in the "About Custom Error Messages" topic in the IIS 6.0 Help, it defines what each sub-status code means. It gives you an easy way to look it up and find out which setting may be causing the problem.
Again, as mentioned on this slide, a required extension can be added anywhere from the master level to the file level in the IISM. If you only want a certain file extension being served from one virtual directory, you only have to add it at that level. It would still be denied at all the other levels, as long as it didn't exist in the master IIS MIME map or in the operating system MIME map.
Moving on to process identity (slide 9), a clean install of IIS 6.0, we've made a small change to the security context that Web app code runs in. In IIS 4.0 and 5.0, everything would run in a system, if you were running in process, or IWAM is out of process. Now in previous versions, running as Local System caused security concerns if a hacker were to somehow take control of that process. System is obviously the most powerful account on the machine, and would allow them to do pretty much anything. So to try and protect against this, Web app code running in the W3WP processes in IIS 6.0 runs in the context of the Network Service account. It's quite limited in what it can do, so if somebody does take control of a process, there really isn't much they're going to be able to do on the box.
You can change this from Network Service if you wish. In the properties for each application pool there is an Identity tab. We have three built-in accounts that you can choose from, Network Service, the System, and I believe Local Service. You can also define a custom account, if you wish. Now as mentioned in the last bullet here, if you do use a custom identity, that identity has to be a member of the IIS_WPG group on that local IIS 6.0 server. If it is not, when the worker process tries to fire up, it will fail, throw a couple errors in the event log, and your application pool will be stopped by default, thanks to rapid-fail protection, which we will discuss in a few minutes. But again, ensure that any custom identity that you're using as a worker process ID is in the local IIS_WPG group for that IIS server in question.
Now we'll move into process recycling and health monitoring (slide 10). We'll start by taking a look at process recycling (slide 11). For those of you who have used ASP.NET in IIS 5.0, ASP.NET also had some recycling and health monitoring features built in. Those, when running in worker process isolation mode, are now controlled in IIS within the settings for the application pool. Process recycling itself allows administrators to configure IIS to proactively restart worker processes that are hosting problematic Web applications.
As listed here, you can recycle based on the elapsed time that the process has been running for. You can recycle based on the number of requests served, scheduled time of day, virtual memory used, used memory, and on-demand, through the IISM or script. If you do it through the UI, you would simply right-click the application pool and choose Recycle. That would cause IIS to terminate the worker process that was currently serving and spin up another one. We're going to talk about something in a few minutes called overlapped recycling, which would come into play here as well, so I will wait until then to talk about that a little more.
Now as mentioned, to configure recycling you would just bring up the properties for the application pool you wanted to set it on (slide 12). The first tab is the Recycling tab. In addition to setting it at the application pool level, you could also set something like template settings at the master level by right-clicking the Application Pools folder and choosing Properties. So any new application pool created after those changes were made at the master level would then inherit those settings. So as we see here, by default, the worker process recycling is only set to Recycle after minutes elapse. So for 1,740 minutes of process uptime, IIS is going to shut down that W3WP and fire up a new one.
You can also configure this by modifying the LogEventOnRecycle property in the metabase if you wish to have these events logged in the event log. By default, I believe we only log based on used memory. So if you wish to have IIS throw an event in the event log each time a process recycles based on hitting one of these triggers, you would modify the LogEventOnRecycle property accordingly. You can look up that property in the IIS 6.0 Help, and it will show you what setting you can use, depending on which events you wish to log.
When we look at health monitoring, one of the first things in health monitoring is worker process pinging (slide 13). This is enabled by default. Every 30 seconds the W3SVC will send what's considered a ping to each worker process to ensure that it's healthy. If the worker process does not respond within 60 seconds after that ping is sent, IIS will assume that the worker process is unhealthy and terminate the process. This is configurable. You can disable pinging if you wish. It is a good feature to have, though. You can also increase or decrease the frequency and, again, that frequency is listed in seconds, for how often IIS will ping that process.
Now looking at rapid-fail protection (slide 14), rapid-fail protection is similar to the concept of ASP FailFast in IIS 5.0. This is enabled by default, as you can see on the screen shot. If, by default, five failures occur within a five-minute time period, IIS is going to mark that application pool as stopped. So for any incoming requests, clients are going to receive a service unavailable message until the administrator goes in and manually starts that application pool. It kind of forces the admin to notice that something happened to that pool and they should take troubleshooting actions from there, whether it be just troubleshooting by himself or calling PSS for assistance.
You can configure this at the application pool level or, again, at the master level, if you're setting template settings for future application pools that you create. You can change the failures and/or the time period in which that number of failures occur. You can disable this if you wish, but again, it's a handy feature to have, so that if you have a frequent problem occurring with a worker process, you're more likely to notice that something is going on if your clients aren't reporting problems to you.
We're going to go through some additional settings (slide 15) now, including application pool – specific settings, some ASP settings, and ASP.NET considerations. After the application pool settings, I'm also going to be going through overlapped recycling a little bit.
Starting with application pool settings (slide 16), MaxProcesses is a setting in the metabase that corresponds to the "Web garden" setting in the IIS manager. You set this by changing the value of Web garden from 1 to any number higher than that in the Performance tab of application pool settings. The maximum number is 4 million, but resources on the box would never allow that many processes to be running. It's a limit of the OS more than anything else. This specifies how many worker processes are spun up to serve requests for that specific application pool.
For instance, if you increased MaxProcesses or the Web garden setting in the UI to 5 as requests start coming in the process, you're going to see five processes spun up to start handling requests. It's kind of load balanced across the process. You can think of it in a round-robin sort of way. Each new request is supposed to go to a different application pool. One thing to note with Web gardening, session state is not persisted across the worker processes for that application pool. So if your application requires session state and you're using Web gardening, you would need to manage session state out of process, whether it be cookie-based, like through SQL or ASP.NET, or something along those lines.
Another consideration with the Web garden settings, if you have a custom component that doesn't support running multi-instance, you are probably going to run into problems here, because it would be loaded on each individual worker process. So when it tries to fire up that second time and fails, you could start running into rapid-fail protection limits within the application pool, errors logged in the event log, and users seeing errors. That's another thing to take into consideration there.
The second setting that I'm going to discuss, AppPoolQueueLength, corresponds to the request queue limit in the IISM, and that's in the application pool's properties as well. The default is 4,000. So after 4,000 requests have been queued up in HTTP.sys, HTTP.sys would know that based on this setting. It's not capable of queuing any more up, and it would start returning "service unavailable" to any subsequent clients after that. You can configure that, again, through the IISM or by modifying the metabase directly. You can reduce or increase that as needed.
And as it mentions, when IIS starts up, W3SVC tells HTTP.sys how many requests to queue up for each application pool based on this setting. Again, after it hits that limit, after that many requests are queued up, clients start getting "service unavailable."
Now to stop DisallowOverlappingRotation (slide 17), I touched on this a little bit earlier. Overlapping rotation is if a worker process is recycled for any reason by IIS, by default, the WWW service is going to spin up a new W3WP to replace the existing one prior to that existing one being shut down. If you set this to True, the WWW service will actually kill the existing W3WP prior to the new one being started. So if we leave the defaults, some of the things to take into consideration if you have, again, a component that doesn't support running multi-instance, overlapping rotation could be a problem for it, because it will be already loaded in the existing W3WP, and then IIS goes to spin up the new one, it will try to load it a second time, which could cause problems for the application. So if your component only supports single-instance, you would want to set DisallowOverlappingRotation to false.
Now note that when IIS does shut a process down, it does give it a time limit to complete any outstanding requests. As soon as IIS detects this process needs to be shut down, requests will stop being routed there, but any existing ones will be given time to complete, by default 90 seconds. If they do not complete by the shutdown time limit, which again is 90 seconds, IIS will just kill the process anyway.
Now if you have overlapping rotation enabled, again it is enabled by default, what's going to happen is for any new requests that come in, when IIS realizes that the existing W3WP is unhealthy, they would be routed to the new process that is spinning up. If you have overlapping rotation disabled, what would happen is after IIS realizes that the W3WP is unhealthy, any new requests would be queued by HTTP.sys until the new one is fired up.
DisallowRotationOnConfigChange specifies whether a worker process is recycled if a config change is made to the host application pool. So anytime you make a change to the configuration of an application pool, by default, if you watch in Task Manager, you'll see IIS fire up a new W3WP after that change is applied, and then terminate the existing one. You can change this behavior if you wish. If you wish to control when the worker process recycles itself, you can change the setting of DiallowRotationOnConfigChange to True, and any changes that you subsequently make to that application pool would not take effect until you recycle it manually. So that's something to take into account there.
If we take a look at how overlapped recycling works (slide 18), just to explain this slide, what actually happens is W3SVC will spin up that new W3WP by default, after it detects that the existing one is either unhealthy or has reached a recycling trigger. At that point, any existing requests in the old W3WP would be given 90 seconds to complete before W3SVC terminates that process. Any new requests would be routed by HTTP.sys to the new W3WP.
So moving on to ASP settings (slide 19), one of the settings that we've seen the most questions and issues on so far is the AspBufferingLimit. There's no corresponding setting in the UI for this. It's set directly in the metabase only. This sets the maximum size of the ASP response buffer. By default, it's 4 MB. If the size of the response buffer exceeds this limit, the buffer is flushed and the client receives an error. The client is generally going to receive a 500 error. If you disable friendly error messages in Internet Explorer, you'll actually see a much more descriptive error indicating that the response buffer limit was exceeded. You can modify this to an increased number if you wish. This was put in there for performance reasons to ensure that we didn't have a single response eating up a ton of available memory. You can increase this as needed. Again, I would only recommend increasing it if you do need it. It's set there to ensure that your performance isn't hindered by slow and sizable responses.
AspMaxRequestEntityAllowed specifies the max number of bytes permitted in the entity body of an ASP request. By default, this is 200,000 bytes. You can modify this if you wish, if you know that your requests are going to exceed that. We've had a few questions on this. I don't know that we've actually seen a case on in yet, but because we've seen questions on this, I threw this one in. It's just something, again, to keep in mind when deploying applications on IIS 6.0.
If we move on, there are more ASP settings (slide 20). The next one is AspRunOnEndAnonymously. By default in IIS 6.0, the SessionOnEnd and ApplicationOnEnd functions in global.asa now run under the context of the anonymous account or IUSR_machinename, or whichever one you have tied to your specific application. If you have a custom one, they run into that account. Previously, these ran in the context of the host process identity. This change, again, is new to IIS 6.0. So if you wish to revert back to the default behavior, you would set AspRunOnEndAnonymously default.
Now AspExecuteInMTA, this one is, again, new to IIS 6.0. I've seen a few questions on it, but no issues so far. This allows ASP to execute in multi-threaded apartment (MTA). In previous versions, ASP would only run in single-threaded apartment (STA) by default. If you want to run in MTA or multi-threaded, assuming that any COM components running in there are either free-threaded or both-threaded to support this, you could flip AspExecuteInMTA to True, and you might get a little performance boost out of your ASP applications. You can set this at the global level, the site level, or the virtual directory or physical folder level. So you can specify which applications you want to run in multi-threaded as opposed to single-threaded.
Looking at some ASP.NET considerations (slide 21), the Microsoft .NET Framework version 1.1.4322 is what ships with Windows Server 2003. If you have an application that needs the 1.0 framework, you would need to install that manually. As it mentions in the second bullet there, the 1.0 framework is only supported in IIS 6.0 when you are running in IIS 5.0 compatibility mode. A lot of the changes in the 1.1 framework were implemented to ensure compatibility with worker process isolation mode, and you may run into some issues, such as view state corruption, if you're running the 1.0 framework in worker process isolation mode.
Now there's a KB article listed there that talks about the support boundaries. Again, the version 1.0 framework is only supported in IIS 5.0 compatibility mode. If you want to run your ASP.NET application in worker process isolation mode, for it to be supported you would need to run the 1.1 framework.
One additional thing to note is if you decide to install the 1.0 framework on Windows Server 2003, you'll need to go to the Web Service Extensions node and manually add the ASP.NET ISAPI for the 1.0 framework for it to be served. Otherwise, you're going to have clients receiving 404s, and you'll see the 404.3 in the IIS hit logs.
In worker process isolation mode, ASP.NET will run in the context of the IIS 6.0 process model, not the ASP.NET process model that you would have seen in IIS 5.0, or if you're running in IIS 5.0 compatibility mode, not IIS 6.0. When you're running in worker process isolation mode, any ASP.NET apps are going to be running in W3WP processes, not ASP.NET. In addition, we ignore all but three settings in the machine.config. The three settings that still are effective in worker process isolation mode, within machine.config, would be maxWorkerThreads, maxIoThreads, and ResponseDeadlockInterval. Short of that, everything else would be ignored. You'd be using the IIS 6.0 process model completely, including the recycling/health monitoring settings.
If for some reason you wish to run it in the ASP.NET process model, you would need to flip IIS to IIS 5.0 compatibility mode. To do so, you would simply right-click the WWW sites folder in the IIS management console, and in the Services tab check the box stating Run in IIS 5.0 Compatibility Mode. Now that is a setting that's service wide for the WWW service. You cannot choose to run specific sites in IIS 5.0 compatibility mode and specific sites in worker process isolation mode. It's global for the WWW service only.
Looking at some key troubleshooting tools in IIS 6.0 (slide 22), this just takes into account things you should keep in mind in addition to the existing troubleshooting methods that you would know from IIS 4.0 and 5.0. In IIS 6.0, the IIS hit logs become even more important because of the addition of protocol sub-status logging. Again, on a clean install of IIS 6.0 that is enabled; on an upgrade, you'll need to enable it manually. If you want to figure out what each sub-status code means when you're looking at the hit logs, you can go into the IIS 6.0 Help; the easiest way to get there, if you have the IIS management console open, is press the F1 key, and it will pop up the Help. If you search for "About Custom Error Messages," there's a table in there that defines the sub-status codes for all protocol logging errors: 403, 404, 401, and so on.
In addition to the IIS hit log, HTTP.sys contains its own error log. This is located in a similar location to the IIS hit logs. Under the Logfiles folder, in addition to the W3 log folders, you'll see an Httperr folder. This will contain one or more files that will show you any requests that HTTP.sys itself rejected, and it should give you a status code and a status code reason to the right of status code. The definitions of the phrases logged in that error log are available in the KB article listed here, 820729. It can give you a general idea of what each code means, and it is extremely helpful, especially in diagnosing 503 server unavailable errors the clients are seeing. When you see a "server unavailable" error, you can pretty much assume that was returned by HTTP.sys, and check in the HTTP error log to get an idea of why that was returned.
503s can happen for several reasons: the application pool is stopped; clients will see server unavailable if the application pool has hit its rapid-fail protection trigger and has been shut down. Again, clients would see "server unavailable" if the queuing limit is reached for the application pool. Again, default is 4,000; after that is reached, the clients sees "server unavailable." That KB article listed at the bottom provides pretty good definitions of why you would see each of the reason codes listed after the protocol status.
Looking at more key troubleshooting tools (slide 23), the IIS 6.0 Resource Kit Tools. This one includes tools to troubleshoot SSL issues and permissions issues, among others. There are several tools in that kit that are extremely helpful in troubleshooting IIS 6.0 issues. There's a download link here (http://www.microsoft.com/downloads/details.aspx?FamilyID=56fc92ee-a71a-4c73-b628-ade629c89499&DisplayLang=en). You can also search for "IIS 6.0 Resource Kit Tools" on Microsoft.com, and search results should include that link as well.
In addition to the resource kit, we have the IIS 6.0 Online Help (slide 24). Anything available in the IIS 6.0 Help .chm file that ships with IIS 6.0 is available through this link on the TechNet site, to kind of help you out in case you're not in front of an IIS 6.0 server at the time, and can't pull up that Help. In addition, we also have an IIS portal off of Microsoft.com. So if you go to http://www.microsoft.com/iis/, there are several resources, whether they be links to deployment guides, the resource kit, the online help, or other white papers. There's a lot of information up there that can be useful when troubleshooting issues, or if you just want to learn more about IIS.
The IIS 6.0 Deployment Guide is out there as well. It can help you whether you're deploying IIS for the first time, migrating from other platforms, such as Apache, or migrating from previous versions of IIS. In addition to this, one resource that released a couple days ago, so it didn't make the slide deck, is the IIS 6.0 Resource Kit Chapters. I think there's a link off of the http://www.microsoft.com/iis/ site for this. You can go there and download the actual chapters from the IIS 6.0 Resource Kit. That can be extremely useful, in addition to the resource kit tools.
We've reached the end of the presentation, which brings us to the Q&A.
Otto Cate: Great. Thank you very much for that presentation. Before we jump into the Q&A session, I'd like to share a couple of quick program notes with our listeners. If any of the details on the PowerPoint® slides were difficult to view in your browser today, or you'd like to simply have a copy of the slides, they are available for download from the Web site. Simply click today's WebCast on www.support.microsoft.com/webcasts (under Past Support WebCasts). In addition, this is the area in which you'll find the on-demand streaming media archive.
The Q&A portion of the Support WebCast is intended to encourage further discussion of the topic that we addressed here today, surrounding IIS. In addition, one-on-one product support issues are really outside the scope of what we're able to address. So if you do find that you need some more complex technical assistance, feel free to contact a Product Support Services support professional, either by phone or through a Web incident.
With that, let's jump into a few questions. Is the Prohibit all Web service extensions a one-time configuration adjustment, or does it set a flag and stay that way? For instance, what happens if an application installs an extension later?
Dean: If you click that Prohibit all Web service extensions link, it will prohibit anything installed at that time. So it would be a one-time configuration adjustment. Anything installed after that, if you install any other application that adds its own Web service extension, if that setup program sets that specific extension to Allowed, then Prohibit all Web service extensions set previously would not affect that.
Otto: If I have four processors in my server, how many aspnet_wp processes will be launched under Windows® 2000?
Dean: That's a good question. I'm not as familiar with the ASP.NET side of this. That's probably something that I would have to research and get back to you on.
Follow-up answer: There is no real hard number that can be determined for this. Processor availability is only one factor. There are other resources that can come in to play that limit the number of aspnet_wp processes on a single server. For example, the number of overall processes running on the box, how much memory is available, and so on. One of the most common limitations we see in IIS 5.0 for the number of out-of-process Web sites (individual Dllhost.exe processes) is the amount of desktop heap used. For one example of what can happen when desktop heap is depleted (for example, too many running processes), see 126962.
The short answer is there is no way of determining exactly how many aspnet_wp processes you can launch on a single server, because there are several factors that could affect this.
Otto: The next question: Is there any way to run IIS 6.0 on Windows XP Pro for development?
Dean: No. Currently IIS 6.0 is only available on Windows Server 2003, and IIS 5.1 is the only version available on XP Pro. If you needed to have IIS 6.0 installed for development, you would need to do the development on a Windows Server 2003 box at this time.
Otto: We're basically looking for a better dashboard interface. We're wondering if there is an "at-a-glance" GUI for monitoring your IIS health and processes, besides relying on Task Manager.
Dean: In addition to Task Manager, you could use the old standby PerfMon. I'm sure you have used that quite a bit. In addition to Performance Monitor, I haven't played with Windows Server Resource Manager in quite a while, and I don't know if it has monitoring capabilities, but that could be something you'd want to look into. Microsoft Operations Manager also has a Service Status Monitor application. It monitors certain health settings throughout the server, including some of the IIS 6.0 settings. That may be something else to look into as well. Again, that one is called Service Status Monitor. It ships with Microsoft Operations Manager.
Otto: When auto restarting an app based on memory usage, is the check for memory an average or a snapshot? I can see a case where the memory usage might spike, but the average may not necessarily be a problem.
Dean: Generally, IIS should, and if it's just like a momentary spike that goes away, that should not be something that restarts it. If the memory goes up past that limit for a minute or longer, possibly, IIS may in fact recycle that site. It checks the status. Again, it's something like every 20 or 30 seconds of each process to see what its stats are based on what the recycling triggers for that app pool are set to. If the memory limit is up at the level, then when the status is checked it is quite possible, even if it was a short spike, that IIS would recycle that process.
Otto: How does the new kernel driver help to keep out malicious code? Or do I look elsewhere to protect my servers?
Dean: The kernel mode driver itself isn't really there for security; it's there more for performance. No Web app code actually runs in HTTP.sys. So you don't have to worry about any Web app code taking down HTTP.sys. As far as performance gains, by not utilizing Winsock in user mode, and moving all that communication up to kernel mode into a driver that's not used as heavily as Winsock, we gain quite a bit of performance. But again, HTTP.sys wasn't really added for security, because we don't run that user code within HTTP.sys itself.
Otto: I have a couple of questions on the lockdown tool: The lockdown tool version 2.5, is that compatible with IIS 6.0?
Dean: The lockdown tool itself, no, simply because those lockdown settings were migrated into the Web Service Extensions node in the UI, so you can lock specific applications down. Now there is a separate 2.5 release of URLScan that you can install on IIS 6.0 if you wish, just to provide a little bit of additional security. But the lockdown tool itself, no. The lockdown tool is not for use in IIS 6.0, just URLScan.
Otto: Would the IIS lockdown tool for IIS 5.0 need to be removed before upgrading to Windows 2003 in IIS 6.0?
Dean: The upgrade process should actually take care of that.
Otto: Great. You mentioned several IIS settings that may only be modified through the metabase directly. Are there tools in the resource kit to do this, or what is the recommended way to do that?
Dean: There are a few ways you can do this. There is a tool that ships with the res kit. If you are familiar with MetaEdit, we used it a lot with IIS 4.0 and 5.0. That has been replaced with a tool called Metabase Explorer that ships with the IIS 6.0 Resource Kit to provide you with a kind of GUI of the metabase, so you can find settings easily and make changes in there. If you do not wish to use MB Explorer, there are a few other ways to do this. We still ship Adsutil with IIS, and it's still located in the C:\Inetpub\Adminscripts folder, by default. You can use Adsutil from the command line to change any metabase settings you need to modify.
In addition to this, in IIS 6.0 we've migrated from a binary metabase to an XML-based metabase. So it is also possible to modify the metabase directly by opening it in a text editor, such as Notepad. There's a setting in IIS that controls whether or not you can edit the metabase directly while IIS is running. Enable Direct Metabase Edit is the setting. You can modify it by bringing up the properties of the master node. After that is set, you could then go in, open Metabase.xml in a text editor, make any change that you needed to, and then save it.
IIS monitors change identifications on most of the keys within the metabase, so it detects that you've changed the .xml file, it will actually propagate that change to the in-memory copies so that they take effect immediately. Again, your options there would be to edit directly, use Adsutil, or use MD Explorer. Those would be in addition to writing any custom scripts, using either ADSI or WMI to do that for you.
Otto: Could you briefly summarize the primary changes and benefits of IIS 6.0 over version 5.0?
Dean: Sure. We have much tighter security in IIS 6.0, out-of-the-box. Again, that assumes a clean install. If you're upgrading from IIS 5.0, we're not going to break any functionality, so you're not as tightly locked down after an upgrade. In an upgrade, you're also running in IIS 5.0 compatibility mode. On a clean install, you'd be running in worker process isolation mode.
So on a clean install we're quite a bit more secure than IIS 5.0 was. We, again, run the process in the context of an account with a much lower privilege level than we had with System in IIS 5.0. Again, we run as Network Service now. We have quite a few performance gains in IIS 6.0, a lot of it through the use of the HTTP.sys kernel driver, as opposed to using Winsock for our HTTP communications, as we did in previous versions. That should provide some performance benefit.
In addition, if you're running SSL sites, with worker process isolation mode in IIS 6.0, SSL handling for IIS is moved into LSASS, which is the OS security process, to shorten it. This provides a few performance gains because we don't have a lot of the overhead sitting in Inetinfo, like we did before. You're running it in the context of a kernel driver in LSASS. So it should provide a little bit faster SSL communications.
Otto: We have another GUI question: Will IIS 6.0 have a GUI available to show current users and current pages that are being served?
Dean: No. We don't currently have the capability to show, as you mentioned, the current users and current pages. The only real way to track the number of current users would be something like PerfMon. On the FTP side you can see who's logged in, but on the Web side, no, we don't currently have a way to track those.
Otto: Will the IIS 6.0 worker process mode replace the need for the machine.config file for .NET apps?
Dean: Yes, it will, with the exception of three settings. There are three settings that ASP.NET still reads in: the maxWorkerThreads, maxIoThreads, and ResponseDeadlockInterval. Beyond that, machine.config is not used. However, I would not recommend removing it, if that's what you were considering. If ASP.NET does need it — for example, if you were to switch to IIS 5.0 isolation mode — then ASP.NET would need to read that file in again, in its entirety. But yes, with the exception of those three settings, that file is not used in worker process isolation mode.
Otto: I ran two security patches, MS03-018 and MS02-015, both for IIS 5.0, and since then I cannot load ASP pages. Do those pages lock down the server in some way, or is that something you've seen?
Dean: Off the top of my head, I don't remember exactly what those two do, but I can't recall any specific security patches that do that. It's more of a lockdown thing. We'd probably need more information on that, like what the clients get in return. That may be something that you'd want to post to the IIS newsgroups as well, because we have a lot of our support people who monitor issues in the newsgroups and who respond as soon as they see them come in.
Otto: Right. It sounds like that may actually be a really good option. In addition, it might be a good idea, if the problem persists, to open up an incident on that and see if that's something that we've seen with version 5.0.
The next question: If a process is waiting for a database query to be returned for over the recycle time limit, would the status of that process show up as active or show as inactive and end up being recycled?
Dean: If you have a request waiting, a long-running query like you mentioned, and for some reason IIS hits another recycle trigger, whether it be time limit, scheduled time, number of requests, whatever, that request will be given 90 seconds to complete from the time that IIS realizes that it needs to recycle that process. If the request has not completed, IIS will terminate that process and, again, all future requests would be sent to the new W3WP. So that outstanding request would be lost.
Otto: Which OS is version 6.0 compatible with?
Dean: Version 6.0 ships with and only runs on Windows Server 2003. It's not currently available on Windows XP or Windows 2000. It wouldn't be on 2000 anyway, simply because of the architecture; there's no way of implementing the overall architecture, HTTP.sys and everything, on Windows 2000. Currently, IIS 6.0 is only available on Windows Server 2003.
Otto: The next question: Where is the default SMTP virtual server, or is that no longer a part of IIS?
Dean: SMTP is still part of IIS. You install it in the same way that you did previously, through the Add/Remove Windows Components section. If you've installed IIS 6.0, you may have noticed that we no longer are at our own root node in Add/Remove Windows Components. We are a sub-node of Application Server. After you get in there, you can install SMTP, and that is available within the IIS management console. You should see it show up just as it did before.
As far as SMTP goes, there were very few changes to SMTP between version 5.0 and version 6.0. That root node in Add/Remove Windows Components would be Application Server. So if you highlight that and click Details, IIS sits under that, and things like SMTP, NNTP, FTP, and the Web services are all sub-components.
Otto: Is there any documentation that would walk me through changing IIS 6.0 to IIS 5.0 mode?
Dean: Within the IIS 6.0 Help, which again you can invoke in the management console, that's probably the easiest way. If you're in the management console, just press the F1 key and IIS Help pops up. You can search on IIS 5.0 compatibility mode, or you can search on compatibility mode. You should see something in there. But to put it simply, all you would need to do in the management console would be to right-click the Web sites node and bring up the properties. There's a Service tab in that property sheet, and the first selection is Isolation Mode. There's a check box for Run WWW Service in IIS 5.0 Isolation Mode.
To put yourself in IIS 5.0 isolation mode, all you do is select that check box and click Apply. This is one of the few settings that requires a full restart of the IIS services. So after you check that and click OK or Apply, either one, it's going to prompt you to restart the IIS service. If you do not, you're going to be running in some kind of pseudo mid-level state. It's really recommended that you restart IIS as soon as you make that change.
Otto: Great. Thank you very much. Is there any documentation, articles or white papers, for instance, that detail all of the metabase.xml settings? Or is there something that maybe details the objects that are available in this file?
Dean: Actually, the IIS Help in IIS 6.0 is probably the best location to look. We have full representation of all of the metabase settings. In the About the Metabase section, that will show you links to the remove settings, some of the new settings and, again, there is a reference to all of the metabase settings within metabase.xml, within the IIS Help. So that is probably the best reference for it. There are way too many to put into one KB article.
Otto: Here's another question regarding watching live activity. There was an old WebMon tool in the IIS 4.0 resource kit. Will that work at all, or do you know of any other resources, or maybe upcoming resource kit tools? I know we kind of touched on this a little earlier.
Dean: I don't know about current tools. The tool that you mentioned for IIS 4.0, I'm not totally sure if that would work in IIS 6.0, because I've never actually used it. I'd probably need to follow-up with you on that. Things that you might want to look into, though, would be either Windows System Resource Manager or the Service Status Manager in Microsoft Operations Manager, depending on what types of things you want to monitor.
Follow-up answer: There may be some confusion here. Per the IIS 4.0 Resource Kit tool page, we didn’t ship a tool called Webmon. See:
http://www.microsoft.com/technet/treeview/default.asp?url=/TechNet/subscriptions/current/masterlist/na/cd29.asp
There was a separate tool available on MSDN® called Webmon, but this was only used to ensure that a site was up and running (this functionality is built into IIS 6.0 – it's worker process pinging). See:
http://msdn.microsoft.com/msdnmag/issues/0700/webmon/default.aspx
Httpmon is more like Webmon, in that it monitors the status of the server (are the specific Web sites being monitored still responding, for example). WebCheck, which was also in the IIS 4.0 resource kit, does not appear to function in IIS 6.0. The only other reference I’ve found to a tool named Webmon was one that shipped in an older Exchange resource kit, but that doesn’t seem to fit the description of what you are looking for. I’ve checked with a few other people on my team as well, and none are aware of a tool (at least a Microsoft tool) that will show you activity real-time (current users [usernames] hitting the server, for example).
Otto: Are all the new functions and settings available through a command-line? I'm thinking specifically of reporting scripts showing current config of various pools and apps.
Dean: Yes. The example you mentioned of monitoring the configuration of app pools, yes, you could easily write a script utilizing either our new WMI provider or the existing ADSI provider to check up on it whenever you wanted, to check up on the settings of each application pool and report them back to you. As far as documentation on the WMI provider for implementing something like that, there is some information in the IIS Help. In addition, the platform SDK has most of our programming references surrounding both our ADSI and WMI providers for IIS 6.0.
Otto: Is there a documented procedure for upgrading the Windows 2000 Server to Windows 2003, but at the same time doing a clean install of IIS 6.0? For instance, uninstall IIS 5.0 from Windows 2000, then upgrade to Windows 2003, and then add/remove IIS 6.0, or something of that nature?
Dean: As far as doing an OS upgrade and uninstalling and reinstalling IIS as part of that, I don't know if we have anything specific to that in the IIS 6.0 Deployment Guide. I know that we have some upgrade scenarios listed in there. So you may want to check out the IIS 6.0 Deployment Guide. Again, you can click that link off of http://www.microsoft.com/iis/ to see if that addresses it. In addition to that, the IIS 6.0 Resource Kit chapters may address some additional questions on that for you as well, so you may want to check both of those out.
Otto: What is the recommended way to handle session state, if the worker process recycling is implemented across multiple production servers? Cookies are no good in that situation.
Dean: If cookies are not an option, it's going to depend on the type of application it is. If it's ASP.NET, probably the best way to handle it would be through the SQL session state capability with ASP.NET, or depending on what the application is, some sort of custom session state monitoring. I'm not as familiar with any of the other options outside of cookies and SQL. That may be something that you'd want to post to the IIS newsgroups as well, to try to get more eyes on it and potentially more feedback.
Otto: Will the current debugging tool, such as ADPlus and CDB, for example, work as they do today on IIS 6.0 Windows 2003?
Dean: Yes. The only changes you really need to make are, first, the -iis switch in ADPlus would not dump everything you needed in worker process isolation mode. You'd actually want to monitor the specific PID instead, so you'd use the -p option with the process ID of the specific W3WP process following that -p. If you just run the -iis, all it's going to do is look for Inetinfo and Dllhost, and we don't use Dllhost in worker process isolation mode. It pretty much isn't going to be monitoring anything useful there. Because we don't execute any Web code in Inetinfo, for pretty much anything other than the metabase, as far as WWW in concerned, you really wouldn't get any good data from the added information dump, either. So again, you'd want to make sure to use the -p option instead of the -iis when you're debugging IIS 6.0 in worker process isolation mode.
Otto: Could you please explain W3WP?
Dean: Sure. Think of that as the replacement for Dllhost, for sites running in worker process isolation mode in IIS 6.0. This is actually a process file itself. So when you're running a site in IIS 6.0 worker process isolation mode, and you open up Task Manager, you'll see at least one W3WP.exe process spun up, hosting each application pool that you have on your box.
Again, the number of W3WP processes that you're going to see spun up is going to depend on a couple factors; first, how many sites you're running and if they're running in separate application pools. If you're running all sites in the same application pool, and you have the Web garden set to 1, which it is by default, you're only going to see one W3WP spun up serving those sites. But if you separate each of those Web sites off into their own application pool, kind of isolating them from each other, you would see one W3wp.exe process for each Web site that's running. So it's the equivalent of looking for the Dllhost processes for medium- and high-isolated applications in IIS 5.0.
Otto: Are FrontPage® extensions still necessary?
Dean: It depends on what you're doing. FrontPage Extensions 2002 ship with IIS 6.0. You can choose to install them or not. If you have a site that's using FrontPage, obviously you would need to install the extensions and extend any FrontPage Web site with the extensions. If you're using it for, let's say, a Visual Studio® .NET application, if you're using that method to connect to the Web server, then yes, you would still need those.
These extensions are a little bit different than the 2002 versions that you can download off the FrontPage site. So the 2002 extensions that you can download would not be the ones you would want to or be able to install on IIS 6.0. You would want to use the ones that are built in, which you select in Add/Remove Windows Components.
Otto: Are there any new outbound SMTP security features in version 6.0 that may affect the use of SmartHost for relaying?
Dean: To be honest, I'm not 100 percent sure on that one. My team doesn't work with SMTP and NNTP a lot. I don't think I have it installed here. I'd probably need to follow up with you on that question just to ensure that I get you the right answer; but my gut would be I don't think so. But that's something I want to verify before giving you a final answer.
Follow-up answer: In looking at the default settings for SMTP in Windows Server 2003, it looks like security hasn’t changed too much. Anonymous access is still the default authentication method, and anyone who successfully authenticates to the SMTP server can relay. So because anonymous is enabled, anyone can relay (out of the box). You should have the same functionality level out of the box with SMTP in Windows Server 2003 that you did in Windows 2000.
Otto: Are there tools available to migrate an entire Web site from one server to another? For instance, from development to staging and then to production?
Dean: Yes. In the IIS 6.0 Resource Kit we have a tool called the IIS 6.0 Migration Utility. You can use that to migrate from IIS 4.0 or IIS 5.0 to IIS 6.0, or from one IIS 6.0 to another IIS 6.0 box. So yes, that would be probably an extremely useful tool in the scenario you're speaking of. You could use it to migrate both the content and the metabase settings particular to those sites.
Otto: Do you know if there is any detailed documentation on the version 2.5 lockdown tool that might offer recommendations for specific settings?
Dean: As far IIS 6.0 goes, as mentioned before, you don't use the lockdown tool itself. You can only use URLScan with IIS 6.0, because most of the lockdown features are implemented in IIS 6.0 by default. But if you're looking for lockdown settings for IIS 5.0, usually the documentation that ships with lockdown contains the best. There is also a white paper on the recommended security configuration for IIS 5.0. I don't think we have one for IIS 6.0 yet, but there is one out there for IIS 5.0. So that may be something that you want to look up on www.microsoft.com/, and there may even be a link to it off of the www.microsoft.com/iis/ site. That white paper, or the help that ships with the lockdown tool, would probably be your best resource for that.
Otto: Is it possible to restart/manage IIS 6.0 using C# .NET?
Dean: Yes. How you implement it in C# is going to depend on your exact needs. If you need to restart all of the IIS services at once, you could simply use a C#-based application to invoke Iisreset, which would restart everything with the exception of HTTP.sys. Iisreset does not touch the HTTP.sys driver. If for some reason you needed to restart HTTP.sys, you'd run a NetStop HTTP/NetStart HTTP, which would require all of the IIS services to recycle as well. But as far as doing it from within C#, again, if you want all of the services, you can invoke Iisreset through your application. If you just want to restart app pools, the easiest way in C# would probably be to call the WMI provider to take care of that for you. I know there are VBScript samples and JScript® samples within the platform SDK. I don't think we have any C# samples out there, but I know it can be done by using WMI in C#. That would probably be the best way. I'd take a look in the platform SDK for more references on that.
Otto: We have a question on the migration tool and the resource kit. When using the IIS 6.0 Migration Tool Resource Kit utility to migrate version 4.0 sites to version 6.0, the IIS 4.0 site will only run in IIS 5.0 compatibility mode on Windows Server 2003. Can I take that version 4.0 – migrated site on IIS 6.0 and install and run it correctly under IIS 5.0 on a Windows 2000 server? Because I don't want to necessarily run the site on Windows Server 2003 in version 5.0 compatibility mode.
Dean: If you wanted to migrate that to IIS 5.0, obviously you've seen the migration utility will only migrate to IIS 6.0 from previous versions. I know there are some third-party migration apps out there that do something more along the lines of migrating to previous versions. As far as problems you may be running into when migrating the site from IIS 4.0 to IIS 6.0, there are a few reasons why an application would not function in worker process isolation mode. A few of those being if you have an ISAPI filter loaded on any of the sites that register for read raw notifications, that ISAPI is not going to be able to run in worker process isolation mode. The only mode that read raw notifications by ISAPI filters are supported in is IIS 5.0 compatibility mode.
In addition to that, there could be a few other reasons, such as the application is written to expect a certain identity such as IWAM or something along those lines. If you have a site that's expecting to run in the context of IWAM, you can test that by switching the process identity for that worker process in the application pool setting to the local IWAM account and see if that works. Those are a couple of the reasons that it might fail.
There is more information on that in both the IIS 6.0 Resource Kit chapters and the IIS 6.0 Help. As far as migrating from IIS 4.0 to IIS 5.0, I don't think we have anything quite as solid as the current migration utility. I know we had an older migration utility. I haven't played with it in a long time, so I don't quite remember how well it would do with migrating from IIS 4.0 to IIS 5.0. There are a couple of freeware utilities out there that seem to work pretty well, from what I've heard. So you may want to search on the Internet for those.
Otto: There are only a couple of questions left here: Are the default accounts of ASP.NET and the IIS metabase the same as the Network Service account? If this is the case, are there any security measures that we should take for ASP.NET applications to have access or not to have access to the metabase?
Dean: With IIS 6.0 running in worker process isolation mode, because you're going to be running in the context of Network Service, even with an ASP.NET application, are you going to be running in that W3WP? You're not going to be taking the ASP.NET process identity into account. That setting in the machine.config is ignored, and you're running in the context of the user identity set in the Identities tab of the application pool. If you customize that account to any degree, you would want to make sure that the account you specify in the Identity tab is a member of the IIS_WPG group on the local machine. That should give you everything you need, everything that the process would need to access the metabase.
Beyond that, first of all, if the identity is not a member of IIS_WPG, it's not going to be able to spin up the worker process anyway. So that is a general requirement. But if you have another account that for some reason needs access to the metabase, you can use the MD Explorer tool that ships with the IIS 6.0 Resource Kit to open the metabase and modify ACLs on specific properties in the metabase. That may be an option. But I would recommend just using the built-in IIS_WPG group for any account that needs access to the metabase.
Otto: After an upgrade from Window 2000 IIS 5.0 with FrontPage Server Extensions 2002 Team Services, basically upgrading this to Windows Server 2003 IIS 6.0, does any other software need to be installed? And will the upgrade process kill or remove those extensions?
Dean: No. The extensions would be upgraded to match the same version of the 2002 extensions that ship with 2003. One thing to take note of, we've seen a lot of this from the SharePoint™ teams lately, SharePoint Team Services version 1 and SharePoint Portal Server version 1 aren't truly supported on IIS 6.0 and Windows Server 2003. The version 2 of those products should be releasing. I'm honestly not sure when; I know they are in beta right now. SharePoint is in version 2 beta, and that will be the version that's truly supported.
Now as far as removing the extensions or any of that, it's not going to remove any of that, and theoretically, assuming you're not running into a compatibility issue with STS version 1 and IIS 6.0, that should still function after an upgrade. None of that is going to be removed.
Otto: This might be a question that might be better answered by this third party, but I'll ask it here just in case: Do you have any knowledge about using Netegrity's Site Minder under IIS 6.0? It runs under IIS 5.0, as an ISAPI filter that intercepts all the calls and it integrates and acts on them according the Site Minder settings.
Dean: That was a big topic of discussion right before release. There is an issue with Site Minder, and I can speak to what I know of it. Site Minder is not going to function properly in worker process isolation mode, simply because that ISAPI attempts to register for read raw notifications. Again, in worker process isolation mode, read raw notifications by ISAPI filters are not supported. Site Minder is aware of the issue, and the last I heard they were working on an update. So you probably would want to talk to Site Minder more about it, but I do know that it's probably not going to function correctly in worker process isolation mode. You may have better luck if you flip IIS 6.0 to IIS 5.0 compatibility mode, but I honestly am not 100 percent sure on that.
Otto: Great. Thank you very much. With that, we've wrapped up all the questions that have been submitted to the queue today, so I'm going to close out the session.
I want to thank everyone for joining us, and I hope that all this information was useful. I certainly want to thank Dean for coming out and giving us a great presentation.
We hope that everyone has the opportunity to tune in again soon. Thank you, and have a great day.
|