|
Do you find the Support WebCast transcripts helpful? Let us know!
Microsoft Support WebCast
Microsoft Internet Information Services (IIS) 6.0, UNC, and Remote Storage
February 26, 2003
Note This document is based on the original spoken WebCast transcript. It has been edited for clarity.
Otto Cate: Richard Ersek is a Lead Program Manager with the Microsoft® IIS team and the Windows® Server product group. In his eight years at Microsoft he’s helped to develop component services and IIS, and several versions of the Windows operating system. Thanks for joining us today, Richard.
Richard Ersek: Thanks Otto. I also have with me David Cox. David is a tester in our group and he has contributed quite a bit to the whole scenario of UNC and IIS configuration. I’m glad he’s here, and if you have any questions for us, we’ll be glad to take them at the end of the presentation.
Here’s a quick overview of what we’re going to cover (slide 2). First, I want to give you an introduction to UNC or network-attached storage (sometimes called NAS) and what we’ve done in IIS 6.0 to support that. We’ll spend a little bit of time talking about authorization and constrained delegation details in Windows Server 2003. Next we’ll jump into some considerations about scale and improvements that we’ve made in IIS 6.0 to improve our scalability and performance in this scenario. Finally, we’ll walk through a couple of real-world configuration scenarios that use IIS with remote storage.
To get started, let’s talk a bit about UNC and NAS support in IIS 6.0 (slide 3). I think most Web service administrators are familiar with NAS by now. It’s network-attached storage, and at its simplest, you’re just talking about a Windows server that’s dedicated to file serving (slide 4). This is something that, in essence, has been around since the beginning of Windows Server and Windows NT®.
It can also take other forms, such as appliance devices. Those can either be Windows-based or non-Windows-based. In the case of non-Windows-based servers, you may have servers that are using UNIX or a variant of UNIX and are emulating the server message block (SMB) file system.
In the Windows case, when you’re using network-attached storage, your storage is mapped as UNC shares. UNC, of course, stands for universal naming convention, and this is the way that we refer to locations of storage that are mapped and available as shares that can be connected to by clients. In the common case, you have \\myserver\myshare, and you can do a NET USE to connect to it.
There are numerous reasons why people like to use NAS in their architecture. One that’s attractive in particular is the ability to centralize the administration, backup, maintenance, and monitoring of your storage. And especially these days, as organizations are looking to consolidate their servers, reduce overlap, and reduce the duplication of storage to make the best use of their resources, this plays in very well.
NAS is also a very good match if you have a blade server architecture. Blades are very low-profile CPU cards that plug into a rack, and they have a full computer on them with storage and memory. They have fairly minimal storage on that board. If you have a lot of content on your Web server, you can’t put all that content, typically, on the onboard disk. You’re using that primarily for the operating system and some miscellaneous files. So you’re relying on remote storage for the bulk of your content in that scenario.
Another benefit of NAS is that if you have multiple front-end servers, you want to avoid replicating content to those servers. And that can take time. For example, there’s a lot of administrative time needed, and replication time needed, if you do use replication. The benefits of replication are that you have redundancy. You have multiple redundant copies. But the drawbacks are the logistics of having to push the files around to each of the servers. So NAS gives you a way to centralize the storage.
The drawbacks of NAS are, first of all, latency. Any time you use remote storage, you have to go off of the local server and out to the network to access your storage resources, to access your content. It can be a single point of failure, in that if you have multiple servers, now you have just one location where your storage is located.
Now, NAS vendors have done a lot and the Windows architecture has done a lot to minimize that single point of failure, and some ways around that are with Distributed File System (DFS) or NLB clustering, or Microsoft Cluster Server (MSCS) clusters. You can use any one of those, along with a remote file server in IIS, to provide improved reliability.
NAS can sometimes be expensive, if you’re going with a storage vendor solution. Usually the cost of that is the cost of the disks plus the cost of the storage device that you’re plugging that into. It doesn’t have to be expensive. There are also low-cost alternatives, and the lowest-cost alternative may often be to just configure your own server as a dedicated file server.
In IIS 5.0 we supported the ability to connect a Web server to remote storage. But there were several limitations to this. We did provide the ability to connect a Web site or a virtual directory to a UNC share. When you did that, we prompted you for your user name and password credentials. Those credentials were used for every single access to that share. And that was a simplification, but also a drawback if you have multiple sites on that share, each of which you’d like to secure individually.
In that case you don’t necessarily want to have the same set of credentials used for authorization, because that could allow one site’s user to browse down to another site’s content, and that’s something you’d want to prevent. There are also numerous limitations, both in terms of IIS implementation and the implementation of authentication and authorization, and also in the implementation of the file system that limited scale. And in Windows Server 2003 we’ve done quite a bit of work to identify and fix those issues.
To summarize what we’ve done in IIS 6.0 (slide 5), first of all we really simplified the UNC configuration story. Today, if you want to use pass-through authentication, and pass-through authentication means that you’re passing through the caller’s credentials rather than using that fixed set of credentials, it just works in IIS 6.0.
Second, we’re taking advantage of a new feature in Windows Server 2003 called constrained delegation, and we delegate for all incoming protocols. We also made numerous improvements in how we handle caching on IIS 6.0, and we did that both in our static file handler and in the ASP handler. And we improved the scalability of the change notification caching that ASP and the static file handler use.
Let’s take a look at each of these in a little bit more detail. First of all, let’s talk about the simplified configuration (slide 6). When you’re accessing a remote file server from a front-end Web server, the file server requires a security token. It needs some kind of credentials to identify who is accessing that. Those credentials are used to authorize access. And authorization is implemented in the form of access control lists (ACLs). You’re probably very familiar with that. So authorization controls who has access and what they can do. Can they read everything? Can they write and read? Can they execute content? And so on.
In IIS 6.0 you have two options. We give you the option to either use a fixed user name or UNCPassword, just as we did in IIS 5.0, but now we allow you to use pass-through authentication. In this case we’re allowing the file server to pass through that security token for the user onto the remote Web server. We allow you to configure this through a check box in the Web site or virtual directory wizard. The check box is Always use the user’s credentials when validating access to the network resource.
When you select that check box it will pass through that user’s authentication. We’ve made that a lot simpler. In the past you had to configure a number of metabase properties. There was a UNC pass-through authentication property you had to set. In the past you could only use pass-through authentication when you were using Basic Authorization. Today we can do that pass-through no matter what incoming authentication method you’re using.
Dave, I think you had a comment?
David Cox: Just that we pass through the request user’s identity from the Web server to the file server, in case that wasn’t clear.
Richard: Let’s talk a little bit more about authorization and constrained delegation features in Windows Server 2003 (slide 7). This is an integrated security feature in Windows 2003, and it’s used so that we can delegate the authenticated user's token to the remote file server for authorization (slide 8). If you recall, before I said you had to pass on a token for that user. If you’re using authentication methods such as NTLM or Kerberos, or if you’re using certificates, you don’t really have a password and an account that can be passed on.
What happens is after that user is authenticated by the security service, a token is given, and that token can be used in authorization. However, we can’t pass that token from the Web server to a remote file server. You couldn’t do that in the past. We’ll show how we do this in Windows Server 2003, using Kerberos tickets.
Now if you’re using Basic Authentication, that does pass what I believe is an encrypted form of the user’s credentials on to the remote server. That has always worked, as long as you set the UNC authentication pass-through property. That will still work in IIS 6.0. But you don’t need to set that property anymore. You just need to leave UNCPassword and UNCUserName empty.
The nice thing about this feature in constrained delegation is that no matter what incoming authentication method you’re using, because we’re getting a Kerberos ticket that we can pass on, we don’t care what kind of authentication protocol you’re using, coming in. We can also constrain the access. We can constrain that for specific services, so it’s not just by server. It can also be done at a more granular level to constrain based on, say, the file service. We can constrain it based on the SQL Server service, and so on.
When you configure this, you do need to specify both the host service principle name (SPN) and the CIFS SPN. These are two service names that you want to allow delegation for. And this locks down that delegation to just those services, so that somebody can’t hack into the system and try to access another service using those credentials. So that’s a security feature. If you’re using SQL Server, there is an additional service principle name that you need to use, and that’s the MSSQLSvc SPN.
This diagram (slide 9) gives an example of how this delegation works. What I’m showing here is a typical ISP configuration. We have multiple front-end Web servers, and I've balanced some sites on one server and other sites on another server. They’re both sharing the same NAS server for the back end. You also see a domain controller in here. If you haven’t picked that up by now, it is required if you are going to use constrained delegation. You do need Active Directory® in Windows Server 2003. We make extensive use of the Kerberos domain controller.
The way this works is an incoming request from a Web client goes to the front-end Web server, and the client authenticates their session. The Web server actually requests a Kerberos ticket from the domain controller. That Kerberos ticket gets returned back to the Web server, and it can now use that when it’s accessing content on the back end. The next thing that happens is that we try to connect and establish an SMB session to the back-end server. That Kerberos ticket is passed along in the first part of that session, and it is used by the file server to check against the ACLs and authorize access or deny access to that content.
Next I want to go over how we handle authentication (slide 10) in the various cases where we either supply credentials or we don’t supply credentials in UNCUserName and UNCPassword. The first thing we look for is: Is this an authenticated request? We’re going to read from left to right. It can either be an authenticated request, and authenticate using NTLM, or I may come in and just do an anonymous request. In the case of an anonymous request, I’m just using the machine’s IUSR account.
If it’s an authenticated request, we check if the credentials are good. Is the password good for that user name? If that’s not true, we return a 401 error. If it is good, then the next thing we have is that the Request User, the user identity that we use for that request, is whoever that authenticated user’s identity is. The identity will either be IUSR if it’s anonymous or the Request User if it’s authenticated.
The next thing we check is whether UNCUserName is empty or not. If UNCUserName is empty, then we’ll just use that Request User as the user. That’s the user that we’re impersonating. If UNCUserName has something in there, anything in there, then we’ll use that rather than the Request User's identity. The next check we’d have to make (if we had UNCUserName specified) is: Is the password valid? If the password is not valid, then we return a 500 error.
If the password is valid, then the UNCUserName is the user that is used for authorization. That’s true even if we’ve authenticated, if you had UNCUserName specified. Now these errors that we return are as generic as possible, so that we don’t tip off a hacker, for example, that they’re succeeding or not succeeding in what they’re doing. We just make it as obscure as possible for them.
David: For that 500 error, there’s a new custom error in that scenario, so you’ll get a 500.13. You can check the sub-status code in your log to verify that you actually have a problem with your UNC configuration.
Richard: Let’s spend a few minutes talking about scale and the changes that we made in IIS 6.0 (slide 11). We’ve made a number of changes. Let’s start with a refresher on SMB and how it works. We’re, again, showing our scenario where we have a Web server and a request coming in from a Web client (slide 12). On the far right we have a file server, and on there we have storage, several partitions for example, with shares mapped on each of those partitions. We probably have multiple shares on those partitions. What you might typically do as an ISP, for example, is put some of your users on one share and balance some other users on another share, and so on. Each of those shares might have several Web sites rooted under that share.
The first thing that is established between that Web server and the file server is an SMB connection. And we’re just using a connection-oriented protocol so that we can pass all of our requests over that. For each authenticated user connection between the Web server and the file server, we'd have a separate SMB connection. For example, in IIS 6.0 I have the ability to configure individual identities, configurable identities, for each application pool. And each application pool can be used to isolate one site from another.
So if I set my server up that way, so that each site has its own application pool and each application pool has its own configurable identity, then for each of those application pools coming in to the file server I’ll establish a separate SMB connection between the Web server and the file server. Let’s talk a little bit about the scale considerations here. We said each Web server uses one connection.
These are also closed down after they’re idle for 15 minutes. But as long as files are open between the Web server and the file server, that will persist. We can have a maximum of 16,000 connections per file server. Depending on the number of application pools you have in your server farm, it’ll cross all of the front-end servers that are accessing that file server, you can have a maximum of, say, 16,000 application pools, 16,000 sites accessing the back-end server. In practice, it will probably be less than that, for most users.
The next thing that happens is after that connection is established, we have an incoming request, and that’s going to want to do some I/O to access content in the site directory on one of those shares (slide 13). We have a specific user that that site, that application pool for that site, is running under, and it accesses the content and sends it back to the client. That establishes an SMB session, and we can have multiple sessions per connection. Across SMB sessions I may pass one or more, typically several, commands. Each command might be for an I/O.
We also use commands in SMB for file change notifications. File change notifications are used to detect when content has changed on a share, on any level of the tree in that share.
I can have a maximum of 16,000 sessions per connection, and so the maximum number of sites I could have, hypothetically, would be 16,000 divided by the number of user accounts per site. If I’m an ISP, for example, what I might want to do is have several user accounts per site.
I can use one user account for anonymous access, I can use another account for an administrator for that site, and perhaps yet another account for an author on that site, where I might have a couple of different administrators who are authorized to use that site. Again, the sessions are transient, and I believe — Dave, is it 5 minutes or is it 20? We’re not quite sure on that, but after a period of time, those sessions will shut down, after being idle.
David: The change notifications are the only things that last for the lifetime of the connection. So as long as you’re caching files on that file server, we’ll try to keep that SMB connection open.
Richard: That’s a very good point. You would have that session open if you’re monitoring change notifications. We will show, in a moment, that you have a choice of whether you’re using file change notifications or not.
The next issue, regarding scale and SMB, has to do with the commands that are passed between the client in this case, which is the Web server, and the server, which is the file server (slide 14). Remember that the Web server is a client to an SMB service.
We have one parameter, MaxWorkItems, that can be the maximum of the computer memory. Any time SMB does any kind of an I/O, it’s using a work item. Any time we’re doing anything like a change notification, that’s using a work item. The limit there is the maximum computer memory. We also have two parameters that are kind of related to each other: one that works on the client and the other that works on the file server. MaxCmds is a Web server or a client-based registry setting, and that just says this is the maximum number of SMB commands that I want to have outstanding, at any given time, with this file server. MaxMpxCt is related to that.
That’s set on the file server, and it says, "This is the maximum number of incoming commands I want per client." So those should be the same. Typically we’ll see maybe 75 percent of commands used for change notification and 25 percent for file input and output, in the case where you have an ISP with a lot of sites and you have very broadly distributed content. In that case, if I had file change notifications enabled on those sites, most of what I’m doing is using those commands for the outstanding change notifications, waiting for the content to change.
To maximize the scale in this scenario, with SMB and IIS 6.0, we did a number of investigations, we did a lot of testing in our labs setting this up, and there were several recommendations we made to the developers (slide 15). One was to improve how we handled caching and to use what you could argue is a less aggressive caching. Rather than looking for change notifications all the time, we also wanted to provide some other options to do that. So the improvements we made here, first of all, were that in some cases NAS and SAN devices don’t reliably support change notification.
What they’ll do is, for example, they’ll just report change notifications at the root of the share, but not in any directories under that share. So we provided a workaround. In the past the workaround was to simply disable caching. Now you’re going to lose the benefits that you gained from caching. Caching provides a performance benefit, because if we can retrieve content from the cache, we may be able to avoid having to go out over the wire to that remote file server to retrieve the file. Caching is our friend.
The new caching algorithm allows us to, instead of using change notifications, use the file’s last modified property. We’re essentially polling the file, but in this case we’re not actually retrieving the full content of the file. We’re just checking its attributes, and we’re only doing this every five seconds. So if you have a lot of accesses to this file, those accesses are still going to go fairly quickly. We don’t have to keep polling for each I/O to that file. We only do that every five seconds.
This turns out to be more reliable than change notifications, in the case where we are using IIS with NAS solutions, especially if you’re using other vendors’ solutions. We've made this algorithm standard for both the ASP handler and the static file handler. The benefits of this are, if you recall the discussion about the work items and the maximums we had there, you can see where you had very a demanding configuration, where you had a lot of sites and a lot of users accessing the same file server, where you might have run out of those SMB work items.
Well, when we’re using the new caching algorithms, we don’t do that. So this increases our scale, in terms of the number of virtual directories or sites that we can support against those file servers on the same server.
Incidentally, we made a number of changes in IIS 6.0 to really improve the time it takes to start up the Web server, and we had even up to 20,000 sites configured. David was telling me on the way over to the studio today that we’ve actually reduced that time to something like 20 seconds of server startup time. That was a limiting factor in the past, where if you had more than something like 5,000 sites on an IIS server, the startup times for the server became prohibitive. That really limited the number of sites we could configure per server. That’s really expanded in IIS 6.0.
David: Actually the 20 seconds will probably be in IIS 6.0 Service Pack 1. That’s only going to get faster and faster as we make some performance improvements.
Richard: Thanks for the clarification. We also provide some options so that we can still use the change notification-based caching or the file attributes-based caching (slide 16). Here’s an overview of why you would use either one. The change notification caching really provides, still, the best performance overall. If you have a really high-volume site, if you have maybe one site or a few sites on the server and just a high amount of traffic coming into there, if the files are really hot or if the files are really large, then change notification caching is going to work best for you.
It also allows us to take the best advantage of the architecture of IIS 6.0, and we can store responses in the Http.sys response cache. The Http.sys response cache has been a big performance win for customers who have been testing IIS 6.0. They’ve seen performance sometimes double, because they have a lot of content that can reside in that cache, and they no longer have to make the transition from kernel mode to user mode to retrieve that content.
So in the case where we can take advantage of change, from change notification-based caching, that is going to work well for you. Again, if you’re a large ISP, if you have a lot of sites, a lot of content distributed across those sites, then you may not be able to take advantage of this.
The file attributes-based caching has slightly slower performance in cases where you don’t have really hot files. If you’re just doing normal accesses, we’ve found that the performance is almost the same as change notification-based caching. I said it checks every five seconds by default.
Below here you see two registry keys for configuring this:
HKLM\System\CurrentControlSet\Services\Inetinfo\Parameters
DoDirMonitoringForUNC (default of 0 is off, 1 is on)
HKLM\System\CurrentControlSet\Services\ASP\Parameters
EnableChangeNotificationForUNC (default of 0 is off, 1 is on)
One of the registry keys is for the static file handler, that's DoDirMonitoringForUNC, and the other is for ASP. Even though they have different names, they’re really turning on or off the same thing. It’s just for each service. We just didn’t name them the same.
We’ve gone through all the configuration and the scale considerations and the improvements we’ve made, and I want to now go into some real-world scenarios where we’ve deployed this, and now talk about, for each case, how we made those choices of settings (slide 17). To get you started, I want to show you, step by step, how you configure UNC shares and remote content with IIS (slide 18). How do we get that all started?
The first step is to enable constrained delegation on the domain controller, and that’s a domain-level setting. The next thing you’re going to do is configure the IIS server, and here you’re just setting up a Web site or a virtual directory and mapping that to a UNC share. Again, remember to leave UNCUserName and UNCPassword empty, and that way either the anonymous user or the authenticated user’s credentials are passed through. By default, LogonMethod = 0, but if you’re comfortable with editing things in the metabase, or if you make a lot of changes in the metabase, you might just want to make sure that’s set to 0. Then finally you want to set authorization on that content. You want to edit your ACLs.
If you’re using an older file server, if you’re using Windows 2000 or Windows NT 4.0 as your file server, there may be an Everyone access control entry in that ACL, at the share. You want to remove that. By default, in Windows Server 2003, that doesn’t exist. We want to lock down access to that share just for specific users who are authorized to access that.
After you’ve removed that, now you’re going to want to add access control entries, either using user names or using groups that specify which users have access to which shares, and to which locations underneath those shares.
David: A lot of times customers will use a group on the share level and then use specific users for particular folders that their sites are on. So if you have one share that happens to host all your sites, you might just want to have the share enabled for the group of users that will actually access any of those sites, and then lock down the ACLs on the file system itself, for the folders.
Richard: This dialog box (slide 19) just shows an example of how we enable delegation on a Windows 2000 domain. In this case the file server is running Windows 2000, or the domain is Windows 2000. We go into the users and computers tool in the administrator tools and select the computer. This is the Web server, and it is always going to be the Web server that you trust for delegation. We’re allowing that Web server to pass on delegation credentials to a remote server. You just have to select the check box Trust computer for delegation.
Now on Windows Server 2003 (slide 20) it’s a little bit different, because we’re taking advantage of constrained delegation. If you recall, in the case of Windows 2000 domains, we just had an on or off at the computer level to select whether we can delegate or not. In the case of Windows Server 2003, we can allow it to either delegate for a computer or not, but also delegate for a specific services.
The first thing you want to do in Windows Server 2003, if you already have a Windows 2000 domain, is raise the domain functional level to Windows Server 2003. After you’ve done that, the change is irreversible. So unless you’re a domain administrator for your organization, don’t do this. You may have to go and consult with your domain administrators as to whether that’s okay to do.
On the Delegation tab in Active Directory Users and Computers, just select Trust this computer for delegation and select To specified services only, and also you need to check Use any authentication protocol.
This dialog box (slide 21) shows the screen for raising the domain functional level. This next screen (slide 22) shows the configuration of constrained delegation. Here you see a couple more choices than in the Windows 2000 case. We’ll select Trust this computer for delegation to specified services and underneath of that we’ll select Use any authentication protocol, and then we add the cifs service and the HOST service. When you set this you’ll see a list of the services that are available here, and then you can click those. This is accessed when you click the Add button. You’ll see all the services listed, and you can select the services that you want to add into this list.
David: Just make sure that you select the SPNs for the file server. That’s a common mistake. People select the Web server SPNs rather than the file server SPNs. We want to be able to delegate to the file server.
Richard: The two cases studies (slide 23) that we’ll walk you through are, first of all, shared hosting, and second, a high-volume site. The high-volume site is Microsoft.com. The shared-hosting case study was a large ISP that has been working with us in the beta program. In shared hosting, you have a large number of sites. That’s really important, so that you get the kinds of economies of scale that you need to run a large ISP. Because you’re offering these sites cheaply, people are going to be likely to buy a site just to show pictures of their dog or their grandmother, or something like that.
In that case you can expect that the volume of accesses to that site, the requests to that site, are going to be very low, and we can take advantage of that. We cannot only take advantage of it in terms of scale, SMB, and remote configuration, but we can also take advantage of that in terms of application pools. We can essentially overbook application pools per server, expecting that not all of those sites are going to be active at the same time. That allows us to make better use of those resources.
In the case of this hoster, they chose to improve the reliability of the back end by using DFS. DFS is a way of just logically mapping multiple file servers and shares into a single logical namespace, and we can do a little bit of balancing to that as well. It is primarily used to maintain reliability. In the case of Microsoft.com, it’s one of the biggest sites on the Web, and it has some of the highest volumes — not just legitimate traffic, but also a lot of people trying to hack into that site unsuccessfully. That all adds up to a lot of volume. We have a lot of hot files on those sites.
If we have a new service advertised on the home page of Microsoft.com, you can expect that there are going to be a lot of requests coming into that service or to that piece of information. In this case Microsoft.com wants to use an NLB-clustered file server. I will add that this, today, has just been tested on their sites. This is a configuration that is probably going to go live in the next couple of months. But they have been live on IIS 6.0 since the summer, since we released RC1. And they’ve been very successful with IIS 6.0.
These are the considerations we made for shared hosting (slide 24). In the case of their architecture, they had two or more front-end Web servers, typically more front-end Web servers. And the idea here is that they want to be able to balance all these sites across several machines, but share the same back-end server. Again, set UNCUserName and UNCPassword to blank for the ISP. We want to pass through the caller’s credentials.
In terms of our caching choices, we disabled change notification-based caching. We allowed the file check threshold, which is five seconds by default, at the default setting. And that seems to be working quite well for them, and they’re serving something like 2,000 sites.
David: Another thing we might add here is that if you do happen to have a lot of sites, like this ISP does, and if they’re all using different user credentials to access the back-end file server, you’re going to need to, depending on how many sites you do, increase the TCP setting called MaxUserPort. Obviously if you have a lot of users accessing the back-end file server, you’re going to need to increase that number. You might look at MSDN® for details on MaxUserPort. And you’ll need to set that on the Web server. It’s hard to figure that out, because you’ll get these strange network errors, like it cannot connect to the file server. It’s not because you couldn’t connect. It’s because you’re out of TCP users that you’re allowed to create.
Richard: That's good to point out. In terms of Microsoft.com, they organized their servers in virtual IPs users or VIPs, which are essentially NLB clusters (slide 25). They have 10 front-end Web servers per cluster and they’ve configured two back-end file servers using NLB. They want to make sure that’s redundant on the back end. They’re using a tool called WCAT, the Microsoft Web Capacity Analysis Tool, to play actual samples of live traffic to those servers. Today that configuration is not live. It’s happening in the labs, but we’re able to basically send a realistic volume of traffic at those front-end servers and see the results, and do some pretty accurate testing.
In the case of Microsoft.com, we wanted to enable dir monitoring. I think we started out assuming that the new caching would benefit them as well. That turned out not to be true. And we really benefit the most in the case of Microsoft.com, where they have very hot files, when we’re caching. They’re able to take advantage of the Http.sys cache and take advantage of change notifications. And in their case, even though they have a lot of content, because there’s really only one authenticated user coming in — most accesses to Microsoft.com are anonymous — we don’t have the same kind of scale concerns that we had in the other scenario.
We also wanted to make sure that we could handle the volume of SMB commands that were coming in. We set MaxCmds in this case to {2048}. That’s the maximum that any of those 10 incoming file servers can fire at any time at the remote file server.
We also set registry key settings for SMB work items. Again, the MaxMpxCt matches, 2048, and we set the MaxWorkItems as high as we could, to 65535. Some of the considerations that you have, in the case of Microsoft.com, have to do with the time it takes to warm up that cache. While we’re waiting for that cache to be populated with content, we have to go out to the file server for each of the I/Os, and that can slow the machines down on startup.
Some things that you can do to avoid that are to perhaps have a warm-up time for your machines before you put them into rotation, and you might even consider running a script or something like that that just goes and accesses all the content, to bring the cache online. And then you can put that machine into rotation. The other thing, in this case, is if it’s a very high-volume site. And any time you’re using this, you should test your assumptions. Test your configuration before you actually go live. Again, this WCAT tool is a really valuable tool for measuring things. I think we did a Network Monitor capture and just fed that directly into WCAT.
David: Actually we were able to use Log Parser to generate the WCAT script itself. Log Parser is a publicly available tool that you can download, and it just allows you to generate a WCAT script with not too much extra work.
Richard: It’s just using Log Parser. That’s excellent.
David: To look at the day’s logs and then create a WCAT script to replay that log.
Richard: This is a good time to put in a recommendation for the Log Parser tool. We recently released that, at the beginning of this year, and it’s a really valuable tool for IIS. It allows you to do SQL-style queries against log files, and we can create filters with that. We can use it to look for hack attempts into our server and so on. I highly recommend it. Go check out the tool. Go to http://www.microsoft.com/downloads/ and you should be able to find it there.
Before I take your questions, I just want to point you to a couple of resources (slide 27) on this topic. We are also working on releasing a white paper at about the time that we’ll launch Windows Server 2003. And that will essentially go over the content that we’ve covered today in a bit more detail. You can take a look at KB article 207671, "HOW TO: Access Network Files from IIS Applications" on the support.microsoft.com site. And also on the same site we have an authentication and security white paper for Internet developers (174811). Consider checking out those resources.
Now I think it’s time to take some of your questions. Otto, do you want to get that started?
Otto: Thank you very much for the presentation. Before we jump into the Q&A, I’d like to share some program notes with our listeners. If any of the details on the PowerPoint® slides were difficult to view in your browser today or you’d like to simply have a copy of the slides, they should be available for download [click Past Support WebCasts] at support.microsoft.com/webcasts/, then go to today's WebCast.
Just a reminder, for those of you who have joined us in the past, and something new for those of you who are new to the program, the Q&A portion of the Support WebCast is intended to encourage further discussion of the topic that we discussed today. In addition, one-on-one product support issues are outside the scope of what we’re able to address. If you do find that you need some more complex technical assistance, feel free to contact Product Support Services directly and speak to a support professional or, alternatively, you can submit an online incident over the Web.
We have a few questions coming in here: Is there one SMB connection for multiple identities?
Richard: Each identity would use a separate SMB connection.
David: That’s when you need to change your MaxUserPort. So you’re going to have a different connection for each user. If you do run out of ports, then you need to increase that MaxUserPort setting.
Otto: You mentioned a maximum of 16,000 per connection from the Web server to the remote file server. What about file handles? Is this the same as SMB connections, or is there an internal limit here as well?
David: We mentioned those work items, and those work items basically consist of change notifications, which is a work item. Each change notification is a work item. A work item can also be a file I/O, which would be like a handle. If I had an open file, that’s one work item. Those all take up non-page pooled memory, so you need to be careful of exhausting your resources when you’re too busy. Hopefully it doesn’t take very long to open a file and then close it. Read the file and then be done.
Richard: Also something to note that’s not exactly related to the question, but it’s an interesting change that we made in the file services: we previously had a limit of, I think it was, 2048 sessions per connection that were allowed in SMB. And we did work with the file services team, and they were able to make some changes in how they handled that and can squeeze in, now, 16,000 sessions per connection. That is the limit of the protocol. But we expect that’s going to be no problem, with some of the changes that we made in our caching, to address scale.
Otto: Regarding the UNC scenario, the high-volume site, you talked about the two back-end file servers with a Network Load Balancing cluster situation. We’re wondering where we can find more information on NLB file servers and whether or not this is a function of Windows Server 2003.
Richard: I believe we released NLB around the time of Windows 2000. It was part of Windows 2000. This is just a clustering service that is part of the operating system that you can configure. There should be some information available at http://www.microsoft.com/technet/. If you do a search on "NLB", you should find the topic. I believe they also have documentation in the help files for Windows Server 2003. Or if you’re using Windows 2000, it’ll be in those help files.
Essentially what you’re doing with NLB is routing requests to each server based on their availability, and it uses a little bit of a load-balancing algorithm across those servers to make sure that you’re able to maximize the use of the resources in that server farm. It's a little bit different than MSCS clustering. MSCS clustering is more of a scenario where you’re virtualizing your services, and, in that case, it really looks a lot more like just a single machine, even though it’s implemented as multiple machines.
Otto: Regarding the use of IIS 6.0 in an ISP setting, is the pass-through authentication able to support multiple users and authentication tokens per site? For instance, site.example.com has an anonymous IUSR_Example1 and a FrontPage® administrative user for publishing content. It looks like it continues: The security features will allow for one user to read only on the back-end file and the other user to write content as necessary.
Richard: That’s exactly the kind of thing that we’re talking about supporting. This is why we’re saying, when we’re looking at some of these SMB parameters, in some cases you may have more than one user configured per site. That’s exactly the reason why, is that one user identity would be for anonymous access. Another user identity might be as an administrator. Another user identity might be used for FrontPage. You might have a couple for FrontPage. Again, each of those identities would be stored in Active Directory and individually checked for authorization against that content, or against the ACLs that have been established by the file system or by FrontPage.
David: There’s no real setting you need to make. We'll just pass through any user that exists that is successfully authenticated in Active Directory, or the anonymous user. There's nothing really to set in IIS. You just have to allow us to use a pass-through. And then, like Richard said, as many users as you have, we just let that happen.
Otto: This next question sounds like it’s related to Active Directory: Will IIS 6.0 support the use of Active Directory UPN for logons?
Richard: I had that question from another customer a few weeks ago. Yes. We do support the UPN format. The UPN format is different from the NTLM format. In NTLM format you have the domain/user. The UPN format is username@example.com. Either way, that can be used. In the case of IIS, we’re just passing that on through to the security service in Windows, and we don’t have to do any explicit parsing on that user name. We just pass that on to the security service, and that provider does the parsing.
Otto: Are you supporting third-party file servers, and if so, which ones?
Richard: That’s a good question. It’s a little difficult to address. What I will say is that we have done testing on third-party file servers, and with a couple of the larger names that you’re familiar with. We have decided, and for various reasons, that we can’t really endorse any vendors. So we’re not really saying whom we support, and it’s pretty much up to those vendors to provide that support. But we have done the requisite testing.
To the extent that a third-party file server emulates SMB, or CIFS, in the case of third parties (usually the third parties will implement the CIFS protocol, which is essentially the same as SMB), as long as it’s a faithful implementation, I think they should be fine.
One thing to point out about some of these implementations is that, again, as we mentioned, sometimes you will get cases where a vendor might not fully implement the file change notifications. It might only do it at the root. I think some of them have become aware of some of these problems and they have made fixes, so you’ll have to check with the vendor on what fixes they’ve implemented.
If they’re doing a UNIX implementation, what they may have to do is map ACLs. At the native level, they may not support ACLs. What they do is they have a mapping table of user credentials to UNIX file service authorization formats. Typically in UNIX you have more limited authorization. It’s just user and group, and read, write, and execute access. So it’s not as granular as what you can get with ACLs. When you map, you may lose something in that translation. I don’t want to scare people off of doing that, but you just want to test it and make sure it works for you.
David: That's why our default is the last modified caching algorithm. So we tend to be more reliable from the get-go.
Otto: Can you elaborate a little bit on the NLB-clustered file server setup?
Richard: That’s probably something that we can’t elaborate on here, in terms of details. I don’t have those details in front of me, and I did not actually configure that myself. Again, these were the Microsoft.com folks who have been doing that the most. What I’d recommend would be that you might want to see if there are any sessions on NLB, in terms of WebCasts, or maybe this is a good suggestion for a WebCast in the near future, on NLB. Again, there are numerous resources for this on the support.microsoft.com and TechNet sites, where you can find out about NLB.
Otto: Are there limitations when working with UNIX file servers that are using Samba? I’m not sure if that’s a third-party product.
Richard: Samba is an emulation of the CIFS file service. It’s basically the Linux world’s answer to Windows file server protocols. I’m not aware of any specific limitations of Samba in that scenario. I think that, again, falls under the same category as third-party NAS vendors. It really is going to depend on that implementation of Samba, in terms of whether and how well that works. The best answer would be to give it a try, if you’re interested in doing that.
We also have done some testing of IIS with the NFS client in Services for UNIX, accessing NFS volumes. We were aware of some issues there. In particular, we had an issue where we could not support case-sensitive file names — accesses between the front-end server and an NFS share. The problem here was that our cache algorithms were not case-sensitive. To optimize the performance of the cache, we treated all the file names as Windows file services does, as case-insensitive.
It was a good assumption to make if you’re just working with Windows systems and if you’re implementing a cache. But in the case of NFS volumes, that became a problem. Because if you had two files with the same names, except that the letters in those names were different — one might be lower-case letters and the other might be upper-case letters — we might have some confusion in the cache. And I believe what we did in that situation was we allowed IIS to work with NFS file servers, as long as you disabled that cache. There’s a registry setting that allows you to disable the cache in ASP, or the static file handler, if you’re working with NFS. It limits the performance, but it allows compatibility.
Otto: We are currently using Application Center 2000 for an IIS 5.0 cluster, and we’re wondering if we can join the existing cluster with IIS 6.0? Or do we have to create a whole new cluster?
Richard: I’m not 100 percent sure about the answer to that question. I think that’s more of an Application Center question. I’m not too familiar with whether they would need to create a new cluster or not. It’s a good question, though.
Otto: Do you know where users can get some more information?
Richard: Again, you could check the Application Server documentation. You might want to check on http://www.microsoft.com/technet/ or support.microsoft.com. I’d also encourage you to post queries in some of the public or beta newsgroups, if you have access to the beta newsgroups. The newsgroups are an excellent place to get information and answers to questions like this. Because not only do the support teams at Microsoft, the PSS personnel, do a great job of monitoring and responding to questions, but so do the product teams.
Often a program manager or a developer on the product team may know the answer to something before that’s disseminated out to the support channels, and we can just jump in with an answer. It’s a really excellent resource. I’d encourage you to get on the microsoft.public.inetserver.iis newsgroup and post your question there, or look for a newsgroup for Application Center. I have heard that Application Center will support IIS 6.0 at RTM.
Otto: Can you set up UNC-based sharing without the use of Active Directory?
David: You could, but you don’t have as much functionality. So you could set up two machines, like say they’re both in a workgroup or they share users and passwords that are identical, but that’s fairly difficult to manage. That seems like quite a nightmare, really. If you have more than a couple of users, it’s going to be fairly difficult to manage over a long period of time.
Richard: You certainly can’t take advantage of any of the delegation features, because there’s no way for us to get a Kerberos ticket, in that case, to pass on the back-end server.
David: You can only use Basic.
Richard: You can only use Basic Authentication and, as David said, you have to do password synchronization between the front end and the back end. I used it before, when I worked in consulting. For those of you who don’t know about that, if you have a user name mapped on two separate machines, if the passwords are identical on the front end and the back end, or on two servers, you can essentially access the content as one user on the front-end machine, and that can be passed through to the back end and honored as valid credentials.
David: I believe you can turn that off. That’s part of the security practice.
Richard: It probably is something you can turn off, too, and you may very well want to.
Otto: Where would you choose replication between duplicate data stores versus NAS? And do you have any information on what the performance impact is of NAS?
Richard: We spoke a little bit about the performance impact. It’s going to vary for each application. But in general what’s happening here is that when you’re using NAS, you’re going out over the wire, unless you can serve that content from the cache. In the case of replicated local storage, if you feel more comfortable with redundancy that’s afforded by having that — if you have maybe a few Web servers, for example — it might make the most sense to just keep the storage local and replicate content between those few Web servers.
As the number of servers grows, especially as you get past like five servers, now you’re losing some of the benefits of that replication, and there’s not that much more incremental advantage and reliability to adding the next copy of that storage. Now you’re starting to get weighed down by the difficulties and the overhead of having to replicate content out to multiple servers, so it becomes more of an administrative headache. At some point, around five or more, you’re going to prefer storing that content on a single server.
David: There’s certainly also something if you have a lot of content that changes often, there’s a lot of bandwidth being used on your back-end network to just synchronize and replicate all that content. And you might find that if you’re changing content really often, you might be better off using that bandwidth for doing change notification or last-modified work. Because that’s certainly a lot of bandwidth to push thousands and thousand of files to a bunch of servers. That does pick up a lot of bandwidth, so there are some performance issues there as well.
Otto: Are there any considerations or limitations to running .NET applications from a NAS?
David: That really depends. You’d want to, again, look at the ASP.NET documentation for code trust issues. You want to be careful about what kind of code you trust and where it’s coming from, and things like that. I don't have any details, but you’d want to check the documentation.
Otto: This one might be better handled through a support incident, but I’m going to ask it just in case you guys have seen this: We occasionally get RPC not available on heavy file server share use. Is that a concern, or has that been addressed in Windows Server 2003?
Richard: That sounds like you’re having problems establishing a connection at that point. That might be related to some of those scale issues.
David: It’s hard to say, without more information about the configuration and the servers they’re using. There are certainly a lot of details that are important to look at when you’re configuring a of UNC. There are a lot of different variables.
Otto: A follow-up on the ISP example: What benefits do you see in running a Windows-powered NAS device, as it relates to using UNC shares with DFS?
Richard: There are a lot of similarities here, first of all. A Windows-powered NAS device doesn’t exclude you from using DFS volumes with your shares. I think the alternative here is are you using an appliance, or are you going to just configure this yourself? And in the case of a Windows-powered device, it should be pretty much the same result as doing it yourself, as long as you get the configuration right.
One of the advantages to doing it with an appliance would be that the vendor has done testing on that architecture, on that platform. They’ve done some configuration for you. They’ve provided some tools to administer that and configure for scale and performance. And they provide tools to just make it a more complete solution. So especially if you’re not an experienced administrator, that might be a preferable alternative to just configuring it with a Windows-powered NAS device. A lot of those vendors also provide backup tools. They may have done hardware optimizations for file services. So to the extent that they’ve done those sorts of things for you, you may see some improvements.
Otto: It appears that we’ve answered all the questions that have been submitted to the queue today. I’m going to wrap up the session here. I wanted to thank you all for joining us, and hope that the information was useful to you. I wanted to definitely thank our presenters for coming out and giving us a great presentation.
As always, you can e-mail us at supweb@microsoft.com. If you have any suggestions for future topics, general comments about the sessions, or even the WebCast program as a whole, we’d love to hear from you. Your feedback is absolutely valid and important to us. We certainly appreciate it.
We hope that you all have the opportunity to tune in again soon. Thanks, and have a great day.
|