Select the product you need help with
- Internet Explorer
- Windows Phone
- More products
How to Write a Robots.txt File
Article ID: 217103 - View products that this article applies to.
This article was previously published under Q217103
We strongly recommend that all users upgrade to Microsoft Internet Information Services (IIS) version 7.0 running on Microsoft Windows Server 2008. IIS 7.0 significantly increases Web infrastructure security. For more information about IIS security-related topics, visit the following Microsoft Web site:
http://www.microsoft.com/technet/security/prodtech/IIS.mspxFor more information about IIS 7.0, visit the following Microsoft Web site:
Web Spiders, often called Robots, are WWW search engines that "crawl" across the Internet and index pages on Web servers. A Web Spider will then catalog that information and make it available to the Internet for searching. This makes it easier for users to find specific information on the Internet by allowing "one-stop shopping" through the Spider's WWW site. Most Robots also prioritize documents that are on the Internet, allowing search results to be "scored" or arranged in order of most likely matches on a search.
A Robots.txt file is a special text file that is always located in your Web server's root directory. This file contains restrictions for Web Spiders, telling them where they have permission to search. It should be noted that Web Robots are not required to respect Robots.txt files, but most well-written Web Spiders follow the rules you define.
A Robot identifies itself when it browses your site, which is known as the "User-agent" and appears in the logs for IIS. Generally, the flow of events when a Web Spider crawls your site is similar to the following:
Article ID: 217103 - Last Review: July 7, 2008 - Revision: 4.2