The SharePoint Server crawler ignores directives in Robots.txt
Original KB number: 3019711
Symptoms
Consider the following scenario:
You use the Microsoft SharePoint Server 2013 or SharePoint Server 2010 search engine to crawl various sites.
For those sites, you want to use directives in the Robots.txt file to define the paths that the search engine can crawl.
You set the following directive for the default user-agent of the crawler:
User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot)
In this scenario, the SharePoint Server crawler doesn't apply the directive.
Cause
This issue occurs because the SharePoint Server crawl engine doesn't recognize its default user-agent in the directive.
Resolution
To resolve this issue, use the following directive in the Robots.txt file:
User-Agent: MS Search 6.0 Robot
Feedback
https://aka.ms/ContentUserFeedback.
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see:Submit and view feedback for