When you search for a URL in Microsoft Office SharePoint Server 2007, the following event may be logged in the Application log:
Event Type: Warning Event Source: Office Server Search Event Category: Gatherer Event ID: 2436 Date: Date Time: Time Description:The start addressURL cannot be crawled. Context: Application 'SharePointServerPortalname', Catalog 'Portal_Content' Details:The specified address was excluded from the index. The crawl rules may have to be modified to include this address. (0x80040d07) For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Additionally, the following message is logged in the crawl log:
URL The specified address was excluded from the index. The crawl rules may have to be modified to include this address. (The item was deleted because it was either not found or the crawler was denied access to it.)
This issue occurs if text (.txt) files are blocked on the Blocked File Types page in SharePoint Server 2007. The gatherer is unable to access the Robots.txt file when it starts to crawl the specified URL.
To resolve this issue, remove the file type entry from the Blocked File Types page. To enable a file type that is currently blocked, remove it from the Web application list. To do this, follow these steps:
Click Start, point to All Programs, point to Administrative Tools, and then click SharePoint 3.0 Central Administration.
In the left navigation pane, click Operations, and then click Blocked file types under Security Configuration.
On the Blocked File Types page, in the Web Application box, click the Web application that you want to configure.
On the Blocked File Types page, in the Web Application box, click Change Web Application.
Under Name, click the entry for the SharePoint site.
Click to select the .txt file name extension, press DELETE, and then click OK.
Repeat steps 4 through 6 for the affected Web application.