Many webmasters have complained that
Microsoft's Msnbot crawler for its as yet vaporware search product have been chewing through way too much bandwidth, and way too hard. In fact many large site, including WebmasterWorld.com have taken to blocking the bot altogether.
A MSN search tech has now posted a solution - the
MSNbot will now respect the following robots.txt command:
User-Agent: msnbot
Crawl-Delay: 20
This will cause the bot to wait 20 seconds between each page request, which will certainly alleviate the server load for larger sites - assuming they want to allow a bot without a search engine to chew up their site.
MSN promises that their search will be ready "real soon now"...