Dec 03
Sorry, almost forgot the previous night when I was posting about slowing down Yahoo Slurp, that the same technique applies to MSNBot as well. The only difference is a different name (bot name) User-agent definition.
User-Agent: msnbot
Crawl-Delay: 10
The above 2 lines in robots.txt file will instruct MSN Bot to delay new request by 10 seconds before it requests new file from your domain web site. Now, back to work ! 
Recent Comments