# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # see also http://www.addedbytes.com/online-marketing/robots-txt-file/ User-agent: googlebot Disallow: User-Agent: msnbot Crawl-Delay: 10 User-Agent: Slurp Crawl-Delay: 10 User-agent: * Disallow: