4577458 [rkeene@sledge /home/rkeene/devel/htp/mirror]$ cat robots.txt
User-agent: msnbot
Disallow: /

User-agent: BecomeBot
Disallow: /

User-agent: psbot
Disallow: /

User-agent: Slurp
Crawl-delay: 86400

User-agent: *
Disallow: /stats/
Disallow: /img/
Disallow: /css/
4577459 [rkeene@sledge /home/rkeene/devel/htp/mirror]$
Click here to go back to the directory listing.
Click here to download this file.
last modified: 2005-10-22 14:32:48