[CentOS] httpd and robots.txt

Sun Jan 17 14:31:17 UTC 2010
Kai Schaetzl <maillists at conactive.com>

Add
User-agent: Slurp

Crawl-delay: 86400
to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at 
least follows these rules. Something you can't say of Googlebot, for 
instance.

Kai

-- 
Get your web at Conactive Internet Services: http://www.conactive.com