[CentOS] httpd and robots.txt

Kai Schaetzl maillists at conactive.com
Sun Jan 17 14:31:17 UTC 2010


Add
User-agent: Slurp

Crawl-delay: 86400
to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at 
least follows these rules. Something you can't say of Googlebot, for 
instance.

Kai

-- 
Get your web at Conactive Internet Services: http://www.conactive.com




More information about the CentOS mailing list