17 Jan
2010
17 Jan
'10
3:31 p.m.
Add User-agent: Slurp
Crawl-delay: 86400 to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at least follows these rules. Something you can't say of Googlebot, for instance.
Kai
--
Get your web at Conactive Internet Services: http://www.conactive.com