20 Jan
2010
20 Jan
'10
3:45 p.m.
The best way is to remove it from your directory from the google webmaster tools. Also some bots don't listen so additionally to robots.txt use the webmaster central.
James
On Sun, Jan 17, 2010 at 9:31 AM, Kai Schaetzl maillists@conactive.comwrote:
Add User-agent: Slurp
Crawl-delay: 86400 to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at least follows these rules. Something you can't say of Googlebot, for instance.
Kai
-- Get your web at Conactive Internet Services: http://www.conactive.com
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos