The best way is to remove it from your directory from the google webmaster tools. Also some bots don't listen so additionally to robots.txt use the webmaster central. James On Sun, Jan 17, 2010 at 9:31 AM, Kai Schaetzl <maillists at conactive.com>wrote: > Add > User-agent: Slurp > > Crawl-delay: 86400 > to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at > least follows these rules. Something you can't say of Googlebot, for > instance. > > Kai > > -- > Get your web at Conactive Internet Services: http://www.conactive.com > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos > -- http://www.astorandblack.com -- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20100120/13778ed7/attachment-0004.html>