The best way is to remove it from your directory from the google webmaster tools. Also some bots don&#39;t listen so additionally to robots.txt use the webmaster central.<br><br>James<br><br><div class="gmail_quote">On Sun, Jan 17, 2010 at 9:31 AM, Kai Schaetzl <span dir="ltr">&lt;<a href="mailto:maillists@conactive.com">maillists@conactive.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Add<br>
User-agent: Slurp<br>
<br>
Crawl-delay: 86400<br>
to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at<br>
least follows these rules. Something you can&#39;t say of Googlebot, for<br>
instance.<br>
<br>
Kai<br>
<font color="#888888"><br>
--<br>
Get your web at Conactive Internet Services: <a href="http://www.conactive.com" target="_blank">http://www.conactive.com</a><br>
</font><div><div></div><div class="h5"><br>
<br>
_______________________________________________<br>
CentOS mailing list<br>
<a href="mailto:CentOS@centos.org">CentOS@centos.org</a><br>
<a href="http://lists.centos.org/mailman/listinfo/centos" target="_blank">http://lists.centos.org/mailman/listinfo/centos</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><a href="http://www.astorandblack.com">http://www.astorandblack.com</a><br><br>--<br><br><br><br><br><br>