[CentOS] httpd and robots.txt

[James Matthews]

The best way is to remove it from your directory from the google webmaster
tools. Also some bots don't listen so additionally to robots.txt use the
webmaster central.

James

On Sun, Jan 17, 2010 at 9:31 AM, Kai Schaetzl <maillists at conactive.com>wrote:

> Add
> User-agent: Slurp
>
> Crawl-delay: 86400
> to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at
> least follows these rules. Something you can't say of Googlebot, for
> instance.
>
> Kai
>
> --
> Get your web at Conactive Internet Services: http://www.conactive.com
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



-- 
http://www.astorandblack.com

--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos/attachments/20100120/13778ed7/attachment.html>