[CentOS] httpd and robots.txt

James Matthews nytrokiss at gmail.com
Wed Jan 20 15:45:32 UTC 2010


The best way is to remove it from your directory from the google webmaster
tools. Also some bots don't listen so additionally to robots.txt use the
webmaster central.

James

On Sun, Jan 17, 2010 at 9:31 AM, Kai Schaetzl <maillists at conactive.com>wrote:

> Add
> User-agent: Slurp
>
> Crawl-delay: 86400
> to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at
> least follows these rules. Something you can't say of Googlebot, for
> instance.
>
> Kai
>
> --
> Get your web at Conactive Internet Services: http://www.conactive.com
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



-- 
http://www.astorandblack.com

--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.centos.org/pipermail/centos/attachments/20100120/13778ed7/attachment.html 


More information about the CentOS mailing list