[CentOS] httpd and robots.txt
Kai Schaetzl
maillists at conactive.comSun Jan 17 14:31:17 UTC 2010
- Previous message: [CentOS] httpd and robots.txt
- Next message: [CentOS] httpd and robots.txt
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Add User-agent: Slurp Crawl-delay: 86400 to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at least follows these rules. Something you can't say of Googlebot, for instance. Kai -- Get your web at Conactive Internet Services: http://www.conactive.com
- Previous message: [CentOS] httpd and robots.txt
- Next message: [CentOS] httpd and robots.txt
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the CentOS mailing list