On Sat, Jan 16, 2010 at 10:18 PM, R-Elists lists07@abbacomm.net wrote:

would anyone out there care to share their robots.txt experience using centos as a webserver and their robots.txt files?

i realize this is a somewhat simple exercise, yet i am sure there are both large and small hosters out there and possibly those that have high traffic modify their robots.txt files differently that others ???

please share if you can or care to please?

for years we have just did a * (allow all) and disallow on things like /cgi-bin

as examples of places to visit for those out or in the know...

http://www.robotstxt.org/

http://en.wikipedia.org/wiki/Robots_exclusion_standard

http://www.google.com/robots.txt

and others...

quite frankly, there are many orgs out there that dont follow this anyways, right?

Right http://blogs.perl.org/users/cpan_testers/2010/01/msnbot-must-die.html

anyone?

tia

- rh

---

CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

Add User-agent: Slurp

Crawl-delay: 86400 to stop misbehaving Yahoo bots. Slurp is often misbehaving, but it at least follows these rules. Something you can't say of Googlebot, for instance.

Kai