[CentOS] httpd and robots.txt

Frank Cox theatre at sasktel.net
Sat Jan 16 22:33:08 UTC 2010


On Sat, 2010-01-16 at 14:18 -0800, R-Elists wrote:
> quite frankly, there are many orgs out there that dont follow this
> anyways,right?

Since robots.txt is a "suggestion" and .htaccess is actually enforced, I
use a simple robots.txt like this:

User-agent: *
Disallow:

and put the bad guys into .htaccess.

-- 
MELVILLE THEATRE ~ Melville Sask ~ http://www.melvilletheatre.com



More information about the CentOS mailing list