[CentOS] httpd and robots.txt
Frank Cox
theatre at sasktel.netSat Jan 16 22:33:08 UTC 2010
- Previous message: [CentOS] httpd and robots.txt
- Next message: [CentOS] httpd and robots.txt
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
On Sat, 2010-01-16 at 14:18 -0800, R-Elists wrote: > quite frankly, there are many orgs out there that dont follow this > anyways,right? Since robots.txt is a "suggestion" and .htaccess is actually enforced, I use a simple robots.txt like this: User-agent: * Disallow: and put the bad guys into .htaccess. -- MELVILLE THEATRE ~ Melville Sask ~ http://www.melvilletheatre.com
- Previous message: [CentOS] httpd and robots.txt
- Next message: [CentOS] httpd and robots.txt
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the CentOS mailing list