28 Aug
2009
28 Aug
'09
7:53 p.m.
On Fri, 28 Aug 2009 08:54:02 -0700 Taproot webmaster@taproothosting.com wrote:
Robots.txt is a file that allows or denies robots from indexing or crawling the site if they behave as they should.
It's a common misconception. Robots.txt does NOT allow or deny... Robots.txt only SUGGESTs what they should crawl or not. It's up to the crawler to respect the robots.txt file.
The big ones like Google, Yahoo, Microsoft do follow the instruction of the robots.txt file, but many, especially the one harvesting emails, photos..., do not follow the instructions of the robots.txt.
--
Thanks
http://www.911networks.com
When the network has to work