On Fri, Aug 28, 2009 at 10:46 AM, Davedave.mehler@gmail.com wrote:
Hello, I'm running an apache 2.2 webserver on centos 5.3. I'm seeing frequent requests for robots.txt and favicon.ico from the logs those files should be in the document root area. What are these files, is this something the rpm installs, or do i have to retrieve or generate them?
My experience is that if you do not have these files, they will generate errors and possibly use more bandwidth than if you had the files. I made one favicon.ico file with a tool on the web, from a photo (sorry, I don't remember the URL) and I got the other favicon.ico file somewhere on the web where they can be legally downloaded (also do not remember that URL). As previous posters pointed out, the robots.txt file will be obeyed by crawlers that follow the rules, others won't pay any attention to it, but if you don't have a robots.txt file I think you will see errors in your logs.