[CentOS] favicon.ico and robots.txt

[Lanny Marcus]

On Fri, Aug 28, 2009 at 10:46 AM, Dave<dave.mehler at gmail.com> wrote:
> Hello,
>        I'm running an apache 2.2 webserver on centos 5.3. I'm seeing
> frequent requests for robots.txt and favicon.ico from the logs those files
> should be in the document root area. What are these files, is this something
> the rpm installs, or do i have to retrieve or generate them?

My experience is that if you do not have these files, they will
generate errors and possibly use more bandwidth than if you had the
files. I made one favicon.ico file with a tool on the web, from a
photo (sorry, I don't remember the URL) and I got the other
favicon.ico file somewhere on the web where they can be legally
downloaded (also do not remember that URL). As previous posters
pointed out, the robots.txt file will be obeyed by crawlers that
follow the rules, others won't pay any attention to it, but if you
don't have a robots.txt file I think you will see errors in your logs.