[CentOS-docs] www.centos.org and the wildcard dns

[JohnS]

On Fri, 2010-04-16 at 17:13 +0100, Karanbir Singh wrote:
> On 16/04/10 17:06, JohnS wrote:
> > I do not think that's the whole problem:  Take into consideration this;
> > site:wiki.centos.org kernelbuild
> > Use that at both bing.com and google.com.  Bing wins hands down.
> >
> > BTW when was the last bing crawl date? Today?
> 
> well, googlebot is always on centos.org, its not uncommon for search 
> results from the forums to show content created within the last hour.
---
Yes that is true but I can't wrap my head around how dns is going to
prevent or cause problems with crawling a site.  My understanding has
been if you submit wiki.centos.org then it is only going to crawl it
via /robots.txt.  However it finding duplicate files so it says is a
problem in it self.  Using the moin site map should just prevent that at
least in theory if it get regenerated.  Of which will happen if you
submit it through bing. Bing can auto pull the site map and every time
that happens it will get regened.  I'm guessing google can still do this
also.

Maybe fill us in on the dns issue effects.  

John