On Fri, 2010-04-16 at 17:13 +0100, Karanbir Singh wrote: > On 16/04/10 17:06, JohnS wrote: > > I do not think that's the whole problem: Take into consideration this; > > site:wiki.centos.org kernelbuild > > Use that at both bing.com and google.com. Bing wins hands down. > > > > BTW when was the last bing crawl date? Today? > > well, googlebot is always on centos.org, its not uncommon for search > results from the forums to show content created within the last hour. --- Yes that is true but I can't wrap my head around how dns is going to prevent or cause problems with crawling a site. My understanding has been if you submit wiki.centos.org then it is only going to crawl it via /robots.txt. However it finding duplicate files so it says is a problem in it self. Using the moin site map should just prevent that at least in theory if it get regenerated. Of which will happen if you submit it through bing. Bing can auto pull the site map and every time that happens it will get regened. I'm guessing google can still do this also. Maybe fill us in on the dns issue effects. John