Jeff Johnson wrote: > You are correct that the scaling depends on the number of packages > not the number of repositories. > > However the solution to a distributed lookup scaling problem *does* depend > on the number of places that have to be searched as well as the cost of a > failed lookup. If you have to look in a large number of repositories to ensure > that some packages does NOT exist anywhere, well there are ways to do that efficiently. > > And none of the right solutions to the increasing cost of a failed lookup > are implemented in yum afiak. > > I was hoping to get an estimate of how bad the scaling problem actually > is from an objective wall clock time seat-of-the-pants measurement. > > Meanwhile I'm happy that you've found a workable solution for > your purposes. I'm rather more interested in what happens when > there hundreds of repositories and 10's of thousands of packages > that MUST be searched. > > I suspect that yum will melt into a puddle if/when faced with depsolving > on that scale. Not that anyone needs depsolving on the scale of > hundreds of repos and 10's of thousands of packages in the "real world", > but that isn't a proper justification for not considering the cost > of a failed lookup carefully which (from what you are telling me) > you are already seeing, and dealing with by enabling/disabling repositories > and inserting a high priority repository that is also acting as a de facto > cache and "working set" for the most useful packages. > Here are 2 speed tests. Repos are on 100Mbps LAN. First is with all repositories enabled and (left from CentOS 5 and old style RPMForge) aTrpms has higher priority over RepoForge. Second is with only safe RepoForge and other default (in my sistem) repositories enabled: [root at mama ~]# yum clean all Loaded plugins: fastestmirror, priorities, refresh-packagekit Cleaning up Everything Cleaning up list of fastest mirrors [root at mama ~]# time yum install vlc --enablerepo=* --showduplicates Loaded plugins: fastestmirror, priorities, refresh-packagekit Loading mirror speeds from cached hostfile 2233 packages excluded due to repository priority protections Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package vlc.x86_64 0:1.1.10-71.el6 set to be updated --> Processing Dependency: bitstream-vera-serif-fonts for package: vlc-1.1.10-71.el6.x86_64 <snip> --> Finished Dependency Resolution Error: Package: vlc-1.1.10-71.el6.x86_64 (plc-atrpms-stable) Requires: libmodplug.so.0()(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest real 1m9.365s user 0m36.585s sys 0m2.830s [root at mama ~]# time yum install vlc --showduplicates Loaded plugins: fastestmirror, priorities, refresh-packagekit Loading mirror speeds from cached hostfile 952 packages excluded due to repository priority protections Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package vlc.x86_64 0:1.1.11-1.el6.rf set to be updated --> Processing Dependency: libavformat.so.52(LIBAVFORMAT_52)(64bit) for package: vlc-1.1.11-1.el6.rf.x86_64 <snip> --> Finished Dependency Resolution Error: Package: vlc-1.1.11-1.el6.rf.x86_64 (plc-repoforge) Requires: libmodplug.so.0()(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest real 0m30.869s user 0m25.619s sys 0m1.261s Difference is double, but time difference is only 30sec on low budget PC (1GB RAM, AMD Sempron 64-bit, 40GB IDE HDD). And first run had to download yum repo files, second was from cache. If you have some specific stres test I would be happy to run it. Oh, yeah, yum reads and process xml files, not actual files, so searches are fast because of it. Here are repolist data (1=all repos, 2=default, 3 = default with no rpmforge and epel): [root at mama ~]# yum clean all Loaded plugins: fastestmirror, priorities, refresh-packagekit Cleaning up Everything Cleaning up list of fastest mirrors [root at mama ~]# time yum repolist --enablerepo=* Loaded plugins: fastestmirror, priorities, refresh-packagekit Loading mirror speeds from cached hostfile 2233 packages excluded due to repository priority protections <snip> repo id repo name status playonlinux 2+6 plc-adobe-linux 0 plc-atrpms-stable 1,636+382 plc-atrpms-testing 43+67 plc-c6-testing 0 plc-centosplus 42 plc-elrepo 92 plc-elrepo-extras 0 plc-elrepo-fasttrack 0 plc-elrepo-kernel 0 plc-elrepo-testing 31 plc-epel 6,080 plc-extras 0 plc-fasttrack 0 plc-kb-el6-ext 0 plc-kb-el6-ext-test 0 plc-kb-el6-misc 0 plc-kb-el6-misc-test 0 plc-os 6,019 plc-releases 0 plc-remi 114+223 plc-remi-test 2+76 plc-repoforge 2,835+1,106 plc-repoforge-buildtools 0 plc-repoforge-dag 0 plc-repoforge-extras 25+357 plc-rpmfusion-free-updates 0 plc-rpmfusion-free-updates-testing 70 plc-rpmfusion-nonfree-updates 0 plc-rpmfusion-nonfree-updates-testing 70 plc-sernet-samba 0 plc-updates 1,042 plc-virtualbox 10 plc-virtualmin-universal 134 plnet-archive 0 plnet-compiled 28 plnet-downloaded 89+16 plnet-releases 0 plnet-replace 0 plnet-test 0 repolist: 18,364 real 0m29.675s user 0m15.167s sys 0m1.633s [root at mama ~]# yum clean all Loaded plugins: fastestmirror, priorities, refresh-packagekit Cleaning up Everything Cleaning up list of fastest mirrors [root at mama ~]# time yum repolist Loaded plugins: fastestmirror, priorities, refresh-packagekit Determining fastest mirrors <snip> 952 packages excluded due to repository priority protections repo id status playonlinux 2+6 plc-adobe-linux 0 plc-centosplus 42 plc-elrepo 92 plc-elrepo-kernel 0 plc-epel 6,080 plc-extras 0 plc-os 6,019 plc-releases 0 plc-repoforge 3,011+930 plc-updates 1,042 plc-virtualbox 10 plc-virtualmin-universal 134 plnet-compiled 28 plnet-downloaded 89+16 plnet-releases 0 repolist: 16,549 real 0m28.276s user 0m14.475s sys 0m1.658s [root at mama ~]# yum clean all Loaded plugins: fastestmirror, priorities, refresh-packagekit Cleaning up Everything Cleaning up list of fastest mirrors [root at mama ~]# time yum repolist --disablerepo=*repoforge* --disablerepo=*epel* Loaded plugins: fastestmirror, priorities, refresh-packagekit Determining fastest mirrors <snip> 17 packages excluded due to repository priority protections repo id status playonlinux 2+6 plc-adobe-linux 0 plc-centosplus 42 plc-elrepo 92 plc-elrepo-kernel 0 plc-extras 0 plc-os 6,019 plc-releases 0 plc-updates 1,042 plc-virtualbox 10 plc-virtualmin-universal 134 plnet-compiled 28 plnet-downloaded 94+11 plnet-releases 0 repolist: 7,463 real 0m19.810s user 0m8.663s sys 0m1.227s -- Ljubomir Ljubojevic (Love is in the Air) PL Computers Serbia, Europe Google is the Mother, Google is the Father, and traceroute is your trusty Spiderman... StarOS, Mikrotik and CentOS/RHEL/Linux consultant