Jeff Johnson wrote:
You are correct that the scaling depends on the number of packages not the number of repositories.
However the solution to a distributed lookup scaling problem *does* depend on the number of places that have to be searched as well as the cost of a failed lookup. If you have to look in a large number of repositories to ensure that some packages does NOT exist anywhere, well there are ways to do that efficiently.
And none of the right solutions to the increasing cost of a failed lookup are implemented in yum afiak.
I was hoping to get an estimate of how bad the scaling problem actually is from an objective wall clock time seat-of-the-pants measurement.
Meanwhile I'm happy that you've found a workable solution for your purposes. I'm rather more interested in what happens when there hundreds of repositories and 10's of thousands of packages that MUST be searched.
I suspect that yum will melt into a puddle if/when faced with depsolving on that scale. Not that anyone needs depsolving on the scale of hundreds of repos and 10's of thousands of packages in the "real world", but that isn't a proper justification for not considering the cost of a failed lookup carefully which (from what you are telling me) you are already seeing, and dealing with by enabling/disabling repositories and inserting a high priority repository that is also acting as a de facto cache and "working set" for the most useful packages.
Here are 2 speed tests. Repos are on 100Mbps LAN. First is with all repositories enabled and (left from CentOS 5 and old style RPMForge) aTrpms has higher priority over RepoForge. Second is with only safe RepoForge and other default (in my sistem) repositories enabled:
[root@mama ~]# yum clean all Loaded plugins: fastestmirror, priorities, refresh-packagekit Cleaning up Everything Cleaning up list of fastest mirrors [root@mama ~]# time yum install vlc --enablerepo=* --showduplicates Loaded plugins: fastestmirror, priorities, refresh-packagekit Loading mirror speeds from cached hostfile 2233 packages excluded due to repository priority protections Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package vlc.x86_64 0:1.1.10-71.el6 set to be updated --> Processing Dependency: bitstream-vera-serif-fonts for package: vlc-1.1.10-71.el6.x86_64 <snip> --> Finished Dependency Resolution Error: Package: vlc-1.1.10-71.el6.x86_64 (plc-atrpms-stable) Requires: libmodplug.so.0()(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
real 1m9.365s user 0m36.585s sys 0m2.830s
[root@mama ~]# time yum install vlc --showduplicates Loaded plugins: fastestmirror, priorities, refresh-packagekit Loading mirror speeds from cached hostfile 952 packages excluded due to repository priority protections Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package vlc.x86_64 0:1.1.11-1.el6.rf set to be updated --> Processing Dependency: libavformat.so.52(LIBAVFORMAT_52)(64bit) for package: vlc-1.1.11-1.el6.rf.x86_64 <snip> --> Finished Dependency Resolution Error: Package: vlc-1.1.11-1.el6.rf.x86_64 (plc-repoforge) Requires: libmodplug.so.0()(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
real 0m30.869s user 0m25.619s sys 0m1.261s
Difference is double, but time difference is only 30sec on low budget PC (1GB RAM, AMD Sempron 64-bit, 40GB IDE HDD).
And first run had to download yum repo files, second was from cache.
If you have some specific stres test I would be happy to run it.
Oh, yeah, yum reads and process xml files, not actual files, so searches are fast because of it.
Here are repolist data (1=all repos, 2=default, 3 = default with no rpmforge and epel):
[root@mama ~]# yum clean all Loaded plugins: fastestmirror, priorities, refresh-packagekit Cleaning up Everything Cleaning up list of fastest mirrors [root@mama ~]# time yum repolist --enablerepo=* Loaded plugins: fastestmirror, priorities, refresh-packagekit Loading mirror speeds from cached hostfile 2233 packages excluded due to repository priority protections
<snip>
repo id repo name status playonlinux 2+6 plc-adobe-linux 0 plc-atrpms-stable 1,636+382 plc-atrpms-testing 43+67 plc-c6-testing 0 plc-centosplus 42 plc-elrepo 92 plc-elrepo-extras 0 plc-elrepo-fasttrack 0 plc-elrepo-kernel 0 plc-elrepo-testing 31 plc-epel 6,080 plc-extras 0 plc-fasttrack 0 plc-kb-el6-ext 0 plc-kb-el6-ext-test 0 plc-kb-el6-misc 0 plc-kb-el6-misc-test 0 plc-os 6,019 plc-releases 0 plc-remi 114+223 plc-remi-test 2+76 plc-repoforge 2,835+1,106 plc-repoforge-buildtools 0 plc-repoforge-dag 0 plc-repoforge-extras 25+357 plc-rpmfusion-free-updates 0 plc-rpmfusion-free-updates-testing 70 plc-rpmfusion-nonfree-updates 0 plc-rpmfusion-nonfree-updates-testing 70 plc-sernet-samba 0 plc-updates 1,042 plc-virtualbox 10 plc-virtualmin-universal 134 plnet-archive 0 plnet-compiled 28 plnet-downloaded 89+16 plnet-releases 0 plnet-replace 0 plnet-test 0 repolist: 18,364
real 0m29.675s user 0m15.167s sys 0m1.633s
[root@mama ~]# yum clean all Loaded plugins: fastestmirror, priorities, refresh-packagekit Cleaning up Everything Cleaning up list of fastest mirrors [root@mama ~]# time yum repolist Loaded plugins: fastestmirror, priorities, refresh-packagekit Determining fastest mirrors
<snip>
952 packages excluded due to repository priority protections repo id status playonlinux 2+6 plc-adobe-linux 0 plc-centosplus 42 plc-elrepo 92 plc-elrepo-kernel 0 plc-epel 6,080 plc-extras 0 plc-os 6,019 plc-releases 0 plc-repoforge 3,011+930 plc-updates 1,042 plc-virtualbox 10 plc-virtualmin-universal 134 plnet-compiled 28 plnet-downloaded 89+16 plnet-releases 0 repolist: 16,549
real 0m28.276s user 0m14.475s sys 0m1.658s
[root@mama ~]# yum clean all Loaded plugins: fastestmirror, priorities, refresh-packagekit Cleaning up Everything Cleaning up list of fastest mirrors [root@mama ~]# time yum repolist --disablerepo=*repoforge* --disablerepo=*epel* Loaded plugins: fastestmirror, priorities, refresh-packagekit Determining fastest mirrors
<snip>
17 packages excluded due to repository priority protections repo id status playonlinux 2+6 plc-adobe-linux 0 plc-centosplus 42 plc-elrepo 92 plc-elrepo-kernel 0 plc-extras 0 plc-os 6,019 plc-releases 0 plc-updates 1,042 plc-virtualbox 10 plc-virtualmin-universal 134 plnet-compiled 28 plnet-downloaded 94+11 plnet-releases 0 repolist: 7,463
real 0m19.810s user 0m8.663s sys 0m1.227s