On Fedora Core 6 yum with yum-fastestmirror dies if ulimit for address space (as reported by ulimit -v) is set and too low. In my tests 'too low' means anything below about 900MB. I believe it depends on the number of mirrors - for each one a thread is created with a few MB of address space cost.
I am wondering if this is same in CentOS 5 (I have not it installed, so I cannot cehck myself at the moment)
I don't know Fedora's case but in my own CentOS boxes I usually don't install fastest-mirror plugin because the same reason you say: the yum management speed becomes slow.
:S