I don't have any servers. I like CentOS on my desktop and my laptop just because it's solid. It's also the Linux distribution of choice for most Asterisk platforms -- which I intend to (eventually) learn. (I'm a telephone tech, who is eventually going to have to go VOIP.)
Have a look at FreeSwitch as well:)
You can get asterisk packages from rpmforge on Centos...but on Ubuntu you do not have to add an extra repository to get asterisk.
Don't bother with that, go straight to the source! http://packages.asterisk.org/ These get updated rather quickly.
jlc