IMHO, I'd go with XMail (http://www.xmailserver.org). It's the easy'est to setup and maintain. all virtual accounts. and it's all contained in 1 directory for easy backup/restore and portability (it can also run under windows).
I've used : Postfix (although simple to setup with system accounts, I found it to be more complicated then it needed to be.). Qmail (as long as you take the qtoaster route it's a simple thing to setup. as long you do what the guides tell you. And using with tools like qmailadmin it's the simplest to maintain) Sendmail (well this dog of a mail server it just over complicated all the way around.) XMail (by far the simplest, smallest, and most versatile gpl mail system out there).
and as far as imap / pop3 server go I always use dovecot. really for the same reason I use Xmail, it's simple,small and fast !