I have an email server running Exim, Dovecot, Spamassassin, Clam, etc.
Firstly, I'd drop dovecot completely, cyrus-imapd has, for me, been a lot faster and better optimised for situations where you have more than a few hand full of users.
Secondly, exim configs out of the box on CentOS are *really* not optimized for performance at all. If you handle more than a few thousand emails an hour, I dont be surprised if exim is to blame for jamming your i/o pipe just trying to work out which email to attempt delivery for next.
This is a Directadmin type server. So I really have no options on changing services. I am running exim version 4.60 and I wander if updating to 4.69 may help out.
Matt