[CentOS] centos email server suddenly much slower. What to do?

Thu Sep 6 17:14:56 UTC 2012
Marco Fioretti <marco.fioretti at gmail.com>

Greetings,
I run my own email server for some domains I administer, on a centos
vps server with a very small number of users.

The only services are smtp, imap/pop, webmail

Everything was running without problems until this morning. I left
home for 1/2 hours, and when I came back everything had become about
100x slower (seriously!). The services were/are still all up and
running, but practically unusable (even running mutt in my ssh session
is almost frozen).

I have not changed/updated anything in the last 1/2 weeks. I have
already done a few checks with the VPS provider, and it looks like:

network & hardware are OK

there seem to be no strange processes running. I didn't manage to save
the output of "top", but it didn't show anything that (AFAICT, of
course)  should not be there on an email server

however, there is something that is using "much more memory than
normal" (see the comment below from the hosting sysadmin after he
checked user_beancounters). Initially we thought it was apache, but
even switching it off didn't change anything. What now?

Any help to understand what the heck happened, and find out what
exactly _started_ to cause this problem is very welcome!

TIA,
Marco



root at vps [/etc/sysconfig]# cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld
         barrier                limit              failcnt
      712:  kmemsize                 17208298            162267136
      2147483646           2147483646                    0
            lockedpages                     0                    8
          999999               999999                    0
            privvmpages                 64694               262143
          262144               262144                   40
            shmpages                       14                 2366
          131072               131072                    0
            dummy                           0                    0
               0                    0                    0
            numproc                        63                  237
          999999               999999                    0
            physpages                   79977               262204
               0               262144                    0
            vmguarpages                     0                    0
          131072           2147483647                    0
            oomguarpages                30261                44087
          131072           2147483647                    0
            numtcpsock                     31                  243
         7999992              7999992                    0
            numflock                        9                   20
          999999               999999                    0
            numpty                          1                    1
          500000               500000                    0
            numsiginfo                      0                   27
          999999               999999                    0
            tcpsndbuf                  545000              7915960
       214748160            396774400                    0
            tcprcvbuf                  507904              3981312
       214748160            396774400                    0
            othersockbuf                21832              1229736
       214748160            396774400                    0
            dgramrcvbuf                     0               118400
       214748160            396774400                    0
            numothersock                   56                  356
         7999992              7999992                    0
            dcachesize               10775271            154640329
      2147483646           2147483646                    0
            numfile                       772                 1155
        23999976             23999976                    0
            dummy                           0                    0
               0                    0                    0
            dummy                           0                    0
               0                    0                    0
            dummy                           0                    0
               0                    0                    0
            numiptent                      57                   57
          999999               999999                    0


As you can see, there are some fails for the privvmpages. This means
your VPS tried to use more RAM than what is available (e.g. more than
1GB RAM). If you are only running some basic mail services on your
VPS, that's definitely not normal and you should investigate that
accordingly. We have fully checked everything for hardware and network
problems and everything is working flawlessly. In combination with the
RAM shortage errors, it is safe to conclude that there's something
within your VPS itself that's malfunctioning.