[CentOS] C6 server responding extremely slow on ssh interactive

I have a C6 server acting as a kvm-host.

When connecting with ssh the console is extremely slow and hangs for minutes at a time. Connecting to this server is not the problem.

If I use: ssh root at host "whatever" I got immediate response even when interactive consoles opened with ssh are hanging.

    Linux [...] 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

                  total       used       free     shared buffers     cached
    Mem:            47         35         11 0          0          0
    -/+ buffers/cache:         35         11
    Swap:            7          0          7

    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/vg-lv_root
                            50G  6,4G   41G  14% /
    tmpfs                  24G     0   24G   0% /dev/shm
    /dev/sda1             477M  123M  329M  28% /boot

      13:33:34 up 1 day, 18:30,  2 users,  load average: 3.39, 2.53, 2.36

(it's an 8-core)

Nothing particular in log/messages.

The vm's are running normally and they are not showing the same behaviour.

Can anybody give me a pointer?

Thanks
Patrick