[CentOS] Openssh connection closes from time to time. why?

Hey there,

I just replaced long lived Gentoo to a Centos 6.3 on 2 servers.
Since the move I had troubles with ssh open sessions.
I am connecting from a windows machine via putty and then all of a 
sudden it saying something about connection closing.
The OpenSSH version is: openssh-server-5.3p1-81.el6.x86_64.rpm
I tired to configure:
tcp_keep_alive
TCPKeepAlive yes
ClientAliveInterval 240
ClientAliveCountMax 50

Each one of them or all together seems to not solve the problem.
It's not iptables issue since it was stopped and disabled.

If someone have an idea of what can be done or checked I will be happy 
to hear about it.

For now I have created a custom OpenSSH-6.1p1 RPM just in case it will 
might help.

Both server has the same issue while another Fedora machine runs by 
which doesn't have the same issues.

They are both connected to the same switch and seems to be fine.

Thanks,
Eliezer