[CentOS] My SSL connection is very slow

Hello all.

I have a problem since months with my CentOS 4 webserver. It came up since I 
intalled it.
The SSL pages are very slow. Typically, I get 2-3 thumbnails from an SSL 
page, and then nothing during 7 seconds, and then again 2 thumbnails, 
nothing again, etc.. and so forth during about 30 seconds after what all non 
loaded images are just discarded (time out).
It also happens with the normal HTTP, but much less, and is usable.

I must have done an error in the configuration, but I don't know where.
I have tried taking the ssl.conf, httpd.conf and php.ini from another clean 
CentOS server (that does not present this) but after rebooting the services 
nothing changes. I don't see anything strange in the logs neither. It just 
takes very long to answer a request.
I also tried to stop MySQL and Tomcat, but no chance.

Did someone already see that kind of behaviour ?

Kind regards,
Daniel