[CentOS] Yum update slow on CentOS 4.1 x86_64?

Hi list

I have several CentOS 4.1 boxes, but only one is running x86_64 CentOS. 
yum update on all boxes is really fast, on the x86_64 box it is horribly 
slow. Even startup and "Setting up Repos" takes minutes to complete. The 
box is just idling all the time (no load at all) and is connected to an 8 
mbit DSL connection. Also the downloading of the headers seems to go very 
slow, about 1 kb per second or so?

I tested the internet connection (did wget on an 700 mb ISO image) which 
downloaded at near max speed so I am sortof ruling out the connection.

Top doesn't show any load when yum is running, nor is anything eating any 
cpu cycles.

Ideas anyone?