[CentOS] Yum update slow on CentOS 4.1 x86_64?

Hi list

I have several CentOS 4.1 boxes, but only one is running x86_64 CentOS. yum update on all boxes is really fast, on the x86_64 box it is horribly slow. Even startup and "Setting up Repos" takes minutes to complete. The box is just idling all the time (no load at all) and is connected to an 8 mbit DSL connection. Also the downloading of the headers seems to go very slow, about 1 kb per second or so?

I tested the internet connection (did wget on an 700 mb ISO image) which downloaded at near max speed so I am sortof ruling out the connection.

Top doesn't show any load when yum is running, nor is anything eating any cpu cycles.

Ideas anyone?