Restarting X doesn't (necessarily) restart gdm
You probably have to restart gdm ...
Yup, that did the trick. Setting "DisallowTCP=false" in /etc/gdm/gdm.conf and then restarting gdm (killall -HUP gdm-binary) let's me have remote X connections again without having to use ssh.
Thanks, Alfred