You won’t be installing GUI! Trust me, yum install virt-manager xauth – Those are the only packages that will be installed, (plus any required support packages, which are minimal)
Then via your SSH session, you get the gui running on *your* local X server – not the remote server. Awesome for remote admin of servers, and also compressed via ssh, and most of my colo servers prioritise ssh traffic, so very good connections, even if your servers are being hammered for traffic. Win win J