In the vein of the "Can't get resolution higher than 1280x1024" and "Another CentOS 5 oddity..." threads, I just installed CentOS5 w/Xen kernel on a Sony VAIO desktop with an nVidia NV5M64 (RIVA TNT2 64 Pro) card, using an old Samsung SyncMaster 700NF monitor.
system-config-display chose the highest possible resolution for the monitor and Xorg did its best to drive it, but it just isn't quite right: rhgb merely made a mess, which only sort of clears up when gdm starts. After I log in, only bits and pieces of menus and windows are painted.
Rebooting single user I re-ran system-config-display and it claimed to have written 1280x1024 into xorg.conf, but in fact it didn't: it omitted the "Screen" section entirely. Rhgb was still messed up, gdm still seemed to be at some very high resolution, and then after logging in I found myself at 800x600 with no option to set the resolution higher.
Three passes of switch-to-text-console-and-run system-config-display (including manually selecting my monitor) later, I finally have everything working, but only at 1024x768. If I choose any higher resolution, the repainting issues return.
I won't really need the GUI very often after I get other things set up, so I'm not particularly worried about this, but I thought I'd report it for benefit of anyone else who might encounter a problem with a similar video card. I'd be interested to hear further thoughts on whether this could be a kernel issue, as suggested by Preston Crawford. (I have installed all the updates, for what that's worth).