[CentOS] cache memory not being used

Since I upgraded to 4.2 it looks like the cache memory doesn't go over about 70K or so. I have over 700K free and it started using swap, which doesn't normally happen for a few days or so. Any hints on what might be up?

This is just after a compile and the cached is only 75k total used free shared buffers cached Mem: 1032848 263992 768856 0 4612 75628 -/+ buffers/cache: 183752 849096 Swap: 1052248 18260 1033988

On a side note, anyone able to have Xorg running for more then 14 days? It consumes about 750megs after 14 days or so.