I reported the same problem (OOM's on a 5GB system which should have had 3-4 GB free). It's a pure kernel problem and hopefully the new U1 kernel fixes this issue - it's in the redhat bugzilla, to quote:
[this is the 2.6.9-5.0.5ELsmp kernel]
OK ... there are known memory leak issues on that kernel
Your problem is similar, but not exactly like:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=150971
we'll see whether the new U1 kernel helps... so long the situation seems much better... Cheers, MaZe.