That's definitly true and why I try to leave most of the code as-is and add nearly no additional modules. I personally like Joomla very much. There where some annoying security issues in the past but I still believe in the stable code-base.
I've been lurking here a little while and thought I'd add my 2 cents on this topic ...
The strategy above is a sensible one. I've found that while many open source CMS products are very good at what they do, they are not "enterprise" products in the sense that development sometimes proceeds very quickly, driven by the laudable enthusiasm of the developers, but creating headaches for users who face a constant upgrade treadmill (apart from security fixes). To make matters worse, plug-in developers don't always keep up with the code base in a timely manner. Take the Drupal project for example, which is currently juggling concurrent support for versions 5 and 6 with 7 just around the corner.
Personally, I am now avoiding CMSs for my own projects (mainly documentation type stuff from now on) for the reasons above and tending more towards XML (DocBook) and XSLT. This frees me from the CMS upgrade treadmill and the constant potential for security issues needing fixing, and has the added benefits of being very flexible in how content is formatted and presented. Content can either be pre-generated and uploaded as static HTML, or generated on the fly with one pretty basic script. I realise that such a solution would be unworkable for a community site, however.