Would you mount all filesystems noatime, or just those that wouldhave any effect on operation?
No. I usually only do this on partitions I share out over smb/cifs where I don't trust windows access timestamps anyway, or on filesystems where I don't care when the last time a file was accessed was (webroots, stuff like that). Mounting with noatime simply tells the filesystem not to update data/metadata to account for file access times. This saves a few cycles here and there, and can add up depending on how often something gets hit.
After that I move to /etc/sysctl.conf and tweak the kernel.shmmax, shmmin, shmall, and vdso values depending on the application I'm most concerned about, as well as fs.file-max.
Any hints on what to look for to determine the best settings?
Not really. Most of the apps we use here (oracle and netegrity stuff) have tuning recommendations in the documentation or associated kb/forums so I benchmark, change stuff to meet their recommendations, benchmark again, and use whichever tests better. It's mostly a trial and error kinda thing.
According to that article, I might be better off staying with the CFQ, but since this is a number crunching thing, I really don't know what to look for, or where to look to see what really is going on behind the monitor. I'm not savvy enough to know what the numerical model really does in the background is what I'm trying to say. Any suggestions, or could I provide more info that might help you help me?
Yeah, much of what I do here is webhosting/filehosting type stuff, so for me disk tuning is my performance limiter. I was hoping some other individuals would offer other performance tips after noticing that my advice was very heavily disk oriented. I'd be interested in hearing what other people are doing for performance tuning, but it seems like the vast majority of people simply leave the defaults and don't mess with things.