I finally figured out my slowdown problem: I had somehow turned off write-caching on the 3Ware controller. Hoo-Boy! Does that kill throughput! What the heck is that option for anyway?
For cases where you do not want to lose your data when you get a blackout. If you do not have a battery power backup for your cache, you will lose data that is in the cache that has not been committed to the disks.
Here are a handful of bonnie++ benchmarks, I decided to just quote the block write and block read numbers:
MB/Sec Write ReadXFS: 231 202 ext2, dir_index: 221 205 ext3, dir_index, data=ordered: 80 196 ext3, dir_index, data=writeback: 95 199 ext3, data=writeback: 95 201
As you hinted, ext2 has almost the same performance as XFS. Data=writeback on ext3 helps some but not a whole lot. Dir_index doesn't seem to do a thing.
Indexes directories are only useful for cases where there are thousands of files in a directory and you want to access a single file (and you know the name in advance) quickly.
I'm really torn here. I can make use of the extra write speeds of ext2 or XFS. But is XFS stable and supported enough for 'production' use? Will I regret a forced fsck on a 1TB ext2 volume?
Are you using the no write cache flag with bonnie++? Otherwise you may not get the same results from whatever it is that you are running.
Steve, you say you've been happy with XFS for a few years. Have you been using it under any kind of load?
Run XFS without write caching and you should be safe. Are you creating thousands of files?