David Finch david@mytsoftware.com wrote:
Glad to know it's not just me.
It's not. I've seen it too.
Using ext3 on a 3ware raid 5 of four 250gb disks.
Doesn't matter what the disks are. The problem is the small cache size on the 3Ware Escalade 7000/8000. They only have 1-4MB of 0 wait state SRAM.
1MB on the original 7200/7210/7400/7410/7800/7810 and 7000-2 and 8000-2, as well the 7006-2 and 8006-2 (xxx6 = 66MHz PCI).
2MB on the 7450/7850 which are, subsequently, the 7500-4/-8 for PATA now, with the 8500-4/-8 and 8506-4/-8 (xxx6 = 66MHz PCI).
4MB on the 7506-12.
SRAM is very expensive, both size and cost-wise. It's the logic used in CPU cache and for networking ASICs. But it has little to no wait -- unlike DRAM which is still 40-70ns on reads (many many wait cycles, typically 6-10 for today's 133-266MHz clocks). That's why 3Ware calls the Escalade 7000/8000 series a "storage switch." It's ideal for RAID-0, 1 and 10.
This size is a serious issue when it comes to Ext3's journal logic, especially pre-2.4.18 kernels IIRC (maybe it was 2.4.15?). With only 2MB typical (4MB on the 7506-12), the commit of the Ext3 journal exceeds that size -- so the card "stalls" on the write when just committing the journal from the data.
Writes are slow and seem to halt the server until they complete, but it's not a server where response time or write speed is critical.
You can play with the kernel buffer settings. It's highly recommended for many of the 3Ware Escalade cards, including the 9000 series.
But if performance is a consideration, do _not_ use RAID-5 on the 3Ware Escalade 7000/8000. Use RAID-10. You can break over 200MBps _writes_ with RAID-10 on the 7000/8000 series.