[CentOS] OT: Caching synchronous writes

Thu Apr 22 20:20:45 UTC 2010
Ray Van Dolson <rayvd at bludgeon.org>

[ Wish there was a generic, active Linux "storage" mailing list out
  there -- something other than the kernel lists I mean ]

To frame the discussion, we use VMware ESX (vSphere) quite a bit with
NFS datastores.  Often times with NetApp, but lately, more often with
Solaris 10 + ZFS + SSD's for ZIL (intent log or write cache).

The ZIL lets us use synchronous writes (safer) without the normal
delay.  Were we to try and get the same level of performance with
Linux, we'd need to use async mode for our NFS shares -- and we'd lose
some reliability.

However, given the latest rumblings and ruminations about Oracle
potentially no longer selling entitlements for Solaris 10 on non-Sun
hardware -- and then turning around and no longer allowing you to run
Solaris 10 "freely", we're left with either OpenSolaris or looking at
Linux again (we run Solaris 10 on Silicon Mechanics hardware).

My question is, what are the various options for getting NFS in "sync"
mode to run fast on Linux?

Obviously we can buy a really nice disk controller with lots of cache,
but I'm thinking more at the filesystem, volume manager or block driver
layer.  Is there a way to shunt write requests to a quicker medium like
an SLC-based SSD (or NVRAM)?

I don't see a way to do this with LVM, ext3/ext4 or even xfs... maybe
btrfs will have some options along this line down the road, but that's
tomorrow and not today.

So is a beefy disk controller our best option?  Even using our 3Ware
9650's w/ BBU so we can enable write-back doesn't seem to give us as
good of write performance via NFS as ZFS+ZIL-on-SSD does...