[CentOS] connection speeds between nodes

[Ross Walker]

On Mar 8, 2011, at 12:25 PM, John Hodrien <J.H.Hodrien at leeds.ac.uk> wrote:

> On Tue, 8 Mar 2011, Ross Walker wrote:
> 
>> Well on my local disk I don't cache the data of tens or hundreds of clients
>> and a server can have a memory fault and oops just as easily as any client.
>> 
>> Also I believe it doesn't sync every single write (unless mounted on the
>> client sync which is only for special cases and not what I am talking about)
>> only when the client issues a sync or when the file is closed. The client is
>> free to use async io if it wants, but the server SHOULD respect the clients
>> wishes for synchronous io.
>> 
>> If you set the server 'async' then all io is async whether the client wants
>> it or not.
> 
> I think you're right that this is how it should work, I'm just not entirely
> sure that's actually generally the case (whether that's because typical
> applications try to do sync writes or if it's for other reasons, I don't
> know).

As always YMMV, but on the whole it's how it works.

ESX is an exception, it does O_FSYNC on each write cause it needs to know for certain that each completed.

> Figures for just changing the server to sync, everything else identical.
> Client does not have 'sync' set as a mount option.  Both attached to the same
> gigabit switch (so favouring sync as far as you reasonably could with
> gigabit):
> 
> sync;time (dd if=/dev/zero of=testfile bs=1M count=10000;sync)
> 
> async: 78.8MB/sec
>  sync: 65.4MB/sec
> 
> That seems like a big enough performance hit to me to at least consider the
> merits of running async.

Yes, disabling the safety feature will make it run faster. Just as disabling the safety on a gun will make it faster in a draw.

> That said, running dd with oflag=direct appears to bring the performance up to
> async levels:
> 
> oflag=direct with  sync nfs export: 81.5 MB/s
> oflag=direct with async nfs export: 87.4 MB/s
> 
> But if you've not got control over how your application writes out to disk,
> that's no help.

Most apps unfortunately don't allow one to configure how it handles io reads/writes, so you're stuck with how it behaves.

A good sized battery backed write-back cache will often negate the O_FSYNC penalty.

-Ross