[CentOS] unfsd scalability issues

[Boris Epstein]

On Wed, Jun 13, 2012 at 10:11 AM, <m.roth at 5-cent.us> wrote:

> Boris Epstein wrote:
> > On Sat, Jun 2, 2012 at 2:50 PM, John R. Dennison <jrd at gerdesas.com>
> wrote:
> >> On Sat, Jun 02, 2012 at 10:59:13AM -0400, Boris Epstein wrote:
> <snip>
> > To be specific, I use UNFSD to export a MooseFS file system. MooseFS, by
> > the way, is userland-process based too.
> >
> > Be that as it may, I've seen situations where a comparably configured
> > MooseFS client get to read at, say, 40 MB/s - which is fine - but the
> > UNFSD at the same time reads at 40K/s(!) Why would that be? I mean, some
> > degradation I can dig but 3 orders of magnitude? What is with this? Am I
> > doing something wrong?
> <snip>
> I wonder... what's the architecture of what you're getting these results?
> I tried opening a bug with upstream over NFS4 and 6.x, and no one ever
> looked at it, and they closed it.
>
> 100% repeatably: unpack a package locally, seconds.
>                 unpack it from an NFS mount onto a local drive, about 1
> min.
>                 unpack it from an NFS mount onto an NFS mount, even when
>                    the target is exported FROM THE SAME MACHINE* that the
>                    process is running on: 6.5 - 7 MINUTES.
>
> * That is,
>     [server 1]                             [server 2]
>        /export/thatdir --NFS-->    /target/dir
>                                    /s2/source
>                                    /source/dir --NFS-->/s2/source
>     and cd [server 2]:/target/dir and unpack from /s2/source
>
> I suppose I'll try logging into upstream's bugzilla using our official
> licensed id; maybe then they'll assign someone to look at it....
>
>        mark
>
>
>
Mark,

Thanks, my architecture is extremely similar to yours, except that in my
case the "second layer", if I may say so, is MooseFS (
http://www.moosefs.org/ ), not NFS. MooseFS itself is blazing, by the way.

So the diagram in my case would look something like this:

       /export/thatdir --NFS-->    /target/dir
                                   /s2/source
                                   /source/dir -- MooseFS mount (mfsmount)
-->/s2/source

The discrepancy in the resultant performance is comparable.

Thanks.

Boris.