[CentOS] DRBD very slow....

[Coert Waagmeester]

On Mon, 2009-07-27 at 18:18 -0400, Ross Walker wrote:
> On Jul 27, 2009, at 4:09 PM, Coert Waagmeester
> <lgroups at waagmeester.co.za> wrote:
> 
> 
> 
> 
> > 
> > On Mon, 2009-07-27 at 12:37 +0200, Coert Waagmeester wrote:
> > > On Mon, 2009-07-27 at 12:02 +0200, Alexander Dalloz wrote:
> > > > > 
> > > > > On Mon, 2009-07-27 at 08:30 +0200, Coert Waagmeester wrote:
> > > > 
> > > > > > Hello Roman,
> > > > > > 
> > > > > > I am running drbd 8.2.6 (the standard centos version)
> > > > 
> > > > have you considered to test the drbd-8.3 packages?
> > > > 
> > > > http://bugs.centos.org/view.php?id=3598
> > > > 
> > > > http://dev.centos.org/centos/5/testing/{i386,x86_64}/RPMS/
> > > > 
> > > 
> > > Thank you very much for this tip! It was one very obvious place
> > > where I
> > > did not look yet.
> > > 
> > > 
> > > Would it be necessary to still recompile it for the TCP_NODELAY
> > > and
> > > such?
> > > 
> > > I am just making sure, because
> > > http://www.nabble.com/Huge-latency-issue-with-8.2.6-td18947965.html
> > > makes it seem unnecessary.
> > > 
> > > Why do the repositories provide both DRBD 8.0.x and 8.2.6?
> > > 
> > 
> > Here is a status update....
> > _______________________________________________________________________________
> > on both hosts I now run from the testing repository:
> > # rpm -qa | grep drbd
> > drbd83-8.3.1-5.el5.centos
> > kmod-drbd83-xen-8.3.1-4.el5.centos
> > _______________________________________________________________________________
> > Here is my config (slightly condensed):
> > -----------------------------------------------------
> > global {
> >  usage-count yes;
> > }
> > common {
> >  protocol C;
> >  syncer { rate 50M; }
> >  net {
> > #        allow-two-primaries; }
> >         sndbuf-size 0; }
> > #  disk {no-disk-flushes;
> > #        no-md-flushes; }
> >  startup { wfc-timeout 0 ; }
> > }
> > resource xenfilesrv {
> >  device    /dev/drbd1;
> >  disk      /dev/vg0/xenfilesrv;
> >  meta-disk internal;
> > 
> >  on baldur.mydomain.local {
> >    address   10.99.99.1:7788;
> >  }
> >  on thor.mydomain.local {
> >    address   10.99.99.2:7788;
> >  }
> > }
> > resource xenfilesrvdata {
> >  device    /dev/drbd2;
> >  disk      /dev/vg0/xenfilesrvdata;
> >  meta-disk internal;
> > 
> >  on baldur.mydomain.local {
> >    address   10.99.99.1:7789;
> >  }
> >  on thor.mydomain.local {
> >    address   10.99.99.2:7789;
> >  }
> > }
> > _______________________________________________________________________________
> > 
> > xenfilesrv is a xen domU
> > in this domU i ran a dd with oflag direct:
> > ---------------------------------------------------------
> > # dd if=/dev/zero of=1gig.file bs=1M count=1000 oflag=direct
> > 1000+0 records in
> > 1000+0 records out
> > 1048576000 bytes (1.0 GB) copied, 147.997 seconds, 7.1 MB/s
> > 
> > Just before I ran the dd this popped up in the secondary hosts
> > syslog:
> > ----------------------------------------------------------------------
> > Jul 27 21:51:42 thor kernel: drbd2: Method to ensure write ordering:
> > flush
> > Jul 27 21:51:42 thor kernel: drbd1: Method to ensure write ordering:
> > flush
> > 
> > 
> > _______________________________________________________________________________
> > 
> > What more can I try?
> > 
> > To be quite honest, I have no idea what to do with/ where to find
> > the
> > TCP_NODELAY socket options......
> > 
> 
> 
> Use drbd option to disable flush/sync, but understand that during a
> power failure or system crash data will not be consistent on disk and
> you will need to sync the storage from the other server.
> 
> 
> -Ross
> 
> 
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos

That also does not really make a difference.
According to DRBD everything goes into barrier mode.

I still get speed of around 7.5 MB/sec

In the config i now have this:  
disk { no-disk-barrier;
       no-disk-flushes;
       no-md-flushes; }

according to /proc/drbd it then goes into 'drain' mode.

I still get only 8MB/sec throughput.

Would it be unwise to consider using Protocol A?

I have just tried Protocol A, and I also only get 8 MB/sec.
But, if I disconnect the secondary node, and do the dd again, I get
32MB/sec!


PS I sent another mail with an attachment. Have a feeling that is
moderated though....