[CentOS] DRBD very slow....

[Ross Walker]

On Jul 22, 2009, at 5:16 AM, Coert Waagmeester <lgroups at waagmeester.co.za 
 > wrote:

> Hello all,
>
> we have a new setup with xen on centos5.3
>
> I run drbd from lvm volumes to mirror data between the two servers.
>
> both servers are 1U nec rack mounts with 8GB RAM, 2x mirrored 1TB
> seagate satas.
>
> The one is a dual core xeon, and the other a quad-core xeon.
>
> I have a gigabit crossover link between the two with an MTU of 9000 on
> each end.
>
> I currently have 6 drbds mirroring across that link.
>
> The highest speed I can get through that link with drbd is 11 MB/sec
> (megabytes)
>
> But if I copy a 1 gig file over that link I get 110 MB/sec.
>
> Why is DRBD so slow?
>
> I am not using drbd encryption because of the back to back link.
> Here is a part of my drbd config:
>
> # cat /etc/drbd.conf
> global {
>  usage-count yes;
> }
> common {
>  protocol C;
>  syncer { rate 80M; }
>  net {
>    allow-two-primaries;
>  }
> }
> resource xenotrs {
>  device    /dev/drbd6;
>  disk      /dev/vg0/xenotrs;
>  meta-disk internal;
>
>  on baldur.somedomain.local {
>    address   10.99.99.1:7793;
>  }
>  on thor.somedomain.local {
>    address   10.99.99.2:7793;
>  }
> }

Use iperf to measure the bandwidth/latency on those nics between the  
two hosts.

If iperf comes back clean use dd with the oflag=direct to test the  
performance of the drives on both sides (create a test LV). Roughly  
you can multiply with the max number of outstanding I/Os your  
application does to get a real number, use 4 if you don't know what  
that is.

DRBD protocol C is completely synchronous and won't return a write  
until it has been committed to disk on both sides.

Having disk controllers with nvram cache can make all the difference  
in the world for this setup.

-Ross