[CentOS] DRBD very slow....

Thu Jul 23 07:44:45 UTC 2009
Coert Waagmeester <lgroups at waagmeester.co.za>

Hello all,


For completeness here is my current setup:

host1:
Xeon Quad-Core
8GB RAM
Centos 5.3 64bit
2x 1TB seagate sata disks in software raid level 1
LVM on top of the raid for dom0 root fs and for all domU root FSses

host2:
Xeon Dual-Core
8GB RAM
Centos 5.3 64bit
2x 1TB seagate sata disks in software raid level 1
LVM on top of the raid for dom0 root fs and for all domU root FSses

common:
hosts are connected to local LAN
and directly to each other with a CAT6 gigabit crossover.

I have 6 DRBDs running for 5 domUs over the back to back link.
DRBD version drbd82-8.2.6-1.el5.centos
_______________________________________________________________________
_______________________________________________________________________




Ok, here is what I have done:

_______________________________________________________________________
I have added the following to the drbd config:
disk { no-disk-flushes;
         no-md-flushes; }

That made the resync go up to 50MB/sec after I issued a
drbdsetup /dev/drbdX syncer -r 110M

It used to stick around at 11MB/sec

As far as i can tell it has improved the domUs disk access as well.

I do see that there are a lot of warnings to be heeded with disk and 
metadata flushing......
_______________________________________________________________________

iperf results:

on host 1:
# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  5] local 10.99.99.1 port 5001 connected with 10.99.99.2 port 58183
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  1.16 GBytes    990 Mbits/sec


on host 2:
# iperf -c 10.99.99.1
------------------------------------------------------------
Client connecting to 10.99.99.1, TCP port 5001
TCP window size: 73.8 KByte (default)
------------------------------------------------------------
[  3] local 10.99.99.2 port 58183 connected with 10.99.99.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.16 GBytes    992 Mbits/sec


I am assuming those results are to be expected from a back to back
gigabit.
_______________________________________________________________________

the dd thing.
I think I did this completely wrong, how is this supposed to be done?

this is what i did

host 1:
nc -l 8123 | dd of=/mnt/data/1gig.file oflag=direct
(/mnt/data is an ext3 FS in LVM mounted on dom0)

host 2:
date; dd if=/dev/zero bs=1M count=1000 | nc 10.99.99.2 8123 ; date


I did not wait for it to finish... according to ifstat the average speed
I got during this transfer was 1.6MB/sec

_______________________________________________________________________

Any tips would be greatly appreciated.


Kind regards,
Coert