[CentOS] DRBD very slow....

Coert Waagmeester lgroups at waagmeester.co.za
Fri Jul 24 07:28:56 UTC 2009


On Fri, 2009-07-24 at 10:21 +0400, Roman Savelyev wrote:
> 1. You are hit by Nagel alghoritm (slow TCP response). You can build DRBD 
> 8.3. In 8.3 "TCP_NODELAY" and "QUICK_RESPONSE" implemented in place.
> 2. You are hit by DRBD protocol. In most cases, "B" is enought.
> 3. You are hit by triple barriers. In most cases you are need only one of 
> "barrier, flush,  drain" - see documentation, it depens on type of storage 
> hardware.
> 

I have googled the triple barriers thing but cant find that much
information.

Would it help if I used IPv6 instead of IPv4?

Ross, here are the results of those tests you suggested:
________________________________________________________________________________________
For completeness here is my current setup:

host1: 10.99.99.2
Xeon Quad-Core
8GB RAM
Centos 5.3 64bit
2x 1TB seagate sata disks in software raid level 1
LVM on top of the raid for dom0 root fs and for all domU root FSses

host2: 10.99.99.1
Xeon Dual-Core
8GB RAM
Centos 5.3 64bit
2x 1TB seagate sata disks in software raid level 1
LVM on top of the raid for dom0 root fs and for all domU root FSses

common:
hosts are connected to local LAN
and directly to each other with a CAT6 gigabit crossover.

I have 6 DRBDs running for 5 domUs over the back to back link.
DRBD version drbd82-8.2.6-1.el5.centos
_______________________________________________________________________
_______________________________________________________________________




Ok, here is what I have done:

_______________________________________________________________________
I have added the following to the drbd config:
disk { no-disk-flushes;
         no-md-flushes; }

That made the resync go up to 50MB/sec after I issued a
drbdsetup /dev/drbdX syncer -r 110M

It used to stick around at 11MB/sec

As far as i can tell it has improved the domUs disk access as well.

I do see that there are a lot of warnings to be heeded with disk and 
metadata flushing......
_______________________________________________________________________

iperf results:

on host 1:
# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  5] local 10.99.99.1 port 5001 connected with 10.99.99.2 port 58183
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  1.16 GBytes    990 Mbits/sec


on host 2:
# iperf -c 10.99.99.1
------------------------------------------------------------
Client connecting to 10.99.99.1, TCP port 5001
TCP window size: 73.8 KByte (default)
------------------------------------------------------------
[  3] local 10.99.99.2 port 58183 connected with 10.99.99.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.16 GBytes    992 Mbits/sec


I am assuming those results are to be expected from a back to back
gigabit.
_______________________________________________________________________

the dd thing.
I think I did this completely wrong, how is this supposed to be done?

this is what i did

host 1:
nc -l 8123 | dd of=/mnt/data/1gig.file oflag=direct
(/mnt/data is an ext3 FS in LVM mounted on dom0)
(Not drbd) i first wanted to try it locally.

host 2:
date; dd if=/dev/zero bs=1M count=1000 | nc 10.99.99.2 8123 ; date


I did not wait for it to finish... according to ifstat the average speed
I got during this transfer was 1.6MB/sec

_______________________________________________________________________

Any tips would be greatly appreciated.




More information about the CentOS mailing list