[CentOS-virt] iowait / Perfomance problems with xen on drbd (centos 5.2)

[Jan Kellermann]

Hi,

we have 2 servers on centos5.2 in a cluster with the redhat cman in the
configuration

SERVER1 > LVM \
CMAN           DRBD > XEN
SERVER2 > LVM /

For each xen we create a new local LVM on each node, put them in a drbd
and install an os (debian 4 or ubuntu 8). The xens are running as pvm.

Everything works fine since over 4 months now.

But we have some performance-Problems:

Writing on the Xen-devices produces an iowait about 50 to 60% on the
xen-processors.

we have 3 szenarios tested:

a) Xen on DRBD
b) Xen on DRBD but disconnected
c) direct mounted DRBD

You can see the difference in Write pro Char and Write per Block.

a: 3792 / 4011
b: 52037 / 103777
c: 57135 / 325002

See bonnie_result.txt for more data.

We attached our drbd.conf and a xen-config for your information.

The Server are each 2xDual-Core AMD Opteron(tm) Processor 2214 HE with
32 Gb RAM and 2TB-Harddrive on Raid5 running von CentOS 5.2. The
XEN-DomUs are Debian 4 or Ubuntu 8. The NICs are bonded Intel 1GBit. For
the DRBD we have an own connection on a seperately subnet.

Though we are wondering what there happens. Any Idea?

And: YES, we tried a some configurations in the last past months before
we are asking you today  :)

Best regards an Thank you in advance
Jan


-------------- next part --------------
resource server206 {
  protocol C;

  handlers {
    pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
    pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
    local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
  }

  startup {
    degr-wfc-timeout 120;    # 2 minutes.
    wait-after-sb;
  }

  disk {
    on-io-error   detach;
  }

  net {
    allow-two-primaries;
    after-sb-0pri discard-least-changes;
    after-sb-1pri violently-as0p;
    after-sb-2pri violently-as0p;
    rr-conflict violently;
    max-buffers 16384;
    max-epoch-size 16384;
    sndbuf-size 1M;

  }

  syncer {
    rate 100M;
    al-extents 3313;
  }

  on server101.werk21system.de {
    device     /dev/drbd6;
    disk       /dev/xendisk/server206;
    address    10.20.0.101:7795;
    meta-disk  internal;

  }

  on server102.werk21system.de {
    device    /dev/drbd6;
    disk      /dev/xendisk/server206    ;
    address   10.20.0.102:7795;
    meta-disk internal;
  }
}
-------------- next part --------------

## Ubuntu 64Bit Kernel
kernel = '/etc/xen/kernel/vmlinuz-2.6.24-19-xen'
ramdisk = '/etc/xen/kernel/initrd.img-2.6.24-19-xen'

memory  = '1024'
root    = '/dev/xvda ro'
disk    = [ 'drbd:server206,xvda,w'  ]
name    = 'server206'
vcpus = 2
vif  = [ 'ip=X.Y.Z.205,bridge=xenbr0','ip=10.10.10.205,bridge=xenbr1' ]
vfb = [ 'type=vnc,vncunused=1,keymap=de' ]
keymap = 'de'
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
-------------- next part --------------
=================================================================================

XEN ON DRBD CONNECTED

Version 1.03b       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ubuntu64         2G  3792   6  4011   0  3170   0 33006  40 182491   6 231.1   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++


=================================================================================

XEN ON DRBD DISCONNECTED

Version 1.03b       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ubuntu64         2G 52037  85 103777  26 43139   5 33510  41 174650   5 212.8   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
ubuntu64,2G,52037,85,103777,26,43139,5,33510,41,174650,5,212.8,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

=================================================================================


DRBD CONNECTED W/O XEN

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
server101        2G 57135  97 325002  94 16498   3 61150  94 1166962 100 +++++ +++
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
server101,2G,57135,97,325002,94,16498,3,61150,94,1166962,100,+++++,+++,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++