Hi,
I'm setting up xen (on centos 5.3) and I'm facing a weird performance issue with iSCSI.
I attach the iSCSI targets to dom0 and if I use them from dom0 I obtain good performance with simple dd tests (~100MB/s both reading and writing).
I then use the block devices for the domU and if I repeat the same dd test from within the domU the write performace is still good (~100MB/s), but the read performance is cut in half (~55MB/s).
I tried changing several parameters like read_ahead and such, but I can not obtain a good read performance in the domU.
Any idea?
Thanks, Daniel.
Are you using separate physical NICs for dom0 (isolated) and the domUs?
In a CentOS domU I just, a few hours ago, had same issue on network reads, but doing:
# ethtool -K eth0 tx off
then rebooted domU and it cleared up and got much closer to max. Yes, that is the TX checksum fix, but it seemed to affect my reads (IO in).
Daniel De Marco wrote:
Hi,
I'm setting up xen (on centos 5.3) and I'm facing a weird performance issue with iSCSI.
I attach the iSCSI targets to dom0 and if I use them from dom0 I obtain good performance with simple dd tests (~100MB/s both reading and writing).
I then use the block devices for the domU and if I repeat the same dd test from within the domU the write performace is still good (~100MB/s), but the read performance is cut in half (~55MB/s).
I tried changing several parameters like read_ahead and such, but I can not obtain a good read performance in the domU.
Any idea?
Thanks, Daniel.
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Ben,
thanks for the suggestion, it didn't work, but it led me to test the network of the domu and it turns out that with netperf to a different machine I can get ~500Mbit/s from the domU and ~950Mbit/s from the dom0.
From another domU on a different dom0 (older systems, centos 5.3 with
kernel 2.6.18-92.1.22.el5xen i686 instead of 2.6.18-164.el5xen x86_64) netperf produces 950Mbit/s as well.
The weird thing is however that it's dom0 that acts as iSCSI initiator, so it is dom0 that is using the network and there it works fine.
I'll go do some more tests. Daniel.
* Ben Montanelli montanelli@rivint.com [09/24/2009 14:45]:
Are you using separate physical NICs for dom0 (isolated) and the domUs?
In a CentOS domU I just, a few hours ago, had same issue on network reads, but doing:
# ethtool -K eth0 tx off
then rebooted domU and it cleared up and got much closer to max. Yes, that is the TX checksum fix, but it seemed to affect my reads (IO in).
Daniel De Marco wrote:
Hi,
I'm setting up xen (on centos 5.3) and I'm facing a weird performance issue with iSCSI.
I attach the iSCSI targets to dom0 and if I use them from dom0 I obtain good performance with simple dd tests (~100MB/s both reading and writing).
I then use the block devices for the domU and if I repeat the same dd test from within the domU the write performace is still good (~100MB/s), but the read performance is cut in half (~55MB/s).
I tried changing several parameters like read_ahead and such, but I can not obtain a good read performance in the domU.
Any idea?
Thanks, Daniel.
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
I think that is an interesting coincidence about x86_64 vs. 32bit kernel. In this same CentOS pv domU I was talking about I created an x86_64 by "accident." I ended up knocking it down and going to i386 tree (its a web app) and found it to be snappier.
Please post your results especially if it is a kernel difference.
Daniel De Marco wrote:
Ben,
thanks for the suggestion, it didn't work, but it led me to test the network of the domu and it turns out that with netperf to a different machine I can get ~500Mbit/s from the domU and ~950Mbit/s from the dom0.
From another domU on a different dom0 (older systems, centos 5.3 with
kernel 2.6.18-92.1.22.el5xen i686 instead of 2.6.18-164.el5xen x86_64) netperf produces 950Mbit/s as well.
The weird thing is however that it's dom0 that acts as iSCSI initiator, so it is dom0 that is using the network and there it works fine.
I'll go do some more tests. Daniel.
- Ben Montanelli montanelli@rivint.com [09/24/2009 14:45]:
Are you using separate physical NICs for dom0 (isolated) and the domUs?
In a CentOS domU I just, a few hours ago, had same issue on network reads, but doing:
# ethtool -K eth0 tx off
then rebooted domU and it cleared up and got much closer to max. Yes, that is the TX checksum fix, but it seemed to affect my reads (IO in).
Daniel De Marco wrote:
Hi,
I'm setting up xen (on centos 5.3) and I'm facing a weird performance issue with iSCSI.
I attach the iSCSI targets to dom0 and if I use them from dom0 I obtain good performance with simple dd tests (~100MB/s both reading and writing).
I then use the block devices for the domU and if I repeat the same dd test from within the domU the write performace is still good (~100MB/s), but the read performance is cut in half (~55MB/s).
I tried changing several parameters like read_ahead and such, but I can not obtain a good read performance in the domU.
Any idea?
Thanks, Daniel.
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
* Ben Montanelli montanelli@rivint.com [09/24/2009 17:06]:
I think that is an interesting coincidence about x86_64 vs. 32bit kernel. In this same CentOS pv domU I was talking about I created an x86_64 by "accident." I ended up knocking it down and going to i386 tree (its a web app) and found it to be snappier.
I tested with a i386 domU and the results are the same.
Daniel.
Please disregard the network speed discrepancy. I was not comparing the same thing. Both dom0 and domU are doing 950Mbit/s, but the iSCSI part is still weird.
Daniel.
-
Ben Montanelli -
Daniel De Marco