[CentOS-virt] very low performance of Xen guests

Manuel Wolfshant

wolfy at nobugconsulting.ro
Mon Jun 15 13:42:05 UTC 2020

On 6/15/20 2:46 PM, Stephen John Smoogen wrote:
> On Sun, 14 Jun 2020 at 14:49, Manuel Wolfshant 
> <wolfy at nobugconsulting.ro <mailto:wolfy at nobugconsulting.ro>> wrote:
>     Hello
>         For the past months I've been testing upgrading my Xen hosts
>     to CentOS 7 and I face an issue for which I need your help to solve.
>         The testing machines are IBM blades, model H21 and H21XM.
>     Initial tests were performed on the H21 with 16 GB RAM; during the
>     last 6=7 weeks I've been using the H21XM with 64 GB. In all cases
>     the guests were fully updated CentOS 7 -- initially 7.6 ( most
>     recent at the time of the initial tests ), and respectively 7.8
>     for the tests performed during the last 2 months.  As host I used
>     initially CentOS 6 with latest kernel available in the centos virt
>     repo at the time of the tests and CentOS 7 with the latest kernel
>     as well. As xen versions I tested 4.8 and 4.12 ( xl info included
>     below ). The storage for the last tests is a Crucial MX500 but
>     results were similar when using traditional HDD.
>         My problem, in short, is that the guests are extremely slow.
>     For instance , in the most recent tests, a yum install kernel
>     takes cca 1 min on the host and 12-15 (!!!) minutes in the guest,
>     all time being spent in dracut regenerating the initramfs images.
>     I've done rough tests with the storage  ( via dd if=/dev/zero
>     of=a_test_file size bs=10M count=1000 ) and the speed was
>     comparable between the hosts and the guests. The version of the
>     kernel in use inside the guest also did not seem to make any
>     difference . OTOH, sysbench (
>     https://github.com/akopytov/sysbench/ ) as well as p7zip benchmark
>     report for the guests a speed which is between 10% and 50% of the
>     host. Quite obviously, changing the elevator had no influence either.
>         Here is the info which I think that should be relevant for the
>     software versions in use. Feel free to ask for any additional info.
> Is there a way to boot up a PV guest versus an HVM?

If I understood the docs correctly, newer xen does only PVHVM ( 
xen_platform_pci=1 activates that ) and HVM. But they say it's better 
than PV. And I did verify, PVHVM is indeed enabled and active

> I could not find a H21XM but found an HS21XM on the iBM

My bad. The blades are indeed HS21 (Type 8853) and HS21 XM (Type 7995). 
The XM blades have 2*Xeon E5450 at 3GHz / 12GB L1 cache processors. The 
options I can fiddle with are https://imgur.com/a/DonXe5P

AFAICS the setttings are reasonable but please do let me know if there 
is anything there that should not be as it is

> site and that seemed to be a 4 core 8 thread cpu which looks 'old' 
> enough that the Spectre/etc fixes to improve performance after the 
> initial hit were not done. (Basically I was told that if the CPU was 
> older than 2012, just turn off hyperthreading altogether to try and 
> get back some performance.. but don't expect much).

I can live with that. My problem is that DomU are much much slower that 
Dom0 so it seems xen virtualization affects ( heavily ) the performance.

> As such I would also try turning off HT on the CPU to see if that 
> improves anything.

I got inspired by Adi's earlier suggestion and after reading 
https://access.redhat.com/articles/3311301 I've tried today all variants 
of disabling the spectre mitigations. Whatever I do, immediately after a 
reboot, yum reinstall kernel does not take less than 5 minutes :( It 
goes down to 2 min if I repeat the operation afterwards so I guess some 
caching kicks in. I will try later today the kernels from elrepo and 
maybe even xen.crc.id.au ( I kind of hate the "disable selinux" 
recommendation from the install page so I postponed it in the hope of 
other solution ).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20200615/37b3f4ca/attachment.html>

More information about the CentOS-virt mailing list