On Sun, 14 Jun 2020 at 14:49, Manuel Wolfshant <wolfy@nobugconsulting.ro> wrote:

Hello

    For the past months I've been testing upgrading my Xen hosts to CentOS 7 and I face an issue for which I need your help to solve.

    The testing machines are IBM blades, model H21 and H21XM. Initial tests were performed on the H21 with 16 GB RAM; during the last 6=7 weeks I've been using the H21XM with 64 GB. In all cases the guests were fully updated CentOS 7 -- initially 7.6 ( most recent at the time of the initial tests ), and respectively 7.8 for the tests performed during the last 2 months.  As host I used initially CentOS 6 with latest kernel available in the centos virt repo at the time of the tests and CentOS 7 with the latest kernel as well. As xen versions I tested 4.8 and 4.12 ( xl info included below ). The storage for the last tests is a Crucial MX500 but results were similar when using traditional HDD.

    My problem, in short, is that the guests are extremely slow. For instance , in the most recent tests, a yum install kernel takes cca 1 min on the host and 12-15 (!!!) minutes in the guest, all time being spent in dracut regenerating the initramfs images. I've done rough tests with the storage  ( via dd if=/dev/zero of=a_test_file size bs=10M count=1000 ) and the speed was comparable between the hosts and the guests. The version of the kernel in use inside the guest also did not seem to make any difference . OTOH, sysbench ( https://github.com/akopytov/sysbench/ ) as well as p7zip benchmark report for the guests a speed which is between 10% and 50% of the host. Quite obviously, changing the elevator had no influence either.

    Here is the info which I think that should be relevant for the software versions in use. Feel free to ask for any additional info.

Is there a way to boot up a PV guest versus an HVM?

If I understood the docs correctly, newer xen does only PVHVM ( xen_platform_pci=1 activates that ) and HVM. But they say it's better than PV. And I did verify, PVHVM is indeed enabled and active

I could not find a H21XM but found an HS21XM on the iBM

My bad. The blades are indeed HS21 (Type 8853) and HS21 XM (Type 7995). The XM blades have 2*Xeon E5450@3GHz / 12GB L1 cache processors. The options I can fiddle with are https://imgur.com/a/DonXe5P

AFAICS the setttings are reasonable but please do let me know if there is anything there that should not be as it is

site and that seemed to be a 4 core 8 thread cpu which looks 'old' enough that the Spectre/etc fixes to improve performance after the initial hit were not done. (Basically I was told that if the CPU was older than 2012, just turn off hyperthreading altogether to try and get back some performance.. but don't expect much).

I can live with that. My problem is that DomU are much much slower that Dom0 so it seems xen virtualization affects ( heavily ) the performance.

As such I would also try turning off HT on the CPU to see if that improves anything.

I got inspired by Adi's earlier suggestion and after reading https://access.redhat.com/articles/3311301 I've tried today all variants of disabling the spectre mitigations. Whatever I do, immediately after a reboot, yum reinstall kernel does not take less than 5 minutes :( It goes down to 2 min if I repeat the operation afterwards so I guess some caching kicks in. I will try later today the kernels from elrepo and maybe even xen.crc.id.au ( I kind of hate the "disable selinux" recommendation from the install page so I postponed it in the hope of other solution ).