[CentOS-virt] very low performance of Xen guests

On 15-06-2020 23:42, Manuel Wolfshant wrote:
> On 6/15/20 2:46 PM, Stephen John Smoogen wrote:
> 
>> On Sun, 14 Jun 2020 at 14:49, Manuel Wolfshant
>> <wolfy at nobugconsulting.ro> wrote:
>> 
>>> Hello
>>> 
>>> For the past months I've been testing upgrading my Xen hosts
>>> to CentOS 7 and I face an issue for which I need your help to
>>> solve.
>>> 
>>> The testing machines are IBM blades, model H21 and H21XM.
>>> Initial tests were performed on the H21 with 16 GB RAM; during the
>>> last 6=7 weeks I've been using the H21XM with 64 GB. In all cases
>>> the guests were fully updated CentOS 7 -- initially 7.6 ( most
>>> recent at the time of the initial tests ), and respectively 7.8
>>> for the tests performed during the last 2 months.  As host I used
>>> initially CentOS 6 with latest kernel available in the centos virt
>>> repo at the time of the tests and CentOS 7 with the latest kernel
>>> as well. As xen versions I tested 4.8 and 4.12 ( xl info included
>>> below ). The storage for the last tests is a Crucial MX500 but
>>> results were similar when using traditional HDD.
>>> 
>>> My problem, in short, is that the guests are extremely slow.
>>> For instance , in the most recent tests, a yum install kernel
>>> takes cca 1 min on the host and 12-15 (!!!) minutes in the guest,
>>> all time being spent in dracut regenerating the initramfs images.
>>> I've done rough tests with the storage  ( via dd if=/dev/zero
>>> of=a_test_file size bs=10M count=1000 ) and the speed was
>>> comparable between the hosts and the guests. The version of the
>>> kernel in use inside the guest also did not seem to make any
>>> difference . OTOH, sysbench (
>>> https://github.com/akopytov/sysbench/ ) as well as p7zip benchmark
>>> report for the guests a speed which is between 10% and 50% of the
>>> host. Quite obviously, changing the elevator had no influence
>>> either.
>>> 
>>> Here is the info which I think that should be relevant for the
>>> software versions in use. Feel free to ask for any additional
>>> info.
>> 
>> Is there a way to boot up a PV guest versus an HVM?
> 
> If I understood the docs correctly, newer xen does only PVHVM (
> xen_platform_pci=1 activates that ) and HVM. But they say it's better
> than PV. And I did verify, PVHVM is indeed enabled and active

You can also test running your Linux domUs as Xen PVH [1], which 
requires kernel 4.11 and Xen 4.10 as a minimum.

[1] https://wiki.xenproject.org/wiki/Linux_PVH

-- 
Adi Pircalabu