<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello</p>
<p><br>
</p>
<p> For the past months I've been testing upgrading my Xen hosts
to CentOS 7 and I face an issue for which I need your help to
solve.</p>
<p> The testing machines are IBM blades, model H21 and H21XM.
Initial tests were performed on the H21 with 16 GB RAM; during the
last 6=7 weeks I've been using the H21XM with 64 GB. In all cases
the guests were fully updated CentOS 7 -- initially 7.6 ( most
recent at the time of the initial tests ), and respectively 7.8
for the tests performed during the last 2 months. As host I used
initially CentOS 6 with latest kernel available in the centos virt
repo at the time of the tests and CentOS 7 with the latest kernel
as well. As xen versions I tested 4.8 and 4.12 ( xl info included
below ). The storage for the last tests is a Crucial MX500 but
results were similar when using traditional HDD.<br>
</p>
<p> My problem, in short, is that the guests are extremely slow.
For instance , in the most recent tests, a yum install kernel
takes cca 1 min on the host and 12-15 (!!!) minutes in the guest,
all time being spent in dracut regenerating the initramfs images.
I've done rough tests with the storage ( via dd if=/dev/zero
of=a_test_file size bs=10M count=1000 ) and the speed was
comparable between the hosts and the guests. The version of the
kernel in use inside the guest also did not seem to make any
difference . OTOH, sysbench (
<a class="moz-txt-link-freetext" href="https://github.com/akopytov/sysbench/">https://github.com/akopytov/sysbench/</a> ) as well as p7zip benchmark
report for the guests a speed which is between 10% and 50% of the
host. Quite obviously, changing the elevator had no influence
either. <br>
</p>
<p> Here is the info which I think that should be relevant for
the software versions in use. Feel free to ask for any additional
info.<br>
</p>
<p><br>
</p>
<p> [root@t7 ~]# xl info <br>
host : t7 <br>
release : 4.9.215-36.el7.x86_64 <br>
version : #1 SMP Mon Mar 2 11:42:52 UTC 2020<br>
machine : x86_64 <br>
nr_cpus : 8 <br>
max_cpu_id : 7 <br>
nr_nodes : 1<br>
cores_per_socket : 4<br>
threads_per_core : 1<br>
cpu_mhz : 3000.122<br>
hw_caps :
bfebfbff:000ce3bd:20100800:00000001:00000000:00000000:00000000:00000000<br>
virt_caps : pv hvm<br>
total_memory : 57343<br>
free_memory : 53620<br>
sharing_freed_memory : 0<br>
sharing_used_memory : 0<br>
outstanding_claims : 0<br>
free_cpus : 0<br>
xen_major : 4<br>
xen_minor : 12<br>
xen_extra : .2.39.g3536f8dc<br>
xen_version : 4.12.2.39.g3536f8dc<br>
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p
hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64<br>
xen_scheduler : credit2<br>
xen_pagesize : 4096<br>
platform_params : virt_start=0xffff800000000000<br>
xen_changeset :<br>
xen_commandline : placeholder dom0_mem=1024M,max:1024M
cpuinfo com1=115200,8n1 console=com1,tty loglvl=all
guest_loglvl=all ucode=-1<br>
cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat
4.8.5-39)<br>
cc_compile_by : mockbuild<br>
cc_compile_domain : centos.org<br>
cc_compile_date : Tue Apr 14 14:22:04 UTC 2020<br>
build_id : 24148a191438467f26a9e16089205544a428f661<br>
xend_config_format : 4</p>
<p><br>
</p>
<p>[root@t5 ~]# xl info<br>
host : t5<br>
release : 4.9.215-36.el6.x86_64<br>
version : #1 SMP Mon Mar 2 10:30:40 UTC 2020<br>
machine : x86_64 <br>
nr_cpus : 8 <br>
max_cpu_id : 7<br>
nr_nodes : 1<br>
cores_per_socket : 4<br>
threads_per_core : 1<br>
cpu_mhz : 2000<br>
hw_caps :
b7ebfbff:0004e33d:20100800:00000001:00000000:00000000:00000000:00000000<br>
virt_caps : hvm<br>
total_memory : 12287<br>
free_memory : 6955<br>
sharing_freed_memory : 0<br>
sharing_used_memory : 0<br>
outstanding_claims : 0<br>
free_cpus : 0<br>
xen_major : 4<br>
xen_minor : 8<br>
xen_extra : .5.86.g8db85532<br>
xen_version : 4.8.5.86.g8db85532<br>
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p
hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64<br>
xen_scheduler : credit<br>
xen_pagesize : 4096<br>
platform_params : virt_start=0xffff800000000000<br>
xen_changeset :<br>
xen_commandline : dom0_mem=1024M,max:1024M cpuinfo
com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all<br>
cc_compiler : gcc (GCC) 4.4.7 20120313 (Red Hat
4.4.7-23)<br>
cc_compile_by : mockbuild<br>
cc_compile_domain : centos.org<br>
cc_compile_date : Thu Dec 12 14:34:48 UTC 2019<br>
build_id : da34ae5b90c82137dcbc466cd66322381bc6fd21<br>
xend_config_format : 4<br>
</p>
<p><i>Note:</i> with all other kernels and xen versions that were
published for C6 during the last year, the performance was the
same, i.e. slow<br>
</p>
<p><br>
</p>
<p>The test VM is exactly the same, copied among servers:</p>
<p>[root@t7 ~]# cat /etc/xen/test7_1<br>
builder = "hvm"<br>
xen_platform_pci=1<br>
name = "Test7"<br>
memory = 2048<br>
maxmem = 4096<br>
vcpus = 2<br>
vif = [ "mac=00:14:5e:d9:df:50,bridge=xenbr0,model=e1000" ]<br>
disk = [ <a class="moz-txt-link-rfc2396E" href="file:/var/lib/xen/images/test7_1,xvda,w">"file:/var/lib/xen/images/test7_1,xvda,w"</a> ]<br>
sdl = 0<br>
vnc = 1<br>
bootloader = 'xenpvnetboot'<br>
#bootloader_args = ['--location',
'<a class="moz-txt-link-freetext" href="http://internal.x.y/mrepo/centos7-x86_64/disc1/">http://internal.x.y/mrepo/centos7-x86_64/disc1/</a>']<br>
on_poweroff = 'destroy'<br>
on_reboot = 'restart'<br>
on_crash = 'restart'<br>
#boot="nd"<br>
boot="d"<br>
pae=1<br>
acpi=1<br>
apic=1<br>
tsc_mode=0<br>
</p>
<p><i>Notes:</i></p>
<p>- the lines past "boot" in the config do not make any difference
either, they were added during the last week's tests. <br>
</p>
<p>- I've tested with 1, 2, 4 and 8 VCPUs . There is no diff for the
real life apps.<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p> Frankly in this moment I have no idea what else to change or
test so .. please help.</p>
<p><br>
</p>
</body>
</html>