[CentOS-virt] CentOS7, Xen 4.6.1, kernel 3.18.25-19 strange performance problem

[T.Weyergraf]

Hi all,

i just stumbled over a strange performance issue with my Xen setup.

I use centos-virt Xen since a long time on my workstation and usually 
never check performance. However, yesterday I booted into the 3.18.25-19 
Dom0 kernel *without* Xen and found the system noticably more 
responsive. That triggered me into running a simple kernel compile 
benchmark and compare the results:

Testsequence:
echo 3 > /proc/sys/vm/drop_caches
make distclean
cp ../.config . #copies an existing valid config for compile
make oldconfig
time make -j <threads> bzImage

Results:

xen 4.6.1 - kernel 3.18.25-19
16 threads:
real    1m41.983s
user    16m37.247s
sys 5m7.121s

32 threads:
real    1m29.240s
user    23m7.170s
sys 6m31.497s

kernel 3.18.25-19
16 threads:
real    1m12.909s
user    14m37.627s
sys 1m36.521s

32 threads:
real    1m0.530s
user    18m45.569s
sys 1m58.807s

Some notes:
- System is a HP z840 with 2*8core E5 v3 Xeon and 128gig Ram. Disk is an 
Intel 750 ssd (PCIe nvme)
- While normally in Xen, I limit Dom0 Memory via dom0_mem, but for the 
tests, Dom0 was allowed to use full memory
- While running the test, no guest was running except Dom0. System was 
booted to graphical desktop and was completely unloaded otherwise
- (boot-)logs showed no issues with ACPI or hardware-support or whatsoever.

I have not yet run any in-depth investigation or tried too many 
different settings, but I find the differences quite noteworthy. So, if 
anybody came across the same issues, has some hints or suggestions, feel 
free. I'll continue poking around and if there is interest, i'll post 
updates.

Regards,
Thomas