On 6/12/19 3:09 AM, Jonathan Archdale - Blacknight wrote:
Hi folks,
Firstly; apologies in advance for what is a head wrecker of keeping on top of the speculative mitigations and also if this is a duplicate email; my first copy didn't seem to make it into the archive. Also a disclaimer that I may have misunderstood elements of the below but please bear with me.
I write this hoping to find out a bit more about the state of the relevant kernel patches for the various speculative attack mitigations in the current 4.9.177-35 kernel used in the centos-virt tree.
4.9.177 should contain all mitigations up to and including md_clear for MDS, but when running on Xen, you need cooperation from Xen for many of these fixes as well.
For reference; I am running a Dell R620, using an E5-2620 v2 CPU, using 4.9.177-35 from the xen-410 tree in the EL6 branch ; though I am still using xen 4.6 at this time. I have been told in the irc channel that the kernel is not built against a specific version of xen so this should make no odds.
This is correct, the same kernel build gets put into all the xen trees from a "common" repo.
As per https://www.dell.com/support/home/ie/en/iebsdt1/drivers/driversdetails?drive... bios version 2.70 (installed) contains mitigations for CVE-2018-3639 (variant 3a route system register read) and CVE-2018-3640 (variant 4, speculative store bypass). I am also running the latest available intel microcode for the CPU; as imported via 'module /GenuineIntel.bin' and 'ucode=2' in my grub.conf to import (as taken from https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files).
# uname -r 4.9.177-35.el6.x86_64
# cat /proc/cmdline ro root=UUID=0e51d8dc-6409-4715-83f4-f86fa24083e2 rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD nodmraid SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet intel_idle.max_cstate=0 processor.max_cstate=0 l1ft=full mds=full,nosmt ssbd=force-on
# grep -e flags -e "model name" -e microcode -e bugs /proc/cpuinfo | sort -u model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz microcode : 0x42e flags: fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc pni pclmulqdq monitor est ssse3 cx16 sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase erms xsaveopt dtherm ida arat pln pts bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds
You will note that there is a lack of flags for ssbd, spec_ctrl, intel_stibp, flush_l1d, ibrs, etc.
Your microcode is correct, per:
https://www.intel.com/content/dam/www/public/us/en/documents/corporate-infor...
The kernel however is not picking up any of the expected microcode features one would expect in the flags list.
I have a E5-2609 v2 that uses the same microcode, but mine shows the correct flags in the list. The key difference between our environments is that you are running your kernel on top of Xen 4.6 which is EOL since Oct 2018 upstream. The CentOS builds were never updated past 4.6.6-12, which was released in May 2018, so it doesn't include the fixes for L1TF or SSBD either.
You didn't include any details of the Xen packages you're running, but make sure you are at least on Xen 4.6.6-12. 'xl info' may be useful here.
That said, I can't explain why you don't see the ibrs or ibpb flags in your list.
As a possible useful exercise, take a look at 'xl dmesg' for messages containing "microcode:" as well as a section labeled "Speculative mitigation facilities:" to verify that Xen is recognizing and supporting the necessary CPU features.
# for i in /sys/devices/system/cpu/vulnerabilities/* ; do echo -n "$i : "; cat $i ; done /sys/devices/system/cpu/vulnerabilities/l1tf : Mitigation: PTE Inversion /sys/devices/system/cpu/vulnerabilities/mds : Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown /sys/devices/system/cpu/vulnerabilities/meltdown : Vulnerable /sys/devices/system/cpu/vulnerabilities/spec_store_bypass : Vulnerable /sys/devices/system/cpu/vulnerabilities/spectre_v1 : Mitigation: __user pointer sanitization /sys/devices/system/cpu/vulnerabilities/spectre_v2 : Mitigation: Full generic retpoline, STIBP: disabled, RSB filling
Despite having relevant microcode, I am still seeing the box as showing vulnerable to meltdown, speculative store bypass (ssbd), as well as MDS via MD_CLEAR mitigation.
Running inside of Xen dom0 is not quite the same as running natively on your hardware. Your dom0 is a type of guest, so the mitigations described in the /sys/ location only reflect the state of the dom0 itself, not Xen itself or other guests.
The meltdown section will always show as vulnerable because the Linux kernel does not recognize the Xen-based mitigation, even though it is effective for dom0. Also, PV domains running inside a Xen release patched for meltdown are protected in spite of the indicator by the guest kernel.
When I mount the debugfs, the flag for ibrs is missing - preventing me enabling it as a mitigation for ssbd (due to lack of relevant cpu flag):
# mount -t debugfs none /sys/kernel/debug/
# ls -lahtr /sys/kernel/debug/x86/ibrs_enabled ls: cannot access /sys/kernel/debug/x86/ibrs_enabled: No such file or directory
I have other R620's with the same CPUs running stock el6 kernels that are showing as fully patched to these issues. Could I please get some feedback from whoever builds out the kernels; if the mitigations/patches are in place in the 4.9.177-35 kernel for the various speculative mitigations?
Specifically; CVE-2018-3639, CVE-2018-3640, CVE-2018-3646, CVE-2018-12126, CVE-2018-12130, CVE-2018-12127 and CVE-2019-11091.
Just to reiterate, these fixes are in 4.9.177 but the mitigations generally require cooperation from Xen, and Xen 4.6 builds from CentOS do not contain fixes for disclosed issues from May 2018 onward.
For proper mitigation, you need to upgrade to Xen 4.8 or newer, but I would suggest you consider going to 4.10 if possible (or 4.12 if you upgrade to CentOS 7) to give you more headroom before EOL as well as a number of performance improvements, both related to the mitigations and otherwise.