[CentOS] kvm: vm root fs becomes ro

Mon Dec 2 23:29:57 UTC 2013
Paul Heinlein <heinlein at madboa.com>

I've a the following happen a couple times now, and my internet 
searches are failing to locate an answer to the problem.

We've got a few servers that primarily house VMs using KVM. They've 
got E-3 cpus and 32 GB RAM, and they run stock CentOS 6.4, fully 
patched (not yet migrated to 6.5). The VM disk images are housed on an 
NFS server. None of the VMs is particularly resource-hungry. They run 
a variety of Linux distros: CentOS 5/6, Debian 6/7.

I'll start to see the VMs fail to write files to their local 
filesytems. No machine in the chain has rebooted or been updated in 
any significant way, but the root filesystem is off-limits. (This will 
happen on just one of our servers; the other VM platforms run without 

In /var/log/messages, I'll see the following entry for each impacted 

<date> <host> kernel: kvm: <pid>: cpu0 disabled perfctr wrmsr: 0xc1 
data 0xabcd

In /var/log/libvirt/qemu/<vm-name>.log, I'll see

block I/O error in device 'drive-virtio-disk0': Stale file handle 

Oddly, the underlying host might be running, say, five VMs, but only 
four of them will get the log messages, and show the read-only 
symptoms, while the fifth just keeps chugging along.

Googling suggests that the "disabled perfctr wrmsr" message is 
harmless, but my experience suggests otherwise.

Any hints, workarounds, or relevent information is very welcome.


Paul Heinlein
heinlein at madboa.com
45°38' N, 122°6' W