KVM on CentOS 6 - Discuss

8 Sep 2011


      Today I suddenly have two VMs that have read only file systems.  The host is CentOS 6, as are the two VMs with this problem.
The first symptom was on a new VM I installed ISPConfig onto.  I got through the entire process with only a dependency issue between php-pecl_apc and php-accelerator.
After completing the installation I noticed some funny things, but I assumed it might be the addition of quotas and remounting with quotas on. so I didn't think much of it and rebooted the VM.
It failed to reboot because the file system should not be switched to read-write.  Since it was a new VM and installing ISPConfig was an experiment, I just wiped it with the intention of starting over.
While I was creating another clone of a CentOS 6 image on the host, I looked into one of the other VMs running on that host, which has been up and running for 47 days.  Same problem, without rebooting.  For example running yum give this:
[root@dev log]# yum update
Loaded plugins: fastestmirror, presto
Cannot open logfile /var/log/yum.log
Could not create lock at /var/run/yum.pid: [Errno 30] Read-only file system: '/var/run/yum.pid'
Another app is currently holding the yum lock; waiting for it to exit...
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 254, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 103, in main
    show_lock_owner(e.pid, logger)
  File "/usr/share/yum-cli/utils.py", line 106, in show_lock_owner
    ps = get_process_info(pid)
  File "/usr/share/yum-cli/utils.py", line 61, in get_process_info
    if (not os.path.exists("/proc/%d/status" % pid) or
TypeError: %d format: a number is required, not str
------------------------------------
And running mount gives this:
[root@dev log]# mount
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/web on /NFS/web type none (rw,bind)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
/etc/named on /var/named/chroot/etc/named type none (rw,bind)
/etc/named.rfc1912.zones on /var/named/chroot/etc/named.rfc1912.zones type none (rw,bind)
/usr/lib64/bind on /var/named/chroot/usr/lib64/bind type none (rw,bind)
/etc/named.iscdlv.key on /var/named/chroot/etc/named.iscdlv.key type none (rw,bind)
/etc/named on /var/named/chroot/etc/named type none (rw,bind)
/usr/lib64/bind on /var/named/chroot/usr/lib64/bind type none (rw,bind)
mount: warning: /etc/mtab is not writable (e.g. read-only filesystem).
       It's possible that information reported by mount(8) is not
       up to date. For actual information about system mount points
       check the /proc/mounts file.
-------------------------------
The VM is running, serving web pages and responding to DNS queries, but it is clear, given my earlier experience with the ISPConfig machine, that I won't be able to reboot it until I figure out the problem.
Now that I am looking at the output from the mount command I wonder where all those named related mounts came from.  Could it be webmin.  Both VMs have webmin installed.  Mostly to allow be to configure bind, since system-config-bind is no more.
Anybody have any idea what happened, or better yet, any ideas on how to fix this?
Emmett