Today I suddenly have two VMs that have read only file systems. The host is CentOS 6, as are the two VMs with this problem.
The first symptom was on a new VM I installed ISPConfig onto. I got through the entire process with only a dependency issue between php-pecl_apc and php-accelerator.
After completing the installation I noticed some funny things, but I assumed it might be the addition of quotas and remounting with quotas on. so I didn't think much of it and rebooted the VM.
It failed to reboot because the file system should not be switched to read-write. Since it was a new VM and installing ISPConfig was an experiment, I just wiped it with the intention of starting over.
While I was creating another clone of a CentOS 6 image on the host, I looked into one of the other VMs running on that host, which has been up and running for 47 days. Same problem, without rebooting. For example running yum give this:
[root@dev log]# yum update Loaded plugins: fastestmirror, presto Cannot open logfile /var/log/yum.log Could not create lock at /var/run/yum.pid: [Errno 30] Read-only file system: '/var/run/yum.pid' Another app is currently holding the yum lock; waiting for it to exit... Traceback (most recent call last): File "/usr/bin/yum", line 29, in <module> yummain.user_main(sys.argv[1:], exit_code=True) File "/usr/share/yum-cli/yummain.py", line 254, in user_main errcode = main(args) File "/usr/share/yum-cli/yummain.py", line 103, in main show_lock_owner(e.pid, logger) File "/usr/share/yum-cli/utils.py", line 106, in show_lock_owner ps = get_process_info(pid) File "/usr/share/yum-cli/utils.py", line 61, in get_process_info if (not os.path.exists("/proc/%d/status" % pid) or TypeError: %d format: a number is required, not str
------------------------------------ And running mount gives this:
[root@dev log]# mount /dev/mapper/VolGroup-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) /web on /NFS/web type none (rw,bind) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) /etc/named on /var/named/chroot/etc/named type none (rw,bind) /etc/named.rfc1912.zones on /var/named/chroot/etc/named.rfc1912.zones type none (rw,bind) /usr/lib64/bind on /var/named/chroot/usr/lib64/bind type none (rw,bind) /etc/named.iscdlv.key on /var/named/chroot/etc/named.iscdlv.key type none (rw,bind) /etc/named on /var/named/chroot/etc/named type none (rw,bind) /usr/lib64/bind on /var/named/chroot/usr/lib64/bind type none (rw,bind)
mount: warning: /etc/mtab is not writable (e.g. read-only filesystem). It's possible that information reported by mount(8) is not up to date. For actual information about system mount points check the /proc/mounts file.
------------------------------- The VM is running, serving web pages and responding to DNS queries, but it is clear, given my earlier experience with the ISPConfig machine, that I won't be able to reboot it until I figure out the problem.
Now that I am looking at the output from the mount command I wonder where all those named related mounts came from. Could it be webmin. Both VMs have webmin installed. Mostly to allow be to configure bind, since system-config-bind is no more.
Anybody have any idea what happened, or better yet, any ideas on how to fix this?
Emmett