This morning I get a call that a server is down. The server in
question is a vmware guest, windows 2003 advanced. The host is vmware
server 1.01, running on centos 4.4 x64 on a poweredge 2950. The
server has 16g of ram and a quadcore cpu, storage is provided by a
perc 5/i, raid 1 across two 146gb sas drives.
I was able to ssh into the host. After trying to ping the guest, and
trying to connect to vmware via the management console, I decided to
restart the vmware service. so I type service vmware restart. it
hung on "shutting down virtual machines". I was able to ctrl-c out,
and decided to manually kill the vmware processes. after killing all
the vmware stuff, I did a service vmware start. I get an error
"cannot touch /etc/vmware/locations: read only file system"
/etc is part of /, which mount claimed was mounted RW
so I try cat /var/log/messages and get nothing
so I tell the machine to reboot (remotely). of course, it doesn't
come back up on its own, so I drive to the location. the machine is
running, but sitting at a black screen. I don't know what state it
was in, so did a forced turn off. turning it back on, it proceeded to
boot normally. it had a slight pause while it ran fsck on / but other
than that, no errors.
the vm's restarted normally, /var/log/messages is back, but has no
entries between June 15 and when I rebooted it the 2nd time on June
18.
any ideas on where I should start looking?
is there some way to read array status from a Perc controller under linux?
any suggestions will be appreciated!
thanks
Gordon