[CentOS] File system goes read-only once in a while

Fri Aug 1 19:21:00 UTC 2008
Mufit Eribol <hme at onart.com.tr>

I think I found the culprit albeit I still don't know how to fix.

1. During boot the screen prints the following errors
"no fstab.sys, mounting internal defaults
No devices found
Setting up Logical Volume Management: /var/lock: mkdir failed: No such 
file or directory"

I have a LV on RAID mounted as /mnt/raid. Then /mnt/raid/var is 
symlinked to /var. I found on the internet that some linux systems look 
for /var/lock or /var/run on / partition only. Obviously LVM can not 
create its file in /var/lock, perhaps /mnt/raid is not mounted yet 
during /var/lock mkdir operation.

2. Second important finding is that /forcefsck forces only software raid 
not the hardware one. It does the check for md0 (/temp), md1 (/boot), 
and md2 (/). It skips /dev/raid_vg0/raid_lv0 (/mnt/raid) altogether. I 
don't know how to force to check it durin reboot.

3. I changed to init level 1. Then tried to umount /mnt/raid. But all I 
received was "device is busy" prompt. "umount -l /mnt/raid" was able 
unmount /mnt/raid. Then tried to run "fsck /mnt/raid". This time I received
"fsck.ext2: Is a directory while trying to open /mnt/raid
The superblock could not be read or does not describe a correct ext2 
filesystem. If the device is valid and it really contains an ext2 
filesystem (and not swap or ufs or something else), then the superblock 
is corrupt, and you might try running e2fsck with on alternate superblock:

e2fsck -b 8193 <device>"

I tried with one of the superblocks on /mnt/raid. This time I get
"fsck.ext3: Device or resource busy while trying to open 
Filesystem mounted or opened exclusively by another program?"

Sorry for the long post.

This is the point I arrived. I am stumped.

Thank your for all the support.