[CentOS] File system goes read-only once in a while

Mon Aug 4 17:24:18 UTC 2008
Mufit Eribol <hme at onart.com.tr>

Toby Bluhm wrote:
> Mufit Eribol wrote:
>> Toby Bluhm wrote:
>>> Mufit Eribol wrote:
>>>> I have a LV on RAID mounted as /mnt/raid. Then /mnt/raid/var is 
>>>> symlinked to /var. 
>>> I was afraid you were going to say that.
>>> Go back to single user mode.
>>> mkdir /new_var
>>> cd /mnt/raid/var
>>> tar cf - . | ( cd /new_var ; tar xvf - )
>>> Make sure both dirs look the same.
>>> Change the link to /new_var. Or remove the old link & mv /new_var /var.
>>> reboot.
>> Toby, Thank you for this nice tip. It worked perfectly. The server is 
>> back in the game again.
>> Just for my learning experience, I would appreciate if you clarify 
>> one point though. Why are you afraid when you hear /mnt/raid/var 
>> symlinked to /var?
> Because it can complicate a recovery, as you just experienced.
> Why did you feel a need to have /var setup as you did? Did you expect 
> to fill it up quickly or a need for speed? You also have /tmp separate 
> - do you expect more than usual activity there?
> Perhaps a better question would be - What is the purpose of this 
> machine? If it's a just a fileserver on a home lan, you don't *need* 
> to make it complicated, although learning is fun :-).
> Running a very active internet facing box with email, mysql, apache, 
> etc. would probably call for a more complicated setup - which would 
> actually make recovery & security easier/better.
This box is loaded with cyrus-imapd, postfix, amavisd, clamd, 
spamassassin, mysql, postgresql, apache, CRM, DMS, named, hylafax etc 
for a small company. I wanted to keep operating system on 2 SATA disks 
(RAID1), data (var and home) on a high capacity RAID10 (4 SATA disks). 
It works also a file server. I just wanted more capacity for home and 
var directories, hence they are on separate RAID controller. It is more 
difficult if the OS is also on RAID controller as the driver should be 
loaded before the OS is up and running. When I install a new kernel, I 
compile the raid driver easily with my setup. So, having OS on soft RAID 
and data files (home and var) on RAID controller seemed better idea when 
I setup the system.
>> Here is my fstab:
>> /dev/md2                /                       ext3    
>> defaults        1 1   <--- md2 Software RAID1
>> /dev/md1                /boot                   ext3    
>> defaults        1 2   <--- md0 Software RAID1
>> /dev/md0                /tmp                    ext3    
>> defaults        1 2   <--- md1 Software RAID1
>> tmpfs                   /dev/shm                tmpfs   
>> defaults        0 0
>> devpts                  /dev/pts                devpts  
>> gid=5,mode=620  0 0
>> sysfs                   /sys                    sysfs   
>> defaults        0 0
>> proc                    /proc                   proc    
>> defaults        0 0
>> LABEL=SWAP-sda3         swap                    swap    
>> defaults,pri=1  0 0
>> LABEL=SWAP-sdb3         swap                    swap    
>> defaults,pri=1  0 0
>> /dev/raid_vg0/raid_lv0  /mnt/raid               ext3    
>> defaults        0 0   <--- Hardware RAID10
>> Before, home and var were under /mnt/raid directory and symlinked to 
>> /home and /var. Now, both directories were copied to / (md2 software 
>> RAID1) as new_home and new_var and /home and /var symlinks are now 
>> pointing to these new directories. /mnt/raid (hardware RAID10) which 
>> is the main storage of my server is not being used at the moment.
> Instead of using links, may as well just mount it where it belongs.
I will follow your advice. I will mount /var and /home on RAID 
controller separately (2 separate VGs). But, some distros, one of them 
is Ubuntu, wants to have /var/run and /var/lock on the same partition as 
/. I don't know if CentOS 5.2 has such a requirement. If it has, I will 
mkdir /var/run and /var/lock on the same partition as / bu umounting 
/var first.
>> I am planning to have 2 logical volumes (for home and var separately) 
>> instead of 1. Then, they will be mounted as separate partitions as 
>> /home and /var to /dev/raid_vg0/raid_lv0 and /dev/raid_vg0/raid_lv1, 
>> respectively. Is it a good approach? Please advise.
> I'm somewhat simple-minded - I like to keep the system that way :-). I 
> split the partitions into 3
>   /
>   swap
>   /home
> either on a single disk or mirrored ( swap mirrored too ) - no lvm. 
> For data storage I use lvm on raid on a separate mount point. Not 
> saying you should do the same - it's just what I do.
Yes, it is simple. Perhaps I am victim of the articles advocating more 
partitions on the internet :-)

Thank you.