Hi all,
Is it possible, safely, to resize /var while a server is running? I urgently need to resize a full 2GB /var on a running server which is located at a remote location where no one can get to it for 2 days?
The system runs on lvm.
I already moved /var/logs to it's own LVM partition but it didn't help at all so I think it's better to rather just resize it. Google searches suggest taking the server offline, or boot into single mode. But this won't be possible for another 2 or so days and I urgently need to install more software on the server.
tia :)
Rudi Ahlers wrote:
Hi all,
Is it possible, safely, to resize /var while a server is running? I urgently need to resize a full 2GB /var on a running server which is located at a remote location where no one can get to it for 2 days?
The system runs on lvm.
Growing on line EXT3 file systems on LVM has always worked ok for me. shrinking needs to be done off line but growing on line File systems has not given me any trouble to date.
I expect that the likelihood of problems with this is proportional to the distance from your current location to the server though ;-)
regards Brendan
On Sun, May 2, 2010 at 10:32 PM, Brendan Minish bminish@minish.org wrote:
Rudi Ahlers wrote:
Hi all,
Is it possible, safely, to resize /var while a server is running? I urgently need to resize a full 2GB /var on a running server which is located at a remote location where no one can get to it for 2 days?
The system runs on lvm.
Growing on line EXT3 file systems on LVM has always worked ok for me. shrinking needs to be done off line but growing on line File systems has not given me any trouble to date.
I expect that the likelihood of problems with this is proportional to the distance from your current location to the server though ;-)
regards Brendan _______________________________________________
Hi Brendan,
I've also safely grown LVM volumes online, but /var is a bit more trickey since a lot of stuff runs straight from it .So I don't want to attempt this if it's going to break the whole system.
I've also safely grown LVM volumes online, but /var is a bit more trickey since a lot of stuff runs straight from it .So I don't want to attempt this if it's going to break the whole system.
OK.. get you now..
The main thing to keep in mind is that you'll need a little bit of free space in order to grow /var. I grow /var often on many systems. There's also a possibility that you'll need to run tune2fs to grow the journal to allow a larger maximum filesystem size. This cannot be done with /var online though.
On Sun, May 2, 2010 at 10:44 PM, Kwan Lowe kwan.lowe@gmail.com wrote:
I've also safely grown LVM volumes online, but /var is a bit more trickey since a lot of stuff runs straight from it .So I don't want to attempt
this
if it's going to break the whole system.
OK.. get you now..
The main thing to keep in mind is that you'll need a little bit of free space in order to grow /var. I grow /var often on many systems. There's also a possibility that you'll need to run tune2fs to grow the journal to allow a larger maximum filesystem size. This cannot be done with /var online though. _______________________________________________
heh, there's plenty space left on the drive:
[root@zaxen02 ~]# pvscan PV /dev/md1 VG LVM01 lvm2 [232.69 GB / 141.69 GB free] Total: 1 [232.69 GB] / in use: 1 [232.69 GB] / in no VG: 0 [0 ]
As far as I know, one needs to unmount a volume in order to resize it, but /var can't be unmounted.
I wonder if I could script something that could run in single user mode when it reboots, but 1. how do I tell it to reboot in ginle user mode, via SSH? 2. how do I tell the server to login by itself, i.e. don't need someone to type in the root password @ the console, then check to make sure it's in single user mode?
From here could be as simple as unmouting /var, running lvresize +2GB
/dev/LVM/var etc
Has anyone done anything like this before, with success? i.e. do you have a working procedure / script to share with me? I won't be able to test the process, if it fails then I'm stuck
-- Kind Regards Rudi Ahlers SoftDux
Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532
On Sun, May 2, 2010 at 5:01 PM, Rudi Ahlers rudiahlers@gmail.com wrote:
heh, there's plenty space left on the drive:
[root@zaxen02 ~]# pvscan PV /dev/md1 VG LVM01 lvm2 [232.69 GB / 141.69 GB free] Total: 1 [232.69 GB] / in use: 1 [232.69 GB] / in no VG: 0 [0 ]
As far as I know, one needs to unmount a volume in order to resize it, but /var can't be unmounted.
With LVM you don't need to unmount /var. You can do it on a running system with the following caveats:
1) You'll need some free space on /var. I.e., not extra space in the volume group, but a little extra space in the /var partition itself. In other words, don't try to resize a completely full /var partition.
2) There is a possibility that you'll need to resize some metadata. If so, this has to be done offline via a boot disk for /var.
On Sun, May 2, 2010 at 11:13 PM, Kwan Lowe kwan.lowe@gmail.com wrote:
On Sun, May 2, 2010 at 5:01 PM, Rudi Ahlers rudiahlers@gmail.com wrote:
heh, there's plenty space left on the drive:
[root@zaxen02 ~]# pvscan PV /dev/md1 VG LVM01 lvm2 [232.69 GB / 141.69 GB free] Total: 1 [232.69 GB] / in use: 1 [232.69 GB] / in no VG: 0 [0 ]
As far as I know, one needs to unmount a volume in order to resize it,
but
/var can't be unmounted.
With LVM you don't need to unmount /var. You can do it on a running system with the following caveats:
- You'll need some free space on /var. I.e., not extra space in the
volume group, but a little extra space in the /var partition itself. In other words, don't try to resize a completely full /var partition.
- There is a possibility that you'll need to resize some metadata. If
so, this has to be done offline via a boot disk for /var. _______________________________________________
That's exactly as I thought :)
For now, a safer way around this, I moved the /var/lib/xend/saved folder to it's own LVM volume, and re-mounted it on /var/lib/xend.
So now /var has 1.4GB free space from 2GB
-- Kind Regards Rudi Ahlers SoftDux
Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532
That's exactly as I thought :) For now, a safer way around this, I moved the /var/lib/xend/saved folder to it's own LVM volume, and re-mounted it on /var/lib/xend. So now /var has 1.4GB free space from 2GB
2G for /var still looks a bit on the small side to me...
/var can be umounted if necessary. Just stop all daemons that write to it and anything else that reads or otherwise has a file handle under /var. Been there, done that.
On Sun, 2010-05-02 at 17:13 -0400, Kwan Lowe wrote:
On Sun, May 2, 2010 at 5:01 PM, Rudi Ahlers rudiahlers@gmail.com wrote:
heh, there's plenty space left on the drive:
[root@zaxen02 ~]# pvscan PV /dev/md1 VG LVM01 lvm2 [232.69 GB / 141.69 GB free] Total: 1 [232.69 GB] / in use: 1 [232.69 GB] / in no VG: 0 [0 ]
As far as I know, one needs to unmount a volume in order to resize it, but /var can't be unmounted.
With LVM you don't need to unmount /var. You can do it on a running system with the following caveats:
- You'll need some free space on /var. I.e., not extra space in the
volume group, but a little extra space in the /var partition itself. In other words, don't try to resize a completely full /var partition.
- There is a possibility that you'll need to resize some metadata. If
so, this has to be done offline via a boot disk for /var.
--- Sounds good but I really do wonder if he has a hardware raid controller that does not support resizing of Linux LVM on top of the HW Raid Volume Set?? No one asked that part.
John
On Sun, 2010-05-02 at 19:12 -0400, JohnS wrote:
Sounds good but I really do wonder if he has a hardware raid controller that does not support resizing of Linux LVM on top of the HW Raid Volume Set?? No one asked that part.
Surely if the Volume group has unallocated space extending the LV should not be a problem since it is not altering things as seen at the raid controller level. Or am I missing something?
regards Brendan
On Sun, May 2, 2010 at 8:23 PM, Brendan Minish bminish@minish.org wrote:
On Sun, 2010-05-02 at 19:12 -0400, JohnS wrote:
Sounds good but I really do wonder if he has a hardware raid controller that does not support resizing of Linux LVM on top of the HW Raid Volume Set?? No one asked that part.
Surely if the Volume group has unallocated space extending the LV should not be a problem since it is not altering things as seen at the raid controller level. Or am I missing something?
Yes, the LVM layer is distinct from the raid level. In some other OSes it is more tightly integrated (bound, coupled, mashed together). It is possible, I suppose, that if the underlying physical volume is backed by an over-commited, data deduped device then resizing an LVM inside a VG could cause issues though :D..
On Sun, 2010-05-02 at 23:01 +0200, Rudi Ahlers wrote:
--snip--
Has anyone done anything like this before, with success? i.e. do you have a working procedure / script to share with me? I won't be able to test the process, if it fails then I'm stuck
yes, on line, with no issues on centos 5.4
just tested it again in a virtual machine, with no issues
root@dhcp-204 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 3.7G 1.3G 2.3G 37% / /dev/mapper/VolGroup00-LogVol02 961M 43M 870M 5% /var /dev/hda1 99M 13M 82M 14% /boot tmpfs 250M 0 250M 0% /dev/shm [root@dhcp-204 ~]# lvextend -L +1G /dev/VolGroup00/LogVol02 Extending logical volume LogVol02 to 1.97 GB Logical volume LogVol02 successfully resized [root@dhcp-204 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 3.7G 1.3G 2.3G 37% / /dev/mapper/VolGroup00-LogVol02 961M 43M 870M 5% /var /dev/hda1 99M 13M 82M 14% /boot tmpfs 250M 0 250M 0% /dev/shm [root@dhcp-204 ~]# resize2fs /dev/VolGroup00/LogVol02 resize2fs 1.39 (29-May-2006) Filesystem at /dev/VolGroup00/LogVol02 is mounted on /var; on-line resizing required Performing an on-line resize of /dev/VolGroup00/LogVol02 to 516096 (4k) blocks. The filesystem on /dev/VolGroup00/LogVol02 is now 516096 blocks long.
[root@dhcp-204 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 3.7G 1.3G 2.3G 37% / /dev/mapper/VolGroup00-LogVol02 2.0G 43M 1.8G 3% /var /dev/hda1 99M 13M 82M 14% /boot tmpfs 250M 0 250M 0% /dev/shm [root@dhcp-204 ~]#
On Sun, May 2, 2010 at 3:15 PM, Rudi Ahlers rudiahlers@gmail.com wrote:
Hi all,
Is it possible, safely, to resize /var while a server is running? I urgently need to resize a full 2GB /var on a running server which is located at a remote location where no one can get to it for 2 days?
The system runs on lvm.
If it's on LVM it's pretty simple:
lvs to show the current lvs.
lvresize -L +500M /dev/VolGroup00/var_lv resize2fs /dev/VolGroup00/var_lv
If you're running earlier versions of lvm you may need to use ex2online instead of resize2fs.
On 05/03/2010 05:15 AM, Rudi Ahlers wrote:
Is it possible, safely, to resize /var while a server is running? I urgently need to resize a full 2GB /var on a running server which is located at a remote location where no one can get to it for 2 days?
I've done it before on a quiet system and it worked fine. Still think its a bit risky if you cant get to it for 2 days.
I already moved /var/logs to it's own LVM partition but it didn't help at all so I think it's better to rather just resize it. Google searches suggest taking the server offline, or boot into single mode. But this won't be possible for another 2 or so days and I urgently need to install more software on the server.
If you "urgently need to install new software", then a small possibility of breaking the system for 2 days is probably a show stopper. I'd continue with your approach (as a temporary solution) of migrating bits of /var to a different LVM. If moving /var/logs didn't help, you need to track down the big directories. Try
du -sh /var/*
then drill down though your big candidates
/var/cache/yum, /var/spool, and /var/tmp are always candidates for clean up.
Hope this helps.
Kal
-
Brendan Minish -
Christopher Chan -
JohnS -
Kahlil Hodgson -
Kwan Lowe -
Rudi Ahlers