Hello,
Today a virtual server under my administration ran out of disk space, so I had to get extra space to it.. while it was running, because it's an important web server without any kind of HA/Load Balancing (don't do that, kids).
So what happened, is that after another administrator had extended the disk from VMWare, I tried to extend it from the OS side.
Everything went well, listing my steps here:
* echo 1 > /sys/class/scsi_device/2:0:0:0/device/rescan * fdisk /dev/sdb (the usual, delete partition and make a new, bigger one)
Now's when the problems happened. After writing the partition table I got the usual "Busy" error. But I needed to do this without downtime. So, what did I try to get the kernel acknowledge the partition size change:
* hdparm -z /dev/sdb * partx -a -v /dev/sdb * partprobe -s * echo "1" > /sys/block/sdb/device/rescan
And while dmesg reported that it recognized the partition change, LVM did still not see it. So eventually, I was still forced to reboot, after which everything (pvresize, lvresize, resize2fs) worked fine. Environment:
* A VMWare virtual machine * CentOS 6.7 (Final)
Is there any other way I'm missing? Is this caused by the fact that I was trying to resize a partition that is under the root file system? If so, then is it really impossible to do it live?
Thanks,
----- Original Message ----- | Hello, | | Today a virtual server under my administration ran out of disk space, so | I had to get extra space to it.. while it was running, because it's an | important web server without any kind of HA/Load Balancing (don't do | that, kids). | | So what happened, is that after another administrator had extended the | disk from VMWare, I tried to extend it from the OS side. | | Everything went well, listing my steps here: | | * echo 1 > /sys/class/scsi_device/2:0:0:0/device/rescan | * fdisk /dev/sdb (the usual, delete partition and make a new, bigger one) | | Now's when the problems happened. After writing the partition table I | got the usual "Busy" error. But I needed to do this without downtime. | So, what did I try to get the kernel acknowledge the partition size change: | | * hdparm -z /dev/sdb | * partx -a -v /dev/sdb | * partprobe -s | * echo "1" > /sys/block/sdb/device/rescan | | And while dmesg reported that it recognized the partition change, LVM | did still not see it. So eventually, I was still forced to reboot, after | which everything (pvresize, lvresize, resize2fs) worked fine. Environment: | | * A VMWare virtual machine | * CentOS 6.7 (Final) | | Is there any other way I'm missing? Is this caused by the fact that I | was trying to resize a partition that is under the root file system? If | so, then is it really impossible to do it live? | | Thanks, | | -- | Sander Kuusemets | University of Tartu, High Performance Computing, IT Specialist |
If the disk is not the OS disk I never put a partition table on it if it's going to be used with tools like LVM. I simply create a full disk physical volume and provision it using volume groups and logical volumes. In this case I just rescan the bus to see the new size, do a pvscan/presize and then expand the VGs or LVs as necessary.
On 2015-12-21, Sander Kuusemets sander.kuusemets@ut.ee wrote:
And while dmesg reported that it recognized the partition change, LVM did still not see it. So eventually, I was still forced to reboot, after which everything (pvresize, lvresize, resize2fs) worked fine. Environment:
- A VMWare virtual machine
- CentOS 6.7 (Final)
Is there any other way I'm missing? Is this caused by the fact that I was trying to resize a partition that is under the root file system? If so, then is it really impossible to do it live?
As far as I know (and perhaps my knowledge is dated) it is not possible to resize a partition on the boot device without a reboot. I'm not sure exactly what you mean by "partition under the root filesystem", but I'm guessing you mean that you may have been trying to resize /. If that's the case, then you may be able to do this only if /boot is on a different device.
If you were resizing / because it filled, what exactly filled up? If it was someplace like /tmp or /var, you may want to consider migrating those to separate filesystems, probably under LVM, so that you have a better chance of this succeeding. If you can have your VMWare admin attach a second device where you can put these filesysems even better, that's sure to work without requiring a reboot.
--keith
On 12/21/2015 11:15 AM, Sander Kuusemets wrote:
So what happened, is that after another administrator had extended the disk from VMWare, I tried to extend it from the OS side.
...
Now's when the problems happened. After writing the partition table I got the usual "Busy" error. But I needed to do this without downtime. So, what did I try to get the kernel acknowledge the partition size change:
...
- partprobe -s
Well, partprobe is the standard advice for this situation, but I don't think it actively does anything with the -s flag. That could have been the problem, but I'm not certain.
And while dmesg reported that it recognized the partition change, LVM did still not see it.
You didn't include "pvresize" in that list, so I presume you didn't do it. That's the command that will inform LVM of the change.
I did not do pvresize because neither "pvs" or "pvdisplay" reported to have free space.
After reboot they did, after which doing pvresize worked successfully.
Sander Kuusemets University of Tartu, High Performance Computing, IT Specialist Skype: sander.kuusemets1 +372 737 5694
On 24/12/15 09:17, Gordon Messmer wrote:
On 12/21/2015 11:15 AM, Sander Kuusemets wrote:
So what happened, is that after another administrator had extended the disk from VMWare, I tried to extend it from the OS side.
...
Now's when the problems happened. After writing the partition table I got the usual "Busy" error. But I needed to do this without downtime. So, what did I try to get the kernel acknowledge the partition size change:
...
- partprobe -s
Well, partprobe is the standard advice for this situation, but I don't think it actively does anything with the -s flag. That could have been the problem, but I'm not certain.
And while dmesg reported that it recognized the partition change, LVM did still not see it.
You didn't include "pvresize" in that list, so I presume you didn't do it. That's the command that will inform LVM of the change.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
The curious thing is that a couple of weeks ago, I used parted to resize a GPT partition under Fedora 23, then resized the LUKS device, then the PV. I didn't have to reboot. I don't seem to be able to do the same thing under CentOS 7, though that test system didn't use a LUKS encrypted partition.
I don't know why this is possible in some circumstances and not others. Anyone else?
-
Gordon Messmer -
James A. Peltier -
Keith Keller -
Sander Kuusemets