[CentOS] Troubles expanding file system. - Solved

Thu Sep 9 15:46:22 UTC 2021
Jeff Boyce <jboyce at meridianenv.com>

To follow-up and provide a conclusion to my issue, in case anyone else 
runs into a similar situation.

TLDR - go to bottom and read item #7.

To recap the issue:
I have a Dell PowerEdge server with a CentOS KVM host (Earth) with
one CentOS guest (Sequoia) that I am trying to expand the partition and
filesystem on.  I have LVM logical volumes on the host system (Earth),
which are used as devices/partitions on the guest system (Sequoia).  In
this particular situation I have successfully extended the logical
volume (lv_SeqEco) on Earth from 500GB to 700GB.

1.  Checking the disk information (lsblk) on Earth shows that the
logical volume (lv_SeqEco) is now listed as 700GB.

2.  Checking disk information (lsblk) on Sequoia shows that the disk
/dev/vde is still listed as 500GB, and partition /dev/vde1 where the
mount point /ecosystem is located is also listed as 500GB.

3.  I had tried using the resize2fs command to expand the filesystem on
/dev/vde1, but it returned with the result that there was nothing to
do.  Which makes sense now after I checked the disk information, since
/dev/vde on Sequoia has not increased from 500GB to 700GB.

4.  On previous occasions when I have done this task, I would just start
GParted on Sequoia and use the GUI to expand the partition and
filesystem.  I am unable to do this now, as the VGA adapter on my server 
has died and I have no graphical output to the attached monitor.

5.  My goal was to not have to not have to reboot the system. Especially 
since it is a 10-year old server and there is no longer VGA output to 
view the boot process.

What I tried, and what worked to solve my issue:

1.  First, I tried rescannning the device on the guest:
echo 1 > 
/sys/devices/pci0000:00/0000:00:01.1/host1/target1:0:0/1:0:0:0/rescan

This resulted in no output, and checking lsblk on the guest (Sequoia) 
showed no change:
vde  500GB  disk
vde1   500GB   part /ecosystem

2.  Second, I tried using "virsh blockresize" from the host (Earth) system.
Get the block size information
earth# virsh domblkinfo SequoiaVM vde
output:  vde capacity 751619276800

Resize the block (express the size in Bytes)
earth# virsh blockresize SequioaVM vde 751619276800B
output:  Block device vde is resized.

Check the block device on the guest to confirm.
sequoia# lsblk
vde   700GB   disk
vde1   500GB   part /ecosystem

Therefore, virsh blockresize was successful.

3.  I now have the larger disk recognized on the guest system, but still 
need to expand the partition and the filesystem.  Tried the following 
options.

resize2fs -p /dev/vde1
    output:  filesystem is already xx blocks long, nothing to do.

growpart /dev/vde 1
    output:  This showed output indicating the partition had changed
       changed:  partition=1 start=63
       old:  size=1048575937
                end=1048576000
       new:  size=1468003572
                  end=1468003635

However, checking lsblk on the guest still showed the same result
    vde   700GB   disk
    vde1   500GB   part /ecosystem

On the off chance that growpart was successful, I ran resize2fs again 
and it produced the same result as above.

partprobe
    output:  failed to re-read the partition table on all attached 
devices (including /dev/vde1); device or resource busy.

Tried using "parted" and its resize command.  First, using the print 
command to get the parameters of /dev/vde (output: end=751617861119B, 
size=751617828864B).  Then changing parted to select (use) /dev/vde1 and 
used print command to view parameters of /dev/vde1 (output:  
end=536870879743B, size=536870879744B).

(parted)# resize 1 0 751617861119
output:  The location 751617861119 is outside of the device /dev/vde1

Looked at using fdisk; however, the documentation stated that the only 
way to change the partition size using fdisk is by deleting it and 
recreating it.  I didn't want to do this.

I finally decided that everything indicated that I would not be able to 
complete this while the system was online and/or mounted, despite all my 
research information showing that.

4.  Unmounted the filesystem of the guest and run partprobe.

sequoia# service smb stop
sequoia# umount /ecosystem
sequoia# partprobe
    output:  This produced the same warnings as before (failed to 
re-read the partition, device busy) for all partitions that were still 
mounted, but did not produce the warning for partition /dev/vde1 that 
had been unmounted.

sequoia# lsblk
   vde   700GB   disk
    vde1   700GB   part /ecosystem

Success.  The new partition is now recognized in the kernel, and I 
should now be able to resize the filesystem.

5.  Run resize2fs

sequoia# resize2fs -p /dev/vde1
    output:  please run e2fsck -f /dev/vde1 first

sequoia# e2fsck -f -C 0 /dev/vde1  (-C 0 will display the progress)
    output:  system passed all checks.

sequoia# resize2fs -p /dev/vde1
    output:  resizing the filesystem on /dev/vde1 to 183500446 (4k) 
blocks, begin Pass 1, extending inode table, filesystem is now 183500446 
blocks long.

6.  Remount the filesystem and check the filesystem size.

sequoia# mount /dev/vde1 /ecosystem
sequoia# service smb start
sequoia# df -h
    filesystem=/dev/vde1
    size=689G
    used=459G
    available=196G
    used%=71%
    mount=/ecosystem

7.  So in conclusion I needed to do the following steps to resize and 
expand my disk without a reboot.  There may still be other ways that 
this can be accomplished, but this worked for my CentOS 6 system (both 
host and guest).

   a.  On the host system, lvextend to expand the logical volume used by 
the guest as it's disk.
   b.  On the host system, virsh blockresize to get the guest system to 
recognize the expanded disk space.
   c.  On the guest system, growpart to grow the partition to the 
expanded space.
   d.  On the guest system, unmount the filesystem (stop services as 
required).
   e.  On the guest system, partprobe to get the kernel to recognize the 
expanded partition.
   f.  On the guest system, resize2fs (run filesystem check if required) 
to expand the filesystem.

Thanks to all that provided suggestions and insight, it was very helpful 
in my research and methodical stepping through this process.

Jeff