Hi all,
I have Centos 8 installed on a physical machine (www6) with separate LVM volumes for /, /var, /var/lib/mysql etc.
System boot proceeds without a hiccup, in terminal systemctl status says everything is OK and running, journalctl says so as well - systemd mounts everything stated in fstab.
However, on reboot systemd echoes problems with filesystem on /var :
...// unmounting all volumes
Nov 26 23:51:30 www6 systemd[1]: Unmounting /var... Nov 26 23:51:30 www6 umount[2118]: umount: /var: target is busy. Nov 26 23:51:30 www6 systemd[1]: Stopped target Swap. Nov 26 23:51:30 www6 systemd[1]: Deactivating swap /dev/disk/by-label/lv_swap... Nov 26 23:51:30 www6 systemd[1]: var.mount: Mount process exited, code=exited status=32 Nov 26 23:51:30 www6 systemd[1]: Failed unmounting /var.
Then proceeds and reboots the machine. This occurs on every reboot.
Did anybody here encounter similar problems on reboot?
I found this bug
https://github.com/systemd/systemd/issues/867
which was closed without a definitive solution, the developer says the issue is hard to solve and just cosmetic since /var gets unmounted in the end anyway, which is strange because there is a log line about unmounting just about every other mountpoint and in the end of the journal, there are about 7 log lines about unmounting swap volume, but zilch about /var. If the system is getting down without proper unmount, data corruption can happen. I would feel much better if there were no such errors...
My /etc/fstab:
/dev/mapper/vg_www6-lv_root / xfs defaults,noatime 0 1 /dev/mapper/vg_www6-lv_var /var xfs defaults,noatime 0 0 /dev/mapper/vg_www6-lv_swap swap swap defaults 0 0 /dev/mapper/vg_www6-lv_tmp /tmp xfs defaults,noatime 0 0 /dev/mapper/vg_www6-lv_usr_local /usr/local xfs defaults,noatime 0 0 UUID=XXX /boot xfs defaults 0 0 #/dev/mapper/vg_www6-lv_tmp_mysql /tmp_mysql xfs defaults,noatime 0 0 tmpfs /tmp_mysql tmpfs defaults,noatime,nosuid,nodev,noexec,mode=0755,size=5G 0 0 /dev/mapper/vg_www6-lv_var_lib_mysql /var/lib/mysql xfs defaults,noatime,uquota,gquota 0 0 /dev/mapper/vg_www6-lv_data /data xfs defaults,noatime,uquota,gquota 0 0
In article db155fbf-7c80-f89f-d401-fcc1097dfe76@c4.cz, Ján Lalinský lalinsky@c4.cz wrote:
Hi all,
I have Centos 8 installed on a physical machine (www6) with separate LVM volumes for /, /var, /var/lib/mysql etc.
System boot proceeds without a hiccup, in terminal systemctl status says everything is OK and running, journalctl says so as well - systemd mounts everything stated in fstab.
However, on reboot systemd echoes problems with filesystem on /var :
...// unmounting all volumes
Nov 26 23:51:30 www6 systemd[1]: Unmounting /var... Nov 26 23:51:30 www6 umount[2118]: umount: /var: target is busy. Nov 26 23:51:30 www6 systemd[1]: Stopped target Swap. Nov 26 23:51:30 www6 systemd[1]: Deactivating swap /dev/disk/by-label/lv_swap... Nov 26 23:51:30 www6 systemd[1]: var.mount: Mount process exited, code=exited status=32 Nov 26 23:51:30 www6 systemd[1]: Failed unmounting /var.
Then proceeds and reboots the machine. This occurs on every reboot.
Did anybody here encounter similar problems on reboot?
I found this bug
https://github.com/systemd/systemd/issues/867
which was closed without a definitive solution, the developer says the issue is hard to solve and just cosmetic since /var gets unmounted in the end anyway, which is strange because there is a log line about unmounting just about every other mountpoint and in the end of the journal, there are about 7 log lines about unmounting swap volume, but zilch about /var. If the system is getting down without proper unmount, data corruption can happen. I would feel much better if there were no such errors...
You need to un-hide and read the hidden comments on that bug. They give a few different workarounds, but also indicate that the issue has indeed been solved in a later version of systemd. But that was only in April of this year; I don't know which version of upstream systemd was used in RHEL/CentOS 8. Maybe the fixed version won't appear until 8.2? Some of the workarounds such as lazy unmount for /var ought to work in the meantime.
Cheers Tony
Thanks, after reading some more complaints online and suggestions on this, I think setting up a lazy umount of /var is the best option until the bug is resolved in Centos 8. The lazy umount prevents systemd from registering and reporting the error that ordinary umount /var would generate (such ordinary umount of /var by systemd is a misconceived action, as /var is still in use, at least by journald).
For others struggling with this mess, here is the procedure that worked for me:
1. Comment out mountpoint /var in /etc/fstab
2. Create a file /etc/systemd/system/var.mount with these lines:
[Unit] Description=Lazy mount file for /var to work around systemd bug 867 DefaultDependencies=no Conflicts=umount.target Requires=system.slice -.mount Before=local-fs.target umount.target After=swap.target
[Mount] What=/dev/disk/by-uuid/<var-filesystem-uuid> Where=/var Type=xfs Options=defaults,noatime LazyUnmount=true
[Install] WantedBy=multi-user.target
3. Then run
systemctl daemon-reload
and that should be it, subsequent reboots should go nice and clean, hopefully without this bug rearing its head again.
Best regards Ján Lalinský
Webhosting C4 ČESKÝ WEBHOSTING s.r.o. Tel: +420 234 139 876 E-mail: info@c4.cz http://www.c4.cz
On 27/11/2019 11:35, Tony Mountifield wrote:
In article db155fbf-7c80-f89f-d401-fcc1097dfe76@c4.cz, Ján Lalinský lalinsky@c4.cz wrote:
Hi all,
I have Centos 8 installed on a physical machine (www6) with separate LVM volumes for /, /var, /var/lib/mysql etc.
System boot proceeds without a hiccup, in terminal systemctl status says everything is OK and running, journalctl says so as well - systemd mounts everything stated in fstab.
However, on reboot systemd echoes problems with filesystem on /var :
...// unmounting all volumes
Nov 26 23:51:30 www6 systemd[1]: Unmounting /var... Nov 26 23:51:30 www6 umount[2118]: umount: /var: target is busy. Nov 26 23:51:30 www6 systemd[1]: Stopped target Swap. Nov 26 23:51:30 www6 systemd[1]: Deactivating swap /dev/disk/by-label/lv_swap... Nov 26 23:51:30 www6 systemd[1]: var.mount: Mount process exited, code=exited status=32 Nov 26 23:51:30 www6 systemd[1]: Failed unmounting /var.
Then proceeds and reboots the machine. This occurs on every reboot.
Did anybody here encounter similar problems on reboot?
I found this bug
https://github.com/systemd/systemd/issues/867
which was closed without a definitive solution, the developer says the issue is hard to solve and just cosmetic since /var gets unmounted in the end anyway, which is strange because there is a log line about unmounting just about every other mountpoint and in the end of the journal, there are about 7 log lines about unmounting swap volume, but zilch about /var. If the system is getting down without proper unmount, data corruption can happen. I would feel much better if there were no such errors...
You need to un-hide and read the hidden comments on that bug. They give a few different workarounds, but also indicate that the issue has indeed been solved in a later version of systemd. But that was only in April of this year; I don't know which version of upstream systemd was used in RHEL/CentOS 8. Maybe the fixed version won't appear until 8.2? Some of the workarounds such as lazy unmount for /var ought to work in the meantime.
Cheers Tony
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-
Ján Lalinský -
tony＠softins.co.uk