[CentOS] Centos 7 guest - long delay on mounting /boot with host disk write cache off

Wed Dec 30 14:26:10 UTC 2015
Andrzej Szymanski <szymans at agh.edu.pl>


I've noticed a strange delay while booting a CentOS 7 guest on a CentOS 
7 host with slow disks (7200RPM) with write cache off.

The guest and host are freshly installed Centos 7 (host was fully 
patched before guest install). Guest is installed on an lvm pool 
residing on an md raid1 with two SATA 7200 RPM drives with their write 
caches off.

The delay is on mounting /boot, the dmesg on the guest says:

[   23.491857] XFS (vda1): Mounting V4 Filesystem
[   46.668396] XFS (vda1): Ending clean mount
[   46.668404] SELinux: initialized (dev vda1, type xfs), uses xattr

Subsequent mounts by hand on a running system last even longer - dmesg 

[   96.442544] XFS (vda1): Unmounting Filesystem
[  100.086741] XFS (vda1): Mounting V4 Filesystem
[  146.098703] XFS (vda1): Ending clean mount
[  146.098721] SELinux: initialized (dev vda1, type xfs), uses xattr

mounting "-o ro" doesn't help - dmesg records:

[  205.552778] XFS (vda1): Unmounting Filesystem
[  212.452724] XFS (vda1): Mounting V4 Filesystem
[  258.537459] XFS (vda1): Ending clean mount
[  258.537477] SELinux: initialized (dev vda1, type xfs), uses xattr

but mounting with "-o ro,norecovery" does help - dmesg records:

[  263.795813] XFS (vda1): Unmounting Filesystem
[  271.973701] XFS (vda1): Mounting V4 filesystem in no-recovery mode. 
Filesystem will be inconsistent.
[  272.003618] SELinux: initialized (dev vda1, type xfs), uses xattr

xfs_logprint -t /dev/vda1 says that the device is clean:

     data device: 0xfc01
     log device: 0xfc01 daddr: 512032 length: 6824

     log tail: 3447 head: 3447 state: <CLEAN>

however the full xfs log contains a lot of entries
xfs_logprint /dev/vda1 results in ~23000 lines of text.

Permanent help is from xfs_repair which zeroes the log - a subsequent 
mount is fast, and subsequent reboots are fast.

[  411.290031] XFS (vda1): Mounting V4 Filesystem
[  411.456661] XFS (vda1): Ending clean mount
[  411.456676] SELinux: initialized (dev vda1, type xfs), uses xattr

During the problematic mounts the host reports a lot of small disk 
writes with queue depth ~1. Also, when I move the problematic filesystem 
using dd to an lvm volume, mount -o ro, unmount, make snapshot, mount -o 
ro again, then the contents of the original volume are the same as the 
snapshot but the snapshot usage increases from 0 to ~2MB - so something 
writes to the original volume in a ro mode - possibly a log replay.

Is is normal for the xfs mount to replay a log after a clean unmount?

With best regards,

Andrzej Szymański, Ph.D.

tel. +48 12 6174035
fax. +48 12 6342372
e-mail: szymans at agh.edu.pl

AGH University of Science and Technology
Department of Telecommunications
Al. Mickiewicza 30
30-059 Kraków