Adding Paolo and Miroslav. On Sat, Apr 8, 2017 at 4:49 PM, Richard Landsman - Rimote <richard at rimote.nl > wrote: > Hello, > > I would really appreciate some help/guidance with this problem. First of > all sorry for the long message. I would file a bug, but do not know if it > is my fault, dm-cache, qemu or (probably) a combination of both. And i can > imagine some of you have this setup up and running without problems (or > maybe you think it works, just like i did, but it does not): > > PROBLEM > LVM cache writeback stops working as expected after a while with a > qemu-kvm VM. A 100% working setup would be the holy grail in my opinion... > and the performance of KVM/qemu is great i must say in the beginning. > > DESCRIPTION > > When using software RAID 1 (2x HDD) + software RAID 1 (2xSSD) and create a > cached LV out of them, the VM performs initially great (at least 40.000 > IOPS on 4k rand read/write)! But then after a while (and a lot of random > IO, ca 10 - 20 G) it effectively turns in to a writethrough cache although > there's much space left on the cachedlv. > > > When working as expected on KVM host all writes go to SSDs > > iostat -x -m 2 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz > avgqu-sz await r_await w_await svctm %util > sda 0.00 324.50 0.00 22.00 0.00 14.94 > 1390.57 1.90 86.39 0.00 86.39 5.32 11.70 > sdb 0.00 324.50 0.00 22.00 0.00 14.94 > 1390.57 2.03 92.45 0.00 92.45 5.48 12.05 > sdc 0.00 3932.00 0.00 *2191.50* 0.00 *270.07* > 252.39 37.83 17.55 0.00 17.55 0.36 * 78.05* > sdd 0.00 3932.00 0.00 *2197.50 * 0.00 *271.01 * > 252.57 38.96 18.14 0.00 18.14 0.36 *78.95* > > > When not working as expected on KVM host all writes go through the SSD on > to the HDDs (effectively disabling writeback so it becomes a writethrough) > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz > avgqu-sz await r_await w_await svctm %util > sda 0.00 7.00 234.50 *173.50 * 0.92 * 1.95* > 14.38 29.27 71.27 111.89 16.37 2.45 *100.00* > sdb 0.00 3.50 212.00 *177.50 * 0.83 * 1.95* > 14.60 35.58 91.24 143.00 29.42 2.57* 100.10* > sdc 2.50 0.00 566.00 *199.00 * 2.69 0.78 > 9.28 0.08 0.11 0.13 0.04 0.10 *7.70* > sdd 1.50 0.00 76.00 *199.00* 0.65 0.78 > 10.66 0.02 0.07 0.16 0.04 0.07 *1.85* > > > Stuff i've checked/tried: > > - The data in the cached LV has then not exceeded even half of the space, > so this should not happen. It even happens when only 20% of cachedata is > used. > - It seems to be triggerd most of the time when %cpy/sync column of `lvs > -a` is about 30%. But this is not always the case! > - changing the cachepolicy from cleaner to smq, wait (check flush ready > with lvs -a) and then back to smq seems to help *sometimes*! But not > always... > > lvchange --cachepolicy cleaner /dev/mapper/XXX-cachedlv > > lvs -a > > lvchange --cachepolicy smq /dev/mapper/XXX-cachedlv > > - *when mounting the LV inside the host this does not seem to happen!!* > So it looks like a qemu-kvm / dm-cache combination issue. Only difference > is that inside host i do mkfs in stead of LVM inside VM (so could be LVM > inside VM on top of LVM on KVM host problem too? small chance probably > because the first 10 - 20GB it works great!) > > - tried disabling Selinux, upgrading to newest kernels (elrepo ml and lt), > played around with dirty_cache thingeys like proc/sys/vm/dirty_writeback_centisecs > /proc/sys/vm/dirty_expire_centisecs cat /proc/sys/vm/dirty_ratio , and > migration threashold of dmsetup, and other probably non important stuff > like vm.dirty_bytes > > - when in "slow state" the systems kworkers are exessively using IO (10 - > 20 MB per kworker process). This seems to be the writeback process > (CPY%Sync) because the cache wants to flush to HDD. But the strange thing > is that after a good sync (0% left), the disk may become slow again after a > few MBs of data. A reboot sometimes helps. > > - have tried iothreads, virtio-scsi, vcpu driver setting on virtio-scsi > controller, cachesettings, disk shedulers etc. Nothing helped. > > - the new samsung 950 PRO SSDs have HPA enabled (30%!!), i have AMD > FX(tm)-8350, 16G RAM > > It feels like the lvm cache has a threshold (about 20G of data that is > dirty) and that is stops allowing the qemu-kvm process to use writeback > caching (the root uses inside the host seems to not have this limitation). > It starts flushing, but only to a certain point. After a few MBs of data > it is right back in the slow spot again. Only solution is waiting for a > long time (independant of CPY%SYNC) or sometimes change cachepolicy and > force flush. This prevents for me the production use of this system. But > it's so promising, so I hope somebody can help. > > desired state: Doing the FIO test (described in section reproduce) > repeatedly should keep being fast till cachedlv is more or less full. If > resyncing back to disc causes this degradation, it should actually flush it > fully within a reasonable time and give opportunity to write fast again up > to a given threshold. It now seems like a one time use cache that only uses > a fraction of the SSD and is useless/very unstable afterwards. > > REPRODUCE > 1. Install newest CentOS 7 on software RAID 1 HDDs with LVM. Keep a lot of > space for the LVM cache (no /home)! So make the VG as large as possible > during anaconda partitioning. > > 2. once installed and booted in to the system, install qemu-kvm > > yum install -y centos-release-qemu-ev > yum install -y qemu-kvm-ev libvirt bridge-utils net-tools > # disbale ksm (probably not important / needed) > systemctl disable ksm > systemctl disable ksmtuned > > 3. create LVM cache > > #set some variables and create a raid1 array with the two SSDs > > VGBASE= && ssddevice1=/dev/sdX1 && ssddevice2=/dev/sdX1 && > hddraiddevice=/dev/mdXXX && ssdraiddevice=/dev/mdXXX && mdadm --create > --verbose ${ssdraiddevice} --level=mirror --bitmap=none --raid-devices=2 > ${ssddevice1} ${ssddevice2} > > # create PV and extend VG > > pvcreate ${ssdraiddevice} && vgextend ${VGBASE} ${ssdraiddevice} > > # create slow LV on HDDs (use max space left if you want) > > pvdisplay ${hddraiddevice} > lvcreate -lXXXX -n cachedlv ${VGBASE} ${hddraiddevice} > > # create the meta and data: for testing purposes I keep about 20G of the > SSD for a uncached lv. To rule out it is not the SSD. > > lvcreate -l XX -n testssd ${VGBASE} ${ssdraiddevice} > > #The rest can be used as cachedata/metadata. > > pvdisplay ${ssdraiddevice} > # about 1/1000 of the space you have left on the SSD for the meta (minimum > of 4) > lvcreate -l X -n cachemeta ${VGBASE} ${ssdraiddevice} > # the rest can be used as cachedata > lvcreate -l XXX -n cachedata ${VGBASE} ${ssdraiddevice} > > # convert/combine pools so cachedlv is actually cached > > lvconvert --type cache-pool --cachemode writeback --poolmetadata > ${VGBASE}/cachemeta ${VGBASE}/cachedata > > lvconvert --type cache --cachepool ${VGBASE}/cachedata ${VGBASE}/cachedlv > > > # my system now looks like (VG is called cl, default of installer) > [root at localhost ~]# lvs -a > LV VG Attr LSize Pool Origin > [cachedata] cl Cwi---C--- 97.66g > > * [cachedata_cdata] cl Cwi-ao---- > 97.66g * > * [cachedata_cmeta] cl ewi-ao---- 100.00m * > > * cachedlv cl Cwi-aoC--- 1.75t [cachedata] > [cachedlv_corig] * > [cachedlv_corig] cl owi-aoC--- 1.75t > > [lvol0_pmspare] cl ewi------- 100.00m > > root cl -wi-ao---- 46.56g > > swap cl -wi-ao---- 14.96g > > > > * testssd cl -wi-a----- 45.47g *[root at localhost ~]#lsblk > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sdd 8:48 0 163G 0 disk > └─sdd1 8:49 0 163G 0 part > └─md128 9:128 0 162.9G 0 raid1 > ├─cl-cachedata_cmeta 253:4 0 100M 0 lvm > │ └─cl-cachedlv 253:6 0 1.8T 0 lvm > ├─cl-testssd 253:2 0 45.5G 0 lvm > └─cl-cachedata_cdata 253:3 0 97.7G 0 lvm > └─cl-cachedlv 253:6 0 1.8T 0 lvm > sdb 8:16 0 1.8T 0 disk > ├─sdb2 8:18 0 1.8T 0 part > │ └─md127 9:127 0 1.8T 0 raid1 > │ ├─cl-swap 253:1 0 15G 0 lvm [SWAP] > │ ├─cl-root 253:0 0 46.6G 0 lvm / > │ └─cl-cachedlv_corig 253:5 0 1.8T 0 lvm > │ └─cl-cachedlv 253:6 0 1.8T 0 lvm > └─sdb1 8:17 0 954M 0 part > └─md126 9:126 0 954M 0 raid1 /boot > sdc 8:32 0 163G 0 disk > └─sdc1 8:33 0 163G 0 part > └─md128 9:128 0 162.9G 0 raid1 > ├─cl-cachedata_cmeta 253:4 0 100M 0 lvm > │ └─cl-cachedlv 253:6 0 1.8T 0 lvm > ├─cl-testssd 253:2 0 45.5G 0 lvm > └─cl-cachedata_cdata 253:3 0 97.7G 0 lvm > └─cl-cachedlv 253:6 0 1.8T 0 lvm > sda 8:0 0 1.8T 0 disk > ├─sda2 8:2 0 1.8T 0 part > │ └─md127 9:127 0 1.8T 0 raid1 > │ ├─cl-swap 253:1 0 15G 0 lvm [SWAP] > │ ├─cl-root 253:0 0 46.6G 0 lvm / > │ └─cl-cachedlv_corig 253:5 0 1.8T 0 lvm > │ └─cl-cachedlv 253:6 0 1.8T 0 lvm > └─sda1 8:1 0 954M 0 part > └─md126 9:126 0 954M 0 raid1 /boot > > # now create vm > wget http://ftp.tudelft.nl/centos.org/6/isos/x86_64/CentOS-6.9- > x86_64-minimal.iso -P /home/ > DISK=/dev/mapper/XXXX-cachedlv > > # watch out, my netsetup uses a custom bridge/network in the following > command. Please replace with what you normally use. > virt-install -n CentOS1 -r 12000 --os-variant=centos6.7 --vcpus 7 --disk > path=${DISK},cache=none,bus=virtio --network bridge=pubbr,model=virtio > --cdrom /home/CentOS-6.9-x86_64-minimal.iso --graphics > vnc,port=5998,listen=0.0.0.0 --cpu host > > # now connect with client PC to qemu > virt-viewer --connect=qemu+ssh://root@192.168.0.XXX/system --name CentOS1 > > And install everything on the single vda disc with LVM (i use defaults in > anaconda, but remove the large /home to prevent SSD beeing over used). > > After install and reboot log in to VM and > > yum install epel-release -y && yum install screen fio htop -y > > and then run disk test: > > fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 > --name=test *--filename=test* --bs=4k --iodepth=64 --size=4G > --readwrite=randrw --rwmixread=75 > > then *keep repeating *but *change the filename* attribute so it does not > use the same blocks over and over again. > > In the beginning the performance is great!! Wow, very impressive 150MB/s > 4k random r/w (close to bare metal, about 20% - 30% loss). But after a few > (usually about 4 or 5) runs (always changing the filename, but not > overfilling the FS, it drops to about 10 MBs/sec. > > normal/in the beginning > > read : io=3073.2MB, bw=183085KB/s, *iops=45771* , runt= 17188msec > write: io=1022.1MB, bw=60940KB/s, *iops=15235* , runt= 17188msec > > but then > > read : io=3073.2MB, bw=183085KB/s, *iops=**2904* , runt= 17188msec > write: io=1022.1MB, bw=60940KB/s, *iops=1751* , runt= 17188msec > > or even worse up to the point that it is actually the HDD that is written > to (about 500 iops). > > P.S. when a test is/was slow, that means it is on the HDDs. So even when > fixing the problem (sometimes just by waiting), that specific file will > keep being slow when redoing the test till its promoted to the lvm cache > (takes a lot of reads I think). And once on the SSD it sometimes keeps > beeing fast, although a new testfile will be slow. So I really recommend > changing the testfile all the time when trying to see if a change in speed > has occurred. > > -- > Met vriendelijke groet, > > Richard Landsmanhttp://rimote.nl > > T: +31 (0)50 - 763 04 07 > (ma-vr 9:00 tot 18:00) > > 24/7 bij storingen: > +31 (0)6 - 4388 7949 > @RimoteSaS (Twitter Serviceberichten/security updates) > > > _______________________________________________ > CentOS-virt mailing list > CentOS-virt at centos.org > https://lists.centos.org/mailman/listinfo/centos-virt > > -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20170410/c61d4193/attachment-0003.html>