Hi All,
I'm seeing tapdisk processes not being terminated after a HVM vm is shutdown or migrated away. I don't see this problem with linux paravirt domu's, just windows hvm ones.
xl.cfg:
name = 'nathanwin'
memory = 4096
vcpus = 2
disk = [ 'file:/mnt/gtc_disk_p1/nathanwin/drive_c,hda,w' ]
vif = [ 'mac=00:16:3D:01:03:E0,bridge=vlan208' ]
builder = "hvm"
kernel = "/usr/lib/xen/boot/hvmloader"
localtime = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "destroy"
vnc = 1
vncunused = 1
cpuid = [
'0:eax=00000000000000000000000000001011',
'1:eax=00000000000000100000011011000010,ecx=10000011101110100010001000000011,edx=00010111100010111111101111111111',
'2:eax=01010101000000110101101000000001',
'7,0:eax=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,ebx=00000000000000000000000000000000,ecx=00000000000000000000000000000000,edx=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
'13,1:eax=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0',
'10:ebx=00000000000000000000000000000000',
'11:edx=00000000000000000000000000000000',
'2147483650:eax=01100101011101000110111001001001,ebx=00101001010100100010100001101100,ecx=01101111011001010101100000100000,edx=00101001010100100010100001101110',
'2147483651:eax=01010101010100000100001100100000,ebx=00100000001000000010000000100000,ecx=00100000001000000010000000100000,edx=01001100001000000010000000100000',
'2147483652:eax=00110000001101000011011000110101,ebx=00100000010000000010000000100000,ecx=00110111001100100010111000110010,edx=00000000011110100100100001000111',
'2147483656:eax=00000000000000000011000000101000',
]
Starting with the VM running initially on another host, I migrate it in:
migration target: Ready to receive domain.
Saving to migration stream new xl format (info 0x0/0x0/1450)
Loading new save file <incoming migration stream> (new xl fmt info 0x0/0x0/1450)
Savefile contains xl domain config
WARNING: ignoring "kernel" directive for HVM guest. Use "firmware_override" instead if you really want a non-default firmware
xc: progress: Reloading memory pages: 56320/1114193 5%
xc: progress: Reloading memory pages: 1003520/1114193 90%
DEBUG libxl__blktap_devpath 37 aio:/mnt/gtc_disk_p1/nathanwin/drive_c
DEBUG libxl__blktap_devpath 40 /dev/xen/blktap-2/tapdev0
DEBUG libxl__blktap_devpath 37 aio:/mnt/gtc_disk_p1/nathanwin/drive_c
DEBUG libxl__blktap_devpath 40 /dev/xen/blktap-2/tapdev2
migration target: Transfer complete, requesting permission to start domain.
migration sender: Target has acknowledged transfer.
migration sender: Giving target permission to start.
migration target: Got permission, starting domain.
migration target: Domain started successsfully.
migration sender: Target reports successful startup.
DEBUG libxl__device_destroy_tapdisk 66 type=aio:/mnt/gtc_disk_p1/nathanwin/drive_c disk=:/mnt/gtc_disk_p1/nathanwin/drive_c
Migration successful.
and now I have 2 tapdisk procs:
gtc-vana-005 ~ # ps auxf | grep tapdisk
root 32491 0.1 0.2 20364 4636 ? SLs 11:06 0:00 tapdisk
root 32520 0.0 0.2 20364 4636 ? SLs 11:06 0:00 tapdisk
Which seems odd given that the VM in question only has a single disk attached to it and the qemu proc indicates it's using tapdev2:
root 32524 0.4 0.7 323208 15040 ? SLsl 11:06 0:00 /usr/lib/xen/bin/qemu-system-i386 -xen-domid 3 -chardev socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-3,server,nowait -mon chardev=libxl-cmd,mode=control -nodefaults -name nathanwin--incoming -vnc 127.0.0.1:0,to=99 -device cirrus-vga -global vga.vram_size_mb=8 -boot order=cda -smp 2,maxcpus=2 -device rtl8139,id=nic0,netdev=net0,mac=00:16:3d:01:03:e0 -netdev type=tap,id=net0,ifname=vif3.0-emu,script=no,downscript=no -incoming fd:13 -machine xenfv -m 4088 -drive file=/dev/xen/blktap-2/tapdev2,if=ide,index=0,media=disk,format=raw,cache=writeback
gtc-vana-005 ~ # lsof -p 32520 | grep blktap-2
tapdisk 32520 root mem CHR 246,2 886671 /dev/xen/blktap-2/blktap2
tapdisk 32520 root 19u CHR 246,2 0t0 886671 /dev/xen/blktap-2/blktap2
gtc-vana-005 ~ # lsof -p 32491 | grep blktap-2
tapdisk 32491 root mem CHR 246,0 903999 /dev/xen/blktap-2/blktap0
tapdisk 32491 root 14u CHR 246,0 0t0 903999 /dev/xen/blktap-2/blktap0
I then migrate this VM off to another host:
migration target: Ready to receive domain.
Saving to migration stream new xl format (info 0x0/0x0/1450)
Loading new save file <incoming migration stream> (new xl fmt info 0x0/0x0/1450)
Savefile contains xl domain config
WARNING: ignoring "kernel" directive for HVM guest. Use "firmware_override" instead if you really want a non-default firmware
xc: progress: Reloading memory pages: 56320/1114193 5%
xc: progress: Reloading memory pages: 1003520/1114193 90%
DEBUG libxl__blktap_devpath 37 aio:/mnt/gtc_disk_p1/nathanwin/drive_c
DEBUG libxl__blktap_devpath 40 /dev/xen/blktap-2/tapdev2
DEBUG libxl__blktap_devpath 37 aio:/mnt/gtc_disk_p1/nathanwin/drive_c
DEBUG libxl__blktap_devpath 40 /dev/xen/blktap-2/tapdev3
migration target: Transfer complete, requesting permission to start domain.
migration sender: Target has acknowledged transfer.
migration sender: Giving target permission to start.
migration target: Got permission, starting domain.
migration target: Domain started successsfully.
migration sender: Target reports successful startup.
DEBUG libxl__device_destroy_tapdisk 66 type=aio:/mnt/gtc_disk_p1/nathanwin/drive_c disk=:/mnt/gtc_disk_p1/nathanwin/drive_c
Migration successful.
and I'm down to one tapdisk proc that didn't get cleaned up:
gtc-vana-005 ~ # ps auxf | grep tapdisk
root 32520 0.0 0.2 20364 4636 ? SLs 11:06 0:00 tapdisk
So it seems like xen is creating a second tapdisk proc on startup for some reason when it doesn't need to, then on cleanup it's only killing one of the two procs.
Any thoughts? This is on the latest 4.4.1-7.el6 packages.
Thanks!
- Nathan