[previously sent to rhelv5 list, apologies to those on both]
I've got a problem I can reproduce easily enough, but really I fail to understand what's going wrong.
I've got a 5.3 Dom0, which is running three guests. One is Fedora 10, that runs with local flat files, and works fine. One is Nexenta 2 (opensolaris-based), and that runs off of physical partitions, and seems to work great. The third runs Fedora 11 and and has for its disks, iSCSI devices that are exported from Nexenta (ZFS-backed).
I have the Dom0 mapping the two iSCSI devices, one for /boot and one for /. They're showing up initially as /dev/sdc and /dev/sdd.
If I go after the iSCSI devices in Dom0, with dd, for instance, they work fine all day. I can read and write the entire devices to and from local files without error. iSCSI seems to work properly in that regard. I'm getting about 38MB/s. I've scrubbed the disk pool and no errors were found and long SMART self-test passed on each of the disks. I've also been able to mount both iSCSI devices and run bonnie++ on them successfully from Dom0.
So, I specify those devices in the Xen config for the domU (tried both real device name and /dev/disk/by-path/ names) and the DomU boots and operates as I'd expect. Installation worked fine and typical operations (low volume) work. However, then I try to do something, which I'm assuming is more disk intensive, like running a yum update, and iSCSI seems to fall over.
In the DomU, I'll see a lock-up, and then filesystem errors. e.g.:
Installing : kernel [############################################ ] 1/33EXT3-fs error (device xvda1) in ext3_ordered_writepage: IO failure
In the Dom0, I'll see:
sd 6:0:0:0: timing out command, waited 360s sd 6:0:0:0: SCSI error: return code = 0x06050000 end_request: I/O error, dev sdc, sector 37319 sd 7:0:0:0: timing out command, waited 360s sd 7:0:0:0: SCSI error: return code = 0x06000000 end_request: I/O error, dev sdd, sector 29792137 sd 7:0:0:0: timing out command, waited 360s sd 7:0:0:0: SCSI error: return code = 0x06000000 end_request: I/O error, dev sdd, sector 29792313
Both (all) iSCSI devices are failed. Under iostat I see activity to the iSCSI block devices, and the whole machine acts mostly I/O blocked (even the Fedora 10 DomU running on flat files will start throwing nagios into a tizzy). If I do 'service iscsi stop' everything picks right back up (though the DomU using them as its disks is obviously unhappy).
When I start iscsi again I can pick right back up (after repairing filesystems in the DomU), and I can repeat the process at will. Sometimes the disks will come back as, e.g. sdd and sde, leaving me to think something still has a handle on sdc. But lsof shows nothing in dom0.
One thing that stood out were some of the block and sector number errors being right on power of two boundries:
scsi 7:0:0:0: SCSI error: return code = 0x00010000 end_request: I/O error, dev sdd, sector 32768 Buffer I/O error on device sdd, logical block 4096 Buffer I/O error on device sdd, logical block 4097 Buffer I/O error on device sdd, logical block 4098 Buffer I/O error on device sdd, logical block 4099 Buffer I/O error on device sdd, logical block 4100 Buffer I/O error on device sdd, logical block 4101 Buffer I/O error on device sdd, logical block 4102 Buffer I/O error on device sdd, logical block 4103 Buffer I/O error on device sdd, logical block 4104 Buffer I/O error on device sdd, logical block 4105 scsi 7:0:0:0: rejecting I/O to dead device
but as I opened with, I'm sort of at as loss as to what is actually causing the problem. Any suggestions for further troubleshooting and/or ideas about what's happening appreciated.
Thanks, -Bill
Bill McGonigle bill@bfccomputing.com writes:
In the DomU, I'll see a lock-up, and then filesystem errors. e.g.:
Installing : kernel [############################################ ] 1/33EXT3-fs error (device xvda1) in ext3_ordered_writepage: IO failure
In the Dom0, I'll see:
sd 6:0:0:0: timing out command, waited 360s sd 6:0:0:0: SCSI error: return code = 0x06050000 end_request: I/O error, dev sdc, sector 37319
try
xm sched-credit -d 0 60000
and see if you still have the problem.
if that doesn't work, try
xm vcpu-set 0 1
then edit all your domu config files and add: cpus="1-7"
(assuming a dual socket quad core box... the idea is to reserve a cpu for the dom0)
the idea is that if your dom0 is starved for CPU, you get all sorts of network and other I/O weirdness.
On 06/15/2009 11:33 PM, Luke S Crawford wrote:
xm sched-credit -d 0 60000
Ah, ha! This appears to work. I didn't need to reserve a CPU for the dom0 (knock on wood). Much obliged, Luke.
I'm academically curious, though - I seem to have created a CPU deadlock of some sort, yet in 'xm top' none of the CPU's were pegged. I've got no reason to not give dom0 utmost priority - that makes perfect sense to me - but I'm surprised the Xen scheduler would allow me to get into this situation by default.
-Bill
On Tue, Jun 16, 2009 at 01:57:55PM -0400, Bill McGonigle wrote:
On 06/15/2009 11:33 PM, Luke S Crawford wrote:
xm sched-credit -d 0 60000
Ah, ha! This appears to work. I didn't need to reserve a CPU for the dom0 (knock on wood). Much obliged, Luke.
I'm academically curious, though - I seem to have created a CPU deadlock of some sort, yet in 'xm top' none of the CPU's were pegged. I've got no reason to not give dom0 utmost priority - that makes perfect sense to me - but I'm surprised the Xen scheduler would allow me to get into this situation by default.
Well, you have pretty weird storage setup.. dom0 iSCSI initiator depends on domU iSCSI target, which again depends on disks on dom0..
-- Pasi
Bill McGonigle bill@bfccomputing.com writes:
On 06/15/2009 11:33 PM, Luke S Crawford wrote:
xm sched-credit -d 0 60000
Ah, ha! This appears to work. I didn't need to reserve a CPU for the dom0 (knock on wood). Much obliged, Luke.
I'm academically curious, though - I seem to have created a CPU deadlock of some sort, yet in 'xm top' none of the CPU's were pegged. I've got no reason to not give dom0 utmost priority - that makes perfect sense to me - but I'm surprised the Xen scheduler would allow me to get into this situation by default.
My understanding of this is entirely janitor-level, but I believe what you are seeing is that the dom0 has exhausted it's 'credits' and so if a DomU wants the CPU the dom0 gets kicked off the cpu, waits a timeslice (I think timeslices are on the order of tens of millaseconds... I've read 60ms, which is quite a long time in terms of sending a packet to a nearby storage box.) then gets back on the CPU.
This is why I'm always loath to give more than 1 or 2 vcpus to my DomUs, and why I always reserve cpu0 for the dom0; that way, the domu can pass a packet to the dom0 which can process it and send it out without waiting. the domU and the Dom0 can run at the same time.
If you look (xm sched-credit -d 0) you will see that xen assigns all DomUs a default priority of 256. It does not assign a higher priority to the Dom0. I assume this is because the xen people very much have an attitude of 'well, I wrote this nice hypervisor. You set it up.' Which is fine with me, as while I can set it up, there's no way I could have written the nice hypervisor. But yeah, I see no reason not to default to giving the dom0 as much cpu as it wants; if the dom0 is unhappy, everyone is unhappy.
It seems like the sort of thing RHEL could do. (well, that and increasing the default dom0-min-mem to something that doesn't crash the dom0.)
The 'xm sched-credit -d 0 60000' line is in the /etc/rc.local of all the Xen hosts I administer. It helps a lot, even when you use local disk.
I have seen this problem without using iscsi, when the DomUs are heavily loaded. I get 'stutter' on the command line and dropped packets on the interface counters. It's irritating, because without iscsi, the problem is usually rare and difficult to reproduce.
On Tue, Jun 16, 2009 at 02:59:54PM -0400, Luke S Crawford wrote:
Bill McGonigle bill@bfccomputing.com writes:
On 06/15/2009 11:33 PM, Luke S Crawford wrote:
xm sched-credit -d 0 60000
Ah, ha! This appears to work. I didn't need to reserve a CPU for the dom0 (knock on wood). Much obliged, Luke.
I'm academically curious, though - I seem to have created a CPU deadlock of some sort, yet in 'xm top' none of the CPU's were pegged. I've got no reason to not give dom0 utmost priority - that makes perfect sense to me - but I'm surprised the Xen scheduler would allow me to get into this situation by default.
My understanding of this is entirely janitor-level, but I believe what you are seeing is that the dom0 has exhausted it's 'credits' and so if a DomU wants the CPU the dom0 gets kicked off the cpu, waits a timeslice (I think timeslices are on the order of tens of millaseconds... I've read 60ms, which is quite a long time in terms of sending a packet to a nearby storage box.) then gets back on the CPU.
afaik Xen credit scheduler assigns/changes credits every 30ms.
although CPU scheduling happens more often?
-- Pasi
On 06/16/2009 02:59 PM, Luke S Crawford wrote:
It seems like the sort of thing RHEL could do.
agreed. I filed a request for this: https://bugzilla.redhat.com/show_bug.cgi?id=506370
I have seen this problem without using iscsi, when the DomUs are heavily loaded. I get 'stutter' on the command line and dropped packets on the interface counters. It's irritating, because without iscsi, the problem is usually rare and difficult to reproduce.
ah... I've had packets go mysteriously missing as well. I'll try this on those machines.
Thanks again!
-Bill
-
Bill McGonigle -
Luke S Crawford -
Pasi Kärkkäinen