[CentOS-virt] Xen vs. iSCSI

Bill McGonigle <bill at bfccomputing.com> writes:

> On 06/15/2009 11:33 PM, Luke S Crawford wrote:
> > xm sched-credit -d 0 60000
> 
> Ah, ha!  This appears to work.  I didn't need to reserve a CPU for the 
> dom0 (knock on wood).  Much obliged, Luke.
> 
> I'm academically curious, though - I seem to have created a CPU deadlock 
> of some sort, yet in 'xm top' none of the CPU's were pegged.  I've got 
> no reason to not give dom0 utmost priority - that makes perfect sense to 
> me - but I'm surprised the Xen scheduler would allow me to get into this 
> situation by default.


My understanding of this is entirely janitor-level, but I believe what you
are seeing is that the dom0 has exhausted it's 'credits'  and so if a
DomU wants the CPU the dom0 gets kicked off the cpu, waits a timeslice
(I think timeslices are on the order of tens of millaseconds...  I've
read 60ms, which is quite a long time in terms of sending a packet to a
nearby storage box.)  then gets back on the CPU.  

This is why I'm always loath to give more than 1 or 2 vcpus to my DomUs,
and why I always reserve cpu0 for the dom0;   that way, the domu
can pass a packet to the dom0 which can process it and send it out
without waiting.   the domU and the Dom0 can run at the same time.

If you look (xm sched-credit -d 0)  you will see that xen assigns
all DomUs a default priority of 256.   It does not assign a higher priority
to the Dom0.  I assume this is because the xen people very much have
an attitude of 'well, I wrote this nice hypervisor.  You set it up.'  
Which is fine with me, as while I can set it up, there's no way I could
have written the nice hypervisor.  But yeah, I see no reason not to 
default to giving the dom0 as much cpu as it wants;  if the dom0 is 
unhappy, everyone is unhappy.

It seems like the sort of thing RHEL could do.   (well, that and 
increasing the default dom0-min-mem to something that doesn't
crash the dom0.)  

The 'xm sched-credit -d 0 60000' line is in the /etc/rc.local
of all the Xen hosts I administer.  It helps a lot, even when you
use local disk. 

I have seen this problem without using iscsi, when the DomUs are heavily
loaded.   I get 'stutter' on the command line and dropped packets 
on the interface counters.   It's irritating, because without iscsi, the 
problem is usually rare and difficult to reproduce.