I am certainly no expert on Xen. I have read through docs and various threads a bit considering the I/O demands and have the impression that there are a couple of primary factors to work with (please correct me if I'm wrong). My comprehension is far from complete. 1 - Select which scheduler, weight and cap to use. Some favor computation over I/O and vice versa. If your dom's are "fighting" over disk I/O this is the referee and enforces the rules YOU choose to keep it "fair" and efficient as possible. Suggested read, document date apparently Jul-18-2009: <http://cseweb.ucsd.edu/~dgupta/papers/per07-3sched-xen.pdf> Xen 3.3 User's Manual <http://bits.xensource.com/Xen/docs/user.pdf> 2 - Given that you can throw more RAM and CPU at each dom than it needs (not CPU or memory constrained), I/O hardware is important. Roughly speaking, the more disk controllers in play, the more bandwidth you have. Likewise, the faster the disks and perhaps the bigger their caches the faster they can offload the task. (My experience has shown me that larger built-in disk cache size substantially improves performance on email servers.) 3 - From what I have gleaned on forums and lists there may be a relationship or synergy between the scheduler selected and the type of disk array in use. HYPOTHETICALLY a RAID1 vs. a RAID5 in a given situation may show their best performance from different Xen scheduling. I don't think there is an "answer" for you that is right without you running tests with your setup and seeing which is best. From what I have read the I/O operation test should be a task that runs longer than 30 seconds. I would test what I think would be the longest possible duration and dd a huge file while hammering the other domUs with simulated "normal" request (e.g. Apache ab running at websites on other domUs). YMMV and you may not know what to do until an actual issue arise post deployment. Compared Scheduler performance can vary in different Xen releases. I have not fully investigated changing the scheduling management of weight and cap using cron/atd type management and allocate more resources at certain times to different domUs to help them get things done faster, or conversely to reduce their impact on other domUs when they are scheduled to be I/O intense. ~~~~~~~~~~~~~ Usage: xm sched-credit -d <Domain> [-w[=WEIGHT]|-c[=CAP]] Get/set credit scheduler parameters. -d DOMAIN, --domain=DOMAIN Domain to modify -w WEIGHT, --weight=WEIGHT Weight (int) -c CAP, --cap=CAP Cap (int) ~~~~~~~~~~~~~ Dennis J. wrote: > On 07/27/2009 03:17 PM, Manuel Wolfshant wrote: >> Dennis J. wrote: >>> Hi, >>> What is the best way to deal with I/O load when running several VMs on a >>> physical machine with local or remote storage? >>> >>> What I'm primarily worried about is the case when several VMs cause disk >>> I/O at the same time. One example would be the "updatedb" cronjob of the >>> mlocate package. If you have say 5 VMs running on a physical System with a >>> local software raid-1 as storage and the all run updatedb at the same time >>> that causes all of them to run really slowly because the starve each other >>> fighting over the disk. >>> >>> >> In this particular case, I would edit /etc/crontab on each VM and move >> the update at another time instead of the traditional 4:02 > > My problem with that solution is that it puts assumptions about the host > environment into the guest. If for example I migrate that client to another > physical host I actually have to remember to modify that entry and maybe > other to fit into an appropriate time slot on the new host machine. > > I'm aware that you cannot make the pain of 5 VMs hammering the disk in > parallel go away completely but I'm interested in finding out what can be > done to reduce that pain as much as possible without making the guest too > aware of its hosting environment. > > Regards, > Dennis > _______________________________________________ > CentOS-virt mailing list > CentOS-virt at centos.org > http://lists.centos.org/mailman/listinfo/centos-virt >