On Thu, Jun 4, 2015 at 11:57 AM, George Dunlap dunlapg@umich.edu wrote:
On Thu, Jun 4, 2015 at 6:03 AM, Sarah Newman srn@prgmr.com wrote:
By default, /var/lib/xenstored is mounted tmpfs in centos 5 but this doesn't appear to be true for Xen4CentOS. This can cause performance issues as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=446089
I'm not sure if this should be part of xencommons, but if not, adding a tmpfs mount should probably be documented on the quick start wiki page.
Sarah,
Thanks for looking into this.
Adding a tmpfs mount to xencommons, if using the C xenstored, might be a good idea -- I'll float that with upstream and see what they say.
As an alternate, can you try using the ocaml-based oxenstored (which doesn't use the filesystem as its atomicity mechanism)?
To use oxenstored:
- Install the xen-ocaml package
- Reboot dom0
- Verify that oxenstored is running:
# ps ax | grep xenstore 654 ? S 0:00 /usr/sbin/oxenstored --pid-file /var/run/xenstored.pid
(On my to-do list is to get oxenstored into the core xen-runtime package, which would also resolve this situation, I believe.)
FWIW, I used the following shell snippet to create 80 VMs (which crash immediately without an initrd but still take up space in xenstore):
for i in {1..80} ; do xl create null.cfg name="t$i" kernel="/boot/vmlinuz-3.18.12-11.el7.x86_64" memory="40" on_crash="preserve" ; done
And then used the shell snippet from the bug report to generate load:
while true; do xl list &> /dev/null; usleep 5000;done
With xenstored normally the CPU load was about 30%. Mounted under tmpfs, it was slightly smaller, around 20%, but swung around wildly. With oxenstored it was about 15%.
This is with xfs as the filesystem.
That doesn't seem big enough to warrant a lot of effort: if you can reproduce the 75% numbers from the bug report I'll pursue it further, otherwise I'll probably just leave it.
-George