On Thu, Jun 4, 2015 at 11:57 AM, George Dunlap dunlapg@umich.edu wrote:

On Thu, Jun 4, 2015 at 6:03 AM, Sarah Newman srn@prgmr.com wrote:

...

By default, /var/lib/xenstored is mounted tmpfs in centos 5 but this doesn't appear to be true for Xen4CentOS. This can cause performance issues as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=446089

I'm not sure if this should be part of xencommons, but if not, adding a tmpfs mount should probably be documented on the quick start wiki page.

Sarah,

Thanks for looking into this.

Adding a tmpfs mount to xencommons, if using the C xenstored, might be a good idea -- I'll float that with upstream and see what they say.

As an alternate, can you try using the ocaml-based oxenstored (which doesn't use the filesystem as its atomicity mechanism)?

To use oxenstored:

• Install the xen-ocaml package
• Reboot dom0
• Verify that oxenstored is running:

# ps ax | grep xenstore 654 ? S 0:00 /usr/sbin/oxenstored --pid-file /var/run/xenstored.pid

(On my to-do list is to get oxenstored into the core xen-runtime package, which would also resolve this situation, I believe.)

FWIW, I used the following shell snippet to create 80 VMs (which crash immediately without an initrd but still take up space in xenstore):

for i in {1..80} ; do xl create null.cfg name="t$i" kernel="/boot/vmlinuz-3.18.12-11.el7.x86_64" memory="40" on_crash="preserve" ; done

And then used the shell snippet from the bug report to generate load:

while true; do xl list &> /dev/null; usleep 5000;done

With xenstored normally the CPU load was about 30%. Mounted under tmpfs, it was slightly smaller, around 20%, but swung around wildly. With oxenstored it was about 15%.

This is with xfs as the filesystem.

That doesn't seem big enough to warrant a lot of effort: if you can reproduce the 75% numbers from the bug report I'll pursue it further, otherwise I'll probably just leave it.

-George