Hello all,
I'm addressing the mailing list following the suggestion of gwd on #centos-virt @freenode IRC.
The subject says most of it. Here are the details :
- Was running 4.4.x and configured such that /etc/xen/auto domUs would be saved/restored on dom0 shutdown/bootup. - Installation was based on "centos-release-xen" such that "yum upgrade" brough Xen to 4.6.1.
After upgrade observed behavior: - dom0 boot starts up the domUs in /etc/xen/auto. - dom0 shutdown does not save/shutdown domUs (the command is "shutdown -h now" via ssh).
Diagnostics: - Running /etc/init.d/xendomains stop/start does the right thing. - chkconfig --list shows "xendomains 0:off 1:off 2:on 3:on 4:on 5:on 6:off" which looks nice. - The rc link is in place: /etc/rc.d/rc0.d/K00xendomains -> ../init.d/xendomains - Edited /etc/init.d/xendomains such that the lines:
start) $LIBEXEC_BIN/xendomains start ;; stop) $LIBEXEC_BIN/xendomains stop ;;
become (added "2>&1 | tee FILE" to the underlying start/stop command)
start) $LIBEXEC_BIN/xendomains start 2>&1 | /usr/bin/tee -a /tmp/xd-start.log ;; stop) $LIBEXEC_BIN/xendomains stop 2>&1 | /usr/bin/tee -a /tmp/xd-stop.log ;;
- The /tmp/xd-stop.log file is not created on dom0 shutdown (make sense, but why?!). - The /tmp/xd-stop.log is created on dom0 boot, as expected. - Both files are created on manual /etc/init.d/xendomains start/stop - Manually added an rc link: /etc/rc.d/rc0.d/K01xendomains -> ../init.d/xendomains - The behavior remains unchanged.
This is looking much more of an upstart/sysvinit issue that a Xen 4.6 issue. However, I believe others may have been through this experience.
Can anyone provide any pointers in further diagnosing this? Thanks in advance.
Regards, -- exvito
On Thu, Apr 7, 2016 at 5:46 PM, exvito here ex.vitorino@gmail.com wrote:
Hello all,
I'm addressing the mailing list following the suggestion of gwd on #centos-virt @freenode IRC.
The subject says most of it. Here are the details :
- Was running 4.4.x and configured such that /etc/xen/auto domUs would be
saved/restored on dom0 shutdown/bootup.
- Installation was based on "centos-release-xen" such that "yum upgrade"
brough Xen to 4.6.1.
After upgrade observed behavior:
- dom0 boot starts up the domUs in /etc/xen/auto.
- dom0 shutdown does not save/shutdown domUs (the command is "shutdown -h
now" via ssh).
Diagnostics:
- Running /etc/init.d/xendomains stop/start does the right thing.
- chkconfig --list shows "xendomains 0:off 1:off 2:on 3:on 4:on 5:on
6:off" which looks nice.
- The rc link is in place: /etc/rc.d/rc0.d/K00xendomains ->
../init.d/xendomains
Edited /etc/init.d/xendomains such that the lines:
start) $LIBEXEC_BIN/xendomains start ;; stop) $LIBEXEC_BIN/xendomains stop ;;
become (added "2>&1 | tee FILE" to the underlying start/stop command)
start) $LIBEXEC_BIN/xendomains start 2>&1 | /usr/bin/tee -a
/tmp/xd-start.log ;; stop) $LIBEXEC_BIN/xendomains stop 2>&1 | /usr/bin/tee -a /tmp/xd-stop.log ;;
- The /tmp/xd-stop.log file is not created on dom0 shutdown (make sense, but
why?!).
- The /tmp/xd-stop.log is created on dom0 boot, as expected.
- Both files are created on manual /etc/init.d/xendomains start/stop
- Manually added an rc link: /etc/rc.d/rc0.d/K01xendomains ->
../init.d/xendomains
- The behavior remains unchanged.
This is looking much more of an upstart/sysvinit issue that a Xen 4.6 issue. However, I believe others may have been through this experience.
Can anyone provide any pointers in further diagnosing this? Thanks in advance.
Hey exvito,
I finally got a chance to look at this. It turns out that one of the paths was incorrectly changed between 4.4 and 4.6.
To fix your problem, after upgrading to 4.6, please edit the following file:
/etc/xen/scripts/hotplugpath.sh
and replace
XEN_LOCK_DIR="/var/lock"
with
XEN_LOCK_DIR="/var/lock/subsys"
I'll be working on a proper fix with upstream -- I'll post here when I've got packages with the backported change to test.
What's happening is that the CentOS init scripts only shut down services that indicate they have an active component running -- which it takes to mean, "Has a lockfile in /var/lock/subsys". As you can probably guess, in 4.6 the lockfile is being created in /var/lock instead; so when shutting down, "xendomains stop" was never called.
The /var/lock/subsys thing is apparently a RHEL thing; other systems seem to call "${service} stop" unconditionally.
Thanks again for reporting this, -George
George,
Thanks a lot for your investigation and feedback. I confirm it works as expected after the change you proposed.
Cheers, -- exvito
On Mon, Apr 11, 2016 at 6:06 PM, exvito here ex.vitorino@gmail.com wrote:
George,
Thanks a lot for your investigation and feedback. I confirm it works as expected after the change you proposed.
The patches have been backported and are available in 4.6.1-9 from virt-xen-testing. Please test it and report any problems here.
-George
On Tue, May 24, 2016 at 3:23 PM, George Dunlap dunlapg@umich.edu wrote:
The patches have been backported and are available in 4.6.1-9 from virt-xen-testing. Please test it and report any problems here.
Thanks for the notice. It doesn't look like the fix went in. See below:
# yum repolist | grep xen centos-virt-xen-testing CentOS-6 - xen - testing 102 # rpm -qf /etc/xen/scripts/hotplugpath.sh xen-runtime-4.6.1-9.el6.x86_64 # rpm -V xen-runtime
Confirmed installation of 4.6.1-9 from virt-xen-testing. Confirmed files "as packaged".
# grep LOCK /etc/xen/scripts/hotplugpath.sh XEN_LOCK_DIR="/var/lock"
The XEN_LOCK_DIR still points to the wrong directory. Am I missing something?
On Tue, May 24, 2016 at 10:20 PM, exvito here ex.vitorino@gmail.com wrote:
On Tue, May 24, 2016 at 3:23 PM, George Dunlap dunlapg@umich.edu wrote:
The patches have been backported and are available in 4.6.1-9 from virt-xen-testing. Please test it and report any problems here.
Thanks for the notice. It doesn't look like the fix went in. See below:
# yum repolist | grep xen centos-virt-xen-testing CentOS-6 - xen - testing 102 # rpm -qf /etc/xen/scripts/hotplugpath.sh xen-runtime-4.6.1-9.el6.x86_64 # rpm -V xen-runtime
Confirmed installation of 4.6.1-9 from virt-xen-testing. Confirmed files "as packaged".
# grep LOCK /etc/xen/scripts/hotplugpath.sh XEN_LOCK_DIR="/var/lock"
The XEN_LOCK_DIR still points to the wrong directory. Am I missing something?
Yes, you are missing someting. :-) XEN_LOCK_DIR is used for a number of different things; but *only* "I am alive" init script files should live in /var/lock/subsys. (Presumably that's why it's a separate directory to begin with.) So the "proper" fix is to have the xendomains script check for the existence of $XEN_LOCK_DIR/subsys and use it if available. See my original patch submission for more details [1].
So, I've tested the packages and they work for me, let me know if you have any problems. :-)
-George
Yes, you are missing someting. :-) XEN_LOCK_DIR is used for a number of different things; but *only* "I am alive" init script files should live in /var/lock/subsys. (Presumably that's why it's a separate directory to begin with.) So the "proper" fix is to have the xendomains script check for the existence of $XEN_LOCK_DIR/subsys and use it if available. See my original patch submission for more details [1].
So, I've tested the packages and they work for me, let me know if you have any problems. :-)
Oh, good! Thanks for the explanation.
Updated to 4.6.1-9 testing, and things are looking good: in particular, I confirm domUs were saved+restored on dom0 restart.
Thanks again,