On Wed, January 2, 2013 03:53, Robert Dinse wrote:
Friday, I moved our servers to a new co-lo facility and ran intoan interesting problem with virtual machines.
I did an orderly shutdown of the CentOS 6.3 host, and it inturn suspends all the guests. It took about an hour and a half to move and fire up the host.
The guests, being suspended, were then an hour and a halfbehind and it seems ntpd does not want to correct more than 1000 seconds of error so it would not automatically adjust the clocks.
I tried the -g argument which is supposed to override the1000 second limit but it did not. I ended up having to manually set the clocks close enough for ntpd to correct.
Since there is no hardware clock for the virtual machinesto use when they boot, it seems that shutdown and reboot of the virtual machines probably would not have avoided this.
Any suggestions for addressing this particular scenerio otherthan having to manually set a bunch of clocks?
I ran into this situation several times whilst testing KVM and the lessons I learned from the experiences can be summarized as:
1. Never allow the kvm hypervisior to handle guests during a host shutdown. Use 'virsh shutdown' on each of the guests first and then shutdown the host. Use autostart to restart guests on a host's reboot. Write a script to process 'virsh list' to feed active domains to 'virsh shutdown' if automation is required and link that to /etc/rc0.d/K10<whatever>.
2. In the situation where a kvm guest pause and restore sequence leads to an excessive disconnect between guest time and wall time use ntpd -q to hard set the time. From the guest's point of view you are always going ahead in time in the case of a pause and resume so this is not likely to ever cause a problem. But, having written that down, it probably will at some point.
3. Run ntpd on the host system and have its guests configured to only use that time server source.
4. On each guest have a cron job that checks for ntpd at regular intervals which reports failures and restarts the time service as necessary. We use: JOBNAME="Check ntpd status and restart if required" ; \ ntpstat > /dev/null && \ if [[ $? -gt 0 ]]; then /sbin/service ntpd start; fi
On Wed, Jan 2, 2013 at 9:41 AM, James B. Byrne byrnejb@harte-lyne.cawrote:
I ran into this situation several times whilst testing KVM and the lessons I learned from the experiences can be summarized as:
- Never allow the kvm hypervisior to handle guests during a host
shutdown. Use 'virsh shutdown' on each of the guests first and then shutdown the host. Use autostart to restart guests on a host's reboot. Write a script to process 'virsh list' to feed active domains to 'virsh shutdown' if automation is required and link that to /etc/rc0.d/K10<whatever>.
@James: Can you specifically cite why you manually power down each node? Have you tried tweaking your libvirt settings in the config file I noted in my earlier response to Robert?
Oh and another note, you can set libvirt so that it _only_ starts the machines that were running when the host machine was issued a shutdown. There are advantages and disadvantages to both (auto-shutdown by host and manually doing it) as there is with anything.
- In the situation where a kvm guest pause and restore sequence
leads to an excessive disconnect between guest time and wall time use ntpd -q to hard set the time. From the guest's point of view you are always going ahead in time in the case of a pause and resume so this is not likely to ever cause a problem. But, having written that down, it probably will at some point.
- Run ntpd on the host system and have its guests configured to only
use that time server source.
Set up a central NTP server and have your hosts (and not just VMs) connect to it. It could be the VM host, but doesn't need to be. Distribute the load to your NTP server and off of the public NTP pool by running an NTP server for your servers to poll [0] ... it's a good practice and everybody is happy.
[0] http://support.ntp.org/bin/view/Support/DesigningYourNTPNetwork#Section_5.7.
- On each guest have a cron job that checks for ntpd at regular
intervals which reports failures and restarts the time service as necessary. We use: JOBNAME="Check ntpd status and restart if required" ; \ ntpstat > /dev/null && \ if [[ $? -gt 0 ]]; then /sbin/service ntpd start; fi
Why not configure the ntpd daemon and stick with that? It does update on its own [1]. And ntpstat prints out the interval, which matches the one mentioned at [1]. I don't believe the ntpstat script/job is necessary (I've never had to do more than set ntpd to run after configuring the servers it should poll).
~]$ ntpstat synchronised to NTP server (x.x.x.x) at stratum 3 time correct to within 67 ms polling server every 1024 s
[1] http://www.ntp.org/ntpfaq/NTP-s-algo.htm#Q-ALGO-POLL-BEST
-- *** E-Mail is NOT a SECURE channel *** James B. Byrne mailto:ByrneJB@Harte-Lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
I thank everyone for their input. I prefer to have the guests suspend rather than shutdown because many of my customers start things up manually and get annoyed when they come back to find them not running. Most of the time it is for a simple reboot of the host to make a new kernel or some other update active. This particular time though I had to move all the equipment to a new co-location facility because the old provider had become far too greedy.
So I guess rdate -s is probably the best solution.
-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_- Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting. Knowledgeable human assistance, not telephone trees or script readers. See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.
-
James B. Byrne -
Robert Dinse -
SilverTip257