On Wed, January 2, 2013 12:51, SilverTip257 wrote:
On Wed, Jan 2, 2013 at 9:41 AM, James B. Byrne byrnejb@harte-lyne.cawrote:
I ran into this situation several times whilst testing KVM and the lessons I learned from the experiences can be summarized as:
- Never allow the kvm hypervisior to handle guests during a host
shutdown. Use 'virsh shutdown' on each of the guests first and then shutdown the host. Use autostart to restart guests on a host's reboot. Write a script to process 'virsh list' to feed active domains to 'virsh shutdown' if automation is required and link that to /etc/rc0.d/K10<whatever>.
@James: Can you specifically cite why you manually power down each node? Have you tried tweaking your libvirt settings in the config file I noted in my earlier response to Robert?
Two reasons. First, I am minimally familiar with kvm. The niceties of the options for it is beyond my kin for the nonce. Second, libvirt does not always work. I have had guests refuse to either suspend or shutdown from an automatic request to do so. When shutdown is done manually one discovers right away that there is a problem and which guest is causing it.
Set up a central NTP server and have your hosts (and not just VMs) connect to it. It could be the VM host, but doesn't need to be. Distribute the load to your NTP server and off of the public NTP pool by running an NTP server for your servers to poll [0] ... it's a good practice and everybody is happy.
I do that as well. However, I run one on each host just to serve its own guests and configure the host to run off our central ntp server.
- On each guest have a cron job that checks for ntpd at regular
intervals which reports failures and restarts the time service as necessary. We use: JOBNAME="Check ntpd status and restart if required" ; \ ntpstat > /dev/null && \ if [[ $? -gt 0 ]]; then /sbin/service ntpd start; fi
Why not configure the ntpd daemon and stick with that? It does update on its own [1]. And ntpstat prints out the interval, which matches the one mentioned at [1]. I don't believe the ntpstat script/job is necessary (I've never had to do more than set ntpd to run after configuring the servers it should poll).
You misunderstand the purpose of the job. Netstat checks to see if the daemon is actually running. If it is not then netstat returns a non-zero exit code. If the ntpstat exit code is not zero then the service script is invoked to restart it. Additionally, netstat writes out to stderr that it could not find the daemon which gets emailed to support. I probably should have used [[ ! $? -eq 0 ]] but what I have written does work.
We found ntpd just stoped on some guests upon occasion without any visible trace of a cause. Not frequently but when it did happen it was a nuisance to detect before clock drift on the guest caused some failure or other. This job detects these occurrences and self corrects.
These are all CentOS-6.3 hosts and guests.
On Wed, Jan 2, 2013 at 7:27 PM, James B. Byrne byrnejb@harte-lyne.cawrote:
@James: Can you specifically cite why you manually power down each node? Have you tried tweaking your libvirt settings in the config file I noted in my earlier response to Robert?
Two reasons. First, I am minimally familiar with kvm. The niceties of the options for it is beyond my kin for the nonce. Second, libvirt does not always work. I have had guests refuse to either suspend or shutdown from an automatic request to do so. When shutdown is done manually one discovers right away that there is a problem and which guest is causing it.
I hear ya. I've never had libvirt not work, but I've had an issue or two with VMs shortly after they were created (not production at that point).
Set up a central NTP server and have your hosts (and not just VMs) connect to it. It could be the VM host, but doesn't need to be. Distribute the load to your NTP server and off of the public NTP pool by running an NTP server for your servers to poll [0] ... it's a good practice and everybody is happy.
I do that as well. However, I run one on each host just to serve its own guests and configure the host to run off our central ntp server.
- On each guest have a cron job that checks for ntpd at regular
intervals which reports failures and restarts the time service as necessary. We use: JOBNAME="Check ntpd status and restart if required" ; \ ntpstat > /dev/null && \ if [[ $? -gt 0 ]]; then /sbin/service ntpd start; fi
Why not configure the ntpd daemon and stick with that? It does update on its own [1]. And ntpstat prints out the interval, which matches the one mentioned at [1]. I don't believe the ntpstat script/job is necessary (I've never had to do more than set ntpd to run after configuring the servers it should poll).
You misunderstand the purpose of the job. Netstat checks to see if the daemon is actually running. If it is not then netstat returns a non-zero exit code. If the ntpstat exit code is not zero then the service script is invoked to restart it. Additionally, netstat writes out to stderr that it could not find the daemon which gets emailed to support. I probably should have used [[ ! $? -eq 0 ]] but what I have written does work.
We found ntpd just stoped on some guests upon occasion without any visible trace of a cause. Not frequently but when it did happen it was a nuisance to detect before clock drift on the guest caused some failure or other. This job detects these occurrences and self corrects.
I'll have to check my hosts tomorrow and see if there's any drift. Thanks for the explanation.
These are all CentOS-6.3 hosts and guests.
-- *** E-Mail is NOT a SECURE channel *** James B. Byrne mailto:ByrneJB@Harte-Lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
On Jan 2, 2013, at 19:27, "James B. Byrne" byrnejb@harte-lyne.ca wrote:
I do that as well. However, I run one on each host just to serve its own guests and configure the host to run off our central ntp server.
Unfortunately, before our upstream vendor's OS release 6, ntp.conf listed several loopback addresses by default. These allowed a confused ntpd to basically marry its siblings and eventually crossbreed itself to a fairly stange state. But it will report ntpd as active, which is why the Nagios check "chek_ntp_)time" actually compares the time to a known good upstream NTP service.
- On each guest have a cron job that checks for ntpd at regular
intervals which reports failures and restarts the time service as necessary. We use: JOBNAME="Check ntpd status and restart if required" ; \ ntpstat > /dev/null && \ if [[ $? -gt 0 ]]; then /sbin/service ntpd start; fi
Why not configure the ntpd daemon and stick with that? It does update on its own [1]. And ntpstat prints out the interval, which matches the one mentioned at [1]. I don't believe the ntpstat script/job is necessary (I've never had to do more than set ntpd to run after configuring the servers it should poll).
See above. The 'check_ntp_time' tool is much more flexible and complete. itten does work. It's part of the "nagios-plugions-ntp" package, available from EPEL and RPMforge.
On Thu, Jan 3, 2013 at 9:01 PM, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Jan 2, 2013, at 19:27, "James B. Byrne" byrnejb@harte-lyne.ca wrote:
I do that as well. However, I run one on each host just to serve its own guests and configure the host to run off our central ntp server.
Unfortunately, before our upstream vendor's OS release 6, ntp.conf listed several loopback addresses by default. These allowed a confused ntpd to basically marry its siblings and eventually crossbreed itself to a fairly stange state. But it will report ntpd as active, which is why the Nagios check "chek_ntp_)time" actually compares the time to a known good upstream NTP service.
Dumb question: why not use the vm host's clock instead? I am coming from libvirt but I would assume (bad idea I know) that if libvirt can see the host's clock in the client, so can kvm. If that is the case, something like
hwclock -s -u
should keep them in sync. I wrote a little cron job (see http://unixwars.blogspot.com/2013/01/restoring-time-on-sleeping-linux-vms.ht...) to compare the client time with the host. If the drift is large enough, sync them back. The reason for the cron job is so it works after client wakes up from a slumber. Just throwing an idea out there.
On a side note, yes I did also write another script using ntpq and ntpdate to adjust time using ntp. I guess each solution has its merits. =)
- On each guest have a cron job that checks for ntpd at regular
intervals which reports failures and restarts the time service as necessary. We use: JOBNAME="Check ntpd status and restart if required" ; \ ntpstat > /dev/null && \ if [[ $? -gt 0 ]]; then /sbin/service ntpd start; fi
Why not configure the ntpd daemon and stick with that? It does update on its own [1]. And ntpstat prints out the interval, which matches the one mentioned at [1]. I don't believe the ntpstat script/job is necessary (I've never had to do more than set ntpd to run after configuring the servers it should poll).
See above. The 'check_ntp_time' tool is much more flexible and complete. itten does work. It's part of the "nagios-plugions-ntp" package, available from EPEL and RPMforge. _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
12.02.2013, 20:50, "Mauricio Tavares" raubvogel@gmail.com:
On Thu, Jan 3, 2013 at 9:01 PM, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Jan 2, 2013, at 19:27, "James B. Byrne" byrnejb@harte-lyne.ca wrote:
I do that as well. However, I run one on each host just to serve its own guests and configure the host to run off our central ntp server.
Unfortunately, before our upstream vendor's OS release 6, ntp.conf listed several loopback addresses by default. These allowed a confused ntpd to basically marry its siblings and eventually crossbreed itself to a fairly stange state. But it will report ntpd as active, which is why the Nagios check "chek_ntp_)time" actually compares the time to a known good upstream NTP service.
Dumb question: why not use the vm host's clock instead? I am coming from libvirt but I would assume (bad idea I know) that if libvirt can see the host's clock in the client, so can kvm. If that is the case, something like
hwclock -s -u
should keep them in sync. I wrote a little cron job (see http://unixwars.blogspot.com/2013/01/restoring-time-on-sleeping-linux-vms.ht...) to compare the client time with the host. If the drift is large enough, sync them back. The reason for the cron job is so it works after client wakes up from a slumber. Just throwing an idea out there.
On a side note, yes I did also write another script using ntpq and ntpdate to adjust time using ntp. I guess each solution has its merits. =)
4. On each guest have a cron job that checks for ntpd at regular intervals which reports failures and restarts the time service as necessary. We use: JOBNAME="Check ntpd status and restart if required" ; \ ntpstat > /dev/null && \ if [[ $? -gt 0 ]]; then /sbin/service ntpd start; fi
Why not configure the ntpd daemon and stick with that? It does update on its own [1]. And ntpstat prints out the interval, which matches the one mentioned at [1]. I don't believe the ntpstat script/job is necessary (I've never had to do more than set ntpd to run after configuring the servers it should poll).
See above. The 'check_ntp_time' tool is much more flexible and complete. itten does work. It's part of the "nagios-plugions-ntp" package, available from EPEL and RPMforge. _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Hi. To get the same at all WM. I installed and configured ntp-server.
-
James B. Byrne -
Mauricio Tavares -
Nico Kadel-Garcia -
SilverTip257 -
Van