Hi friends,
I am running Nagios 2.7-1 on Centos 5.0 32-bit hosted on Vmware ESX 4.0. The issue I am seeing on the server is sometimes nagios is showing the below messages in /var/log/messages and as the system time gets changed some false alarms gets generated. I searched it on the google but I am not able to find the correct solution. I even posted on the nagios forum and they asked me to see elsewhere why the server shitfs so much before looking at nagios.
Nov 21 20:37:12 linuxmonitoring nagios: Warning: A system time change of 4398 seconds (forwards in time) has been detected. Compensating... Nov 21 19:23:54 linuxmonitoring nagios: Warning: A system time change of 4398 seconds (backwards in time) has been detected. Compensating..
Earlier this server was syncing time through ntp daemon and below is the ntp.conf file. Now I have set a cronjob which sync the time with the ntp server every 5 minutes but still the problem persist.
ntp.conf file
restrict default ignore restrict 127.0.0.1 driftfile /var/lib/ntp/drift broadcastdelay 0.008 #authenticate yes keys /etc/ntp/keys restrict 172.16.6.3 nomodify notrap noquery server 172.16.6.3 restrict 172.16.6.2 nomodify notrap noquery server 172.16.6.2
Please see the output of hwclock and date at the same time. hwclock Sat 21 Nov 2009 08:19:02 PM IST -0.496922 seconds
date Sat Nov 21 20:19:55 IST 2009
Please advice what I need to do to fix this error.
Regards
Ankush
On Sun, Nov 22, 2009 at 12:19 AM, ankush grover ankushcentos@gmail.com wrote:
Hi friends,
I am running Nagios 2.7-1 on Centos 5.0 32-bit hosted on Vmware ESX 4.0. The issue I am seeing on the server is sometimes nagios is showing the below messages in /var/log/messages and as the system time gets changed some false alarms gets generated. I searched it on the google but I am not able to find the correct solution. I even posted on the nagios forum and they asked me to see elsewhere why the server shitfs so much before looking at nagios.
Nov 21 20:37:12 linuxmonitoring nagios: Warning: A system time change of 4398 seconds (forwards in time) has been detected. Compensating... Nov 21 19:23:54 linuxmonitoring nagios: Warning: A system time change of 4398 seconds (backwards in time) has been detected. Compensating..
Earlier this server was syncing time through ntp daemon and below is the ntp.conf file. Now I have set a cronjob which sync the time with the ntp server every 5 minutes but still the problem persist.
ntp.conf file
restrict default ignore restrict 127.0.0.1 driftfile /var/lib/ntp/drift broadcastdelay 0.008 #authenticate yes keys /etc/ntp/keys restrict 172.16.6.3 nomodify notrap noquery server 172.16.6.3 restrict 172.16.6.2 nomodify notrap noquery server 172.16.6.2
Please see the output of hwclock and date at the same time. hwclock Sat 21 Nov 2009 08:19:02 PM IST -0.496922 seconds
date Sat Nov 21 20:19:55 IST 2009
Please advice what I need to do to fix this error.
Regards Ankush
Because you are using VMware you must become very familiar with how timekeeping works on computer systems. VMware has some helpful guides on this: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd... http://www.vmware.com/pdf/vmware_timekeeping.pdf
ankush grover wrote:
Hi friends,
I am running Nagios 2.7-1 on Centos 5.0 32-bit hosted on Vmware ESX 4.0. The issue I am seeing on the server is sometimes nagios is showing the below messages in /var/log/messages and as the system time gets changed some false alarms gets generated. I searched it on the google but I am not able to find the correct solution. I even posted on the nagios forum and they asked me to see elsewhere why the server shitfs so much before looking at nagios.
Well, VMWare ESX sometime causes time jumps ;)
Possibly your service provider is overbooking cpu time of that ESX host ..
-- Eero
ankush grover schrieb:
Hi friends,
I am running Nagios 2.7-1 on Centos 5.0 32-bit hosted on Vmware ESX 4.0. The issue I am seeing on the server is sometimes nagios is showing the below messages in /var/log/messages and as the system time gets changed some false alarms gets generated. I searched it on the google but I am not able to find the correct solution. I even posted on the nagios forum and they asked me to see elsewhere why the server shitfs so much before looking at nagios.
Nov 21 20:37:12 linuxmonitoring nagios: Warning: A system time change of 4398 seconds (forwards in time) has been detected. Compensating... Nov 21 19:23:54 linuxmonitoring nagios: Warning: A system time change of 4398 seconds (backwards in time) has been detected. Compensating..
Earlier this server was syncing time through ntp daemon and below is the ntp.conf file. Now I have set a cronjob which sync the time with the ntp server every 5 minutes but still the problem persist.
ntp.conf file
restrict default ignore restrict 127.0.0.1 driftfile /var/lib/ntp/drift broadcastdelay 0.008 #authenticate yes keys /etc/ntp/keys restrict 172.16.6.3 nomodify notrap noquery server 172.16.6.3 restrict 172.16.6.2 nomodify notrap noquery server 172.16.6.2
Please see the output of hwclock and date at the same time. hwclock Sat 21 Nov 2009 08:19:02 PM IST -0.496922 seconds
date Sat Nov 21 20:19:55 IST 2009
Please advice what I need to do to fix this error.
Regards
Ankush
Be sure that only 1 mechanism for syncing time is active for your VM: either NTP client configuration through ntpd running (like you have) OR through the VMware tools. Both being active will result in trouble.
VMware recommends to use time syncronisation through NTP within the VM.
To debug your time sources in NTP make use of "ntpq" and/or "ntpdc".
Regards
Alexander
ankush grover wrote:
Earlier this server was syncing time through ntp daemon and below is the ntp.conf file. Now I have set a cronjob which sync the time with
Best not to run NTP inside a ESX VM. I've never gotten NTP to sync inside of VMware outside of a kernel with VMI enabled (no versions of RHEL support VMI at this time as far as I know).
What I do for my ~40 ESX/ESXi hosts: - Have your ESX hosts sync to a good NTP server - Make sure vmware tools is installed and running correctly (/etc/init.d/vmware-tools status) - Enable time sync for your guest, either via the UI or via this command in the guest(I have this command run in cron every 5 minutes as I have seen for some reason time sync turn itself off: /usr/sbin/vmware-guestd --cmd "vmx.set_option synctime 0 1" - On top of all of that I have another cron set to run ntpdate every 5 minutes against a local NTP server: /usr/sbin/ntpdate `cat /etc/ntp/step-tickers | grep -v #`
For providing NTP services themselves, currently I run 3 VMs at each site with Fedora 8 with VMI enabled for the guest VM (the kernel in FC8 supports VMI, I suspect newer Fedoras work fine too I just have no reason to change right now). And I have these FC8 VMs sync to internet hosts(mainly time.nist.gov) so my internal ESX and other systems can sync against them(they are load balanced behind a F5 BigIP).
from FC8 kernel log: VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0 vmi: registering clock event vmi-timer. mult=9483317 shift=22 Booting paravirtualized kernel on vmi vmi: registering clock source khz=2260999 Time: vmi-timer clocksource has been installed.
I currently run roughly 400 VMs this way and don't have any noticeable time-related issues.
nate
On Sun, Nov 22, 2009 at 7:08 AM, nate centos@linuxpowered.net wrote:
ankush grover wrote:
Earlier this server was syncing time through ntp daemon and below is the ntp.conf file. Now I have set a cronjob which sync the time with
Best not to run NTP inside a ESX VM. I've never gotten NTP to sync inside of VMware outside of a kernel with VMI enabled (no versions of RHEL support VMI at this time as far as I know).
As quoted by Brian earlier in this thread, the VMware knowledge base offers their "best practices for Linux guests" in :
http://kb.vmware.com/kb/1006427
for VMware products including ESX and ESXi. According to their current recommendations, " In all cases use NTP instead of VMware Tools periodic time synchronization."
The above KB articles gets updated from time to time, so it is a good idea to check it back occasionally.
Akemi
Akemi Yagi wrote:
for VMware products including ESX and ESXi. According to their current recommendations, " In all cases use NTP instead of VMware Tools periodic time synchronization."
I've been using vmware for 10 years, and I've never, ever ever gotten NTP to hold sync inside of a VM outside of using a VMI enabled kernel.
nate
nate wrote:
Akemi Yagi wrote:
for VMware products including ESX and ESXi. According to their current recommendations, " In all cases use NTP instead of VMware Tools periodic time synchronization."
I've been using vmware for 10 years, and I've never, ever ever gotten NTP to hold sync inside of a VM outside of using a VMI enabled kernel.
I've got more than 20 VMs spread over 5 machines (VMware Server 2.x), both 32 and 64 bit (hosts and VMs) that hold time perfectly using NTP.
1) Make *SURE* that 'cpuspeed' and any BIOS 'power saving' modes are disabled on your host and your VMs. Nothing screws timekeeping like having the CPU speed vary.
2) Use 'divider=10' on your grub kernel boot lines for your virtual machines.
Benjamin Franz schrieb:
nate wrote:
Akemi Yagi wrote:
for VMware products including ESX and ESXi. According to their current recommendations, " In all cases use NTP instead of VMware Tools periodic time synchronization."
I've been using vmware for 10 years, and I've never, ever ever gotten NTP to hold sync inside of a VM outside of using a VMI enabled kernel.
I've got more than 20 VMs spread over 5 machines (VMware Server 2.x), both 32 and 64 bit (hosts and VMs) that hold time perfectly using NTP.
- Make *SURE* that 'cpuspeed' and any BIOS 'power saving' modes are
disabled on your host and your VMs. Nothing screws timekeeping like having the CPU speed vary.
- Use 'divider=10' on your grub kernel boot lines for your virtual
machines.
That kernel parameter information is out of date with CentOS 5.4:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd...
As Akemi said, users of VMware virtualization products (bare metal hypervisor products like ESX/ESXi and the other solutions) should closely follow the KB article recommendations which are frequently updated.
Alexander
nate wrote:
ankush grover wrote:
Earlier this server was syncing time through ntp daemon and below is the ntp.conf file. Now I have set a cronjob which sync the time with
Best not to run NTP inside a ESX VM. I've never gotten NTP to sync inside of VMware outside of a kernel with VMI enabled (no versions of RHEL support VMI at this time as far as I know).
What I do for my ~40 ESX/ESXi hosts:
- Have your ESX hosts sync to a good NTP server
- Make sure vmware tools is installed and running correctly (/etc/init.d/vmware-tools status)
- Enable time sync for your guest, either via the UI or via this command in the guest(I have this command run in cron every 5 minutes as I have seen for some reason time sync turn itself off: /usr/sbin/vmware-guestd --cmd "vmx.set_option synctime 0 1"
- On top of all of that I have another cron set to run ntpdate every 5 minutes against a local NTP server: /usr/sbin/ntpdate `cat /etc/ntp/step-tickers | grep -v #`
For providing NTP services themselves, currently I run 3 VMs at each site with Fedora 8 with VMI enabled for the guest VM (the kernel in FC8 supports VMI, I suspect newer Fedoras work fine too I just have no reason to change right now). And I have these FC8 VMs sync to internet hosts(mainly time.nist.gov) so my internal ESX and other systems can sync against them(they are load balanced behind a F5 BigIP).
from FC8 kernel log: VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0 vmi: registering clock event vmi-timer. mult=9483317 shift=22 Booting paravirtualized kernel on vmi vmi: registering clock source khz=2260999 Time: vmi-timer clocksource has been installed.
I currently run roughly 400 VMs this way and don't have any noticeable time-related issues.
nate
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
The OP should also reference this document http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd...
On Sun, Nov 22, 2009 at 12:19 PM, Clint Dilks clintd@scms.waikato.ac.nz wrote:
The OP should also reference this document http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd...
This is the 3rd time that KB article was quoted in this thread... :-D
Akemi
-
Akemi Yagi -
Alexander Dalloz -
ankush grover -
Benjamin Franz -
Brian Mathis -
Clint Dilks -
Eero Volotinen -
nate