Hello All,
Consider a CentOS-5.1 Xen server (2.6.18-53.1.4.el5xen) hosting two domains running CentOS-5.1 (2.6.18-53.1.4.el5). One domain has a fairly accurate clock, the other domain has a clock that gains ungodly amounts of time, roughly one minute every two or three minutes. For a fix, one suggestion is to run this command in DomU:
echo 1 > /proc/sys/xen/independent_wallclock
This didn't change anything. As an experiment, I wrote a script to call ntpd -q, sleep 60, and repeat indefinitely. Here are a couple of snippets of output:
goodclock# ksh ./xenclockdrift ntpd: time slew +0.001211s ntpd: time slew +0.001200s ntpd: time slew +0.001855s ntpd: time slew +0.001532s ntpd: time slew +0.001603s ntpd: time slew +0.001320s ntpd: time slew +0.001931s
badclock# ksh ./xenclockdrift ntpd: time slew -0.000193s ntpd: time set -57.356377s ntpd: time slew +0.002352s ntpd: time slew +0.003018s ntpd: time set -57.417488s ntpd: time slew +0.012089s ntpd: time slew -0.000985s
These domains are fully virtualized and set up identically, except "badclock" is allocated two processors versus one processor for "goodclock". DomU's clock is running normally.
Anyone know what's going or know how to fix it?
Jack
Jack Bailey wrote:
These domains are fully virtualized and set up identically, except "badclock" is allocated two processors versus one processor for "goodclock". DomU's clock is running normally.
Anyone know what's going or know how to fix it?
This is a known issue that has come up on this list a lot.
For C5.1 see the first known issue:
http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.1/
Rick
On Jan 8, 2008 4:51 AM, Rick Barnes linux@sitevision.com wrote:
Jack Bailey wrote:
These domains are fully virtualized and set up identically, except "badclock" is allocated two processors versus one processor for "goodclock". DomU's clock is running normally.
Anyone know what's going or know how to fix it?
This is a known issue that has come up on this list a lot.
For C5.1 see the first known issue:
http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.1/
Rick
Please note that the clock rate issue in that description applies to non-xen kernels. xen kernels are set to 250Hz by default.
Akemi
Akemi Yagi wrote:
On Jan 8, 2008 4:51 AM, Rick Barnes linux@sitevision.com wrote:
Jack Bailey wrote:
These domains are fully virtualized and set up identically, except "badclock" is allocated two processors versus one processor for "goodclock". DomU's clock is running normally.
Anyone know what's going or know how to fix it?
This is a known issue that has come up on this list a lot.
For C5.1 see the first known issue:
http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.1/
Rick
Please note that the clock rate issue in that description applies to non-xen kernels. xen kernels are set to 250Hz by default.
"With this option you can reduce the clock rate from the default of 1000HZ to 100HZ which is desirable in a virtual machine."
If it does not apply to xen then this should be made more clear.
Rick
Rick Barnes wrote:
Akemi Yagi wrote:
On Jan 8, 2008 4:51 AM, Rick Barnes linux@sitevision.com wrote:
Jack Bailey wrote:
These domains are fully virtualized and set up identically, except "badclock" is allocated two processors versus one processor for "goodclock". DomU's clock is running normally.
Anyone know what's going or know how to fix it?
This is a known issue that has come up on this list a lot.
For C5.1 see the first known issue:
http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.1/
Rick
Please note that the clock rate issue in that description applies to non-xen kernels. xen kernels are set to 250Hz by default.
"With this option you can reduce the clock rate from the default of 1000HZ to 100HZ which is desirable in a virtual machine."
If it does not apply to xen then this should be made more clear.
The first note is to address the fact the "divider=" and not "tick_divider=" should be used ... though the actual usage is detailed in the upstream release notes:
http://www.centos.org/docs/5/html/release-notes/as-x86/RELEASE-NOTES-U1-x86-...
This does say (well sort of) that it can not be used on Xen:
"Note that the virtualized kernel does not support multiple timer rates on guests. dom0 uses a fixed timing rate set across all guests; this reduces the load that multiple tick rates could cause."
Regardless ... divider= is not for xen kernels, only normal kernels.
Thanks, Johnny Hughes
On Jan 8, 2008 5:05 AM, Rick Barnes linux@sitevision.com wrote:
Akemi Yagi wrote:
On Jan 8, 2008 4:51 AM, Rick Barnes linux@sitevision.com wrote:
Jack Bailey wrote:
These domains are fully virtualized and set up identically, except "badclock" is allocated two processors versus one processor for "goodclock". DomU's clock is running normally.
Anyone know what's going or know how to fix it?
This is a known issue that has come up on this list a lot.
For C5.1 see the first known issue:
http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.1/
Rick
Please note that the clock rate issue in that description applies to non-xen kernels. xen kernels are set to 250Hz by default.
"With this option you can reduce the clock rate from the default of 1000HZ to 100HZ which is desirable in a virtual machine."
If it does not apply to xen then this should be made more clear.
Rick
I think it is a good idea to add a note about xen kernels. But this is noted in the upstream Release Notes where the tick divider option is mentioned:
"Note that the virtualized kernel does not support multiple timer rates on guests. dom0 uses a fixed timing rate set across all guests; this reduces the load that multiple tick rates could cause."
http://www.centos.org/docs/5/html/release-notes/as-x86/RELEASE-NOTES-U1-x86-... (scroll down to tick_divider in the Feature Updates section)
Akemi
From: Jack Bailey jack@internetguy.net Date: Jan 8, 2008 3:17 AM To: centos@centos.org
Hello All,
Consider a CentOS-5.1 Xen server (2.6.18-53.1.4.el5xen) hosting two domains running CentOS-5.1 (2.6.18-53.1.4.el5). One domain has a fairly accurate clock, the other domain has a clock that gains ungodly amounts of time, roughly one minute every two or three minutes. For a fix, one suggestion is to run this command in DomU:
echo 1 > /proc/sys/xen/independent_wallclock
This didn't change anything. As an experiment, I wrote a script to call ntpd -q, sleep 60, and repeat indefinitely. Here are a couple of snippets of output:
goodclock# ksh ./xenclockdrift ntpd: time slew +0.001211s ntpd: time slew +0.001200s ntpd: time slew +0.001855s ntpd: time slew +0.001532s ntpd: time slew +0.001603s ntpd: time slew +0.001320s ntpd: time slew +0.001931s
badclock# ksh ./xenclockdrift ntpd: time slew -0.000193s ntpd: time set -57.356377s ntpd: time slew +0.002352s ntpd: time slew +0.003018s ntpd: time set -57.417488s ntpd: time slew +0.012089s ntpd: time slew -0.000985s
These domains are fully virtualized and set up identically, except "badclock" is allocated two processors versus one processor for "goodclock". DomU's clock is running normally.
Anyone know what's going or know how to fix it? Jack
I'm not sure with Xen, but on VMWare one should not be using NTP at all. Time syncing should be done with the vmware tools and the host, with NTP only running on the host (not the guests). Using NTP on a vmware system will result in similar behavior to what you are seeing here.
badclock# ksh ./xenclockdrift ntpd: time slew -0.000193s ntpd: time set -57.356377s ntpd: time slew +0.002352s ntpd: time slew +0.003018s ntpd: time set -57.417488s ntpd: time slew +0.012089s ntpd: time slew -0.000985s
These domains are fully virtualized and set up identically, except "badclock" is allocated two processors versus one processor for "goodclock". DomU's clock is running normally.
I'm not sure with Xen, but on VMWare one should not be using NTP at all. Time syncing should be done with the vmware tools and the host, with NTP only running on the host (not the guests). Using NTP on a vmware system will result in similar behavior to what you are seeing here.
ntpd is not running. I'm using it here manually to show the clock and how it periodically jumps ahead by 57 seconds.
Jack
On 08/01/2008 15:15, Brian Mathis wrote:
From: Jack Bailey jack@internetguy.net Date: Jan 8, 2008 3:17 AM To: centos@centos.org
Hello All,
Consider a CentOS-5.1 Xen server (2.6.18-53.1.4.el5xen) hosting two domains running CentOS-5.1 (2.6.18-53.1.4.el5). One domain has a fairly accurate clock, the other domain has a clock that gains ungodly amounts of time, roughly one minute every two or three minutes. For a fix, one suggestion is to run this command in DomU:
echo 1 > /proc/sys/xen/independent_wallclock
This didn't change anything. As an experiment, I wrote a script to call ntpd -q, sleep 60, and repeat indefinitely. Here are a couple of snippets of output:
goodclock# ksh ./xenclockdrift ntpd: time slew +0.001211s ntpd: time slew +0.001200s ntpd: time slew +0.001855s ntpd: time slew +0.001532s ntpd: time slew +0.001603s ntpd: time slew +0.001320s ntpd: time slew +0.001931s
badclock# ksh ./xenclockdrift ntpd: time slew -0.000193s ntpd: time set -57.356377s ntpd: time slew +0.002352s ntpd: time slew +0.003018s ntpd: time set -57.417488s ntpd: time slew +0.012089s ntpd: time slew -0.000985s
These domains are fully virtualized and set up identically, except "badclock" is allocated two processors versus one processor for "goodclock". DomU's clock is running normally.
Anyone know what's going or know how to fix it? Jack
I'm not sure with Xen, but on VMWare one should not be using NTP at all. Time syncing should be done with the vmware tools and the host, with NTP only running on the host (not the guests). Using NTP on a vmware system will result in similar behavior to what you are seeing here.
At least, that is the theory. Empirically, using the vmware tools time sync feature will push a slow VM's clock forwards, but it won't push a fast clock backwards. I'm yet to see a "best practice" for ensuring proper time synchronisation within VMware VMs, but for now, NTP seems the best option.
cheers Luke
On Jan 9, 2008 7:41 AM, Luke Dudney listmail@lukedudney.com wrote:
At least, that is the theory. Empirically, using the vmware tools time sync feature will push a slow VM's clock forwards, but it won't push a fast clock backwards. I'm yet to see a "best practice" for ensuring proper time synchronisation within VMware VMs, but for now, NTP seems the best option.
cheers Luke
A note by Johnny Hughes in http://bugs.centos.org/view.php?id=2189 (comment 6644):
"As a side note ... if the clock GAINS (runs to fast) time you should be able to fix it with this:
(by setting the correct host.cpukHz) and vmware tools should adjust a clock that is too slow.
Also see this blog entry concerning host.cpukHz:
http://blog.autoedification.com/2006/11/vmware-guest-clock-runs-fast.html "
Akemi
A note by Johnny Hughes in http://bugs.centos.org/view.php?id=2189 (comment 6644):
"As a side note ... if the clock GAINS (runs to fast) time you should be able to fix it with this:
(by setting the correct host.cpukHz) and vmware tools should adjust a clock that is too slow.
Also see this blog entry concerning host.cpukHz:
http://blog.autoedification.com/2006/11/vmware-guest-clock-runs-fast.html "
Taking the advice from a few people and web pages, I made this change to grub.conf:
kernel /xen.gz-2.6.18-53.1.4.el5 divider=10 clock=pit
The clock is better but still not fixed. It jumps forward less often.
Jack
On Jan 9, 2008 12:35 PM, Jack Bailey jack@internetguy.net wrote:
A note by Johnny Hughes in http://bugs.centos.org/view.php?id=2189 (comment 6644):
"As a side note ... if the clock GAINS (runs to fast) time you should be able to fix it with this:
(by setting the correct host.cpukHz) and vmware tools should adjust a clock that is too slow.
Also see this blog entry concerning host.cpukHz:
http://blog.autoedification.com/2006/11/vmware-guest-clock-runs-fast.html "
Taking the advice from a few people and web pages, I made this change to grub.conf:
kernel /xen.gz-2.6.18-53.1.4.el5 divider=10 clock=pitThe clock is better but still not fixed. It jumps forward less often.
Jack
Despite the subject, the contents of this thread have drifted to vmware. No, you cannot use the divider= option for the xen kernel. This is for the standard kernel only.
Akemi
-
Akemi Yagi -
Brian Mathis -
Jack Bailey -
Johnny Hughes -
Luke Dudney -
Rick Barnes