We've noticed our fully-virtualized KVM guests' time keeps getting faster, despite running ntpd.
A quick google reveals this to be a known problem with virtual guests. It seems there are a variety of solutions, some apparently vendor-specific.
What is the best practice as of now for KVM guests?
* Setting independent_wallclock=1 (or is that a xen-only thing)?
* Passing kernel parameters? If so, which? The vmware KB mentions "notsc divider=10" -- is that vmware-specific?
* Just don't run ntpd on virtual guests? And let it get its time from the host.
The last seems like the easiest/cleanest, if it really does always get its time from the host.
Thanks for any info.
johnny
On 21 Jul 2009, at 00:41, Johnny Tan wrote:
- Setting independent_wallclock=1 (or is that a xen-only thing)?
[Can't comment - I'm not a Xen user]
- Passing kernel parameters? If so, which? The vmware KB
mentions "notsc divider=10" -- is that vmware-specific?
The divider=10 will help - reduces the number of timer interrupts through the system which tends to mean it gets itself less confused when counting them :-) [problem tends to be that the interrupts get batched due to the guest not being running all the time].
- Just don't run ntpd on virtual guests? And let it get its
time from the host.
VMware BP is now to run local ntpd - up to last year or so it was use the correction from vmtools (which I never found to work well). However VMware does have this back channel through vmtools to set the time, so that will only work well under Xen if it has an equivalent.
My suggestions, from my position of complete ignorance on Xen (but I do quite a lot under VMware) is to tweak your kernel params (definitely divider, maybe some other tweaks depending on how the virtual hardware appears, but on our systems we use clocksource=acpi_pm)
The last seems like the easiest/cleanest, if it really does always get its time from the host.
Linux time generally picks up from the RTC on boot, and then relies on counting interrupts (gross simplification alert!). In a virtualised environment where your clocking is being externally tampered with this does not work well.
Hope that this helps but you have opened a very messy can of worms here!
Nigel. -- [ Nigel Metheringham Nigel.Metheringham@InTechnology.com ] [ - Comments in this message are my own and not ITO opinion/policy - ]
Nigel Metheringham wrote:
On 21 Jul 2009, at 00:41, Johnny Tan wrote:
- Setting independent_wallclock=1 (or is that a xen-only thing)?
[Can't comment - I'm not a Xen user]
Ok, I've done a bit more research, it seems the above is a xen-only thing, so it's a no-go for KVM.
- Passing kernel parameters? If so, which? The vmware KB
mentions "notsc divider=10" -- is that vmware-specific?
This is NOT vmware-specific. In fact, it seems like divider=10 is the only thing that works. "notsc" (at least by itself) does not help.
- Just don't run ntpd on virtual guests? And let it get its
time from the host.
This does not work. On Xen, domU's apparently get their time from the physical host, so it would work for Xen. But for KVM, this does not seem to be the case.
My suggestions, from my position of complete ignorance on Xen (but I do quite a lot under VMware) is to tweak your kernel params (definitely divider, maybe some other tweaks depending on how the virtual hardware appears, but on our systems we use clocksource=acpi_pm)
Yes, divider works. I tried "clocksource=acpi_pm" -- it does not work for our KVM guests, at least not by itself. (We're using KVM-85.)
In summary: For KVM guests, pass "divider=10" as kernel option in grub.conf for your KVM guest, then use ntp to keep time (also on your KVM guest).
(p.s. However, I believe these issues are fixed with RHEL 5.4.)
Hope that this helps but you have opened a very messy can of worms here!
I figured it'd be a can of worms, but a useful one. I'm surprised it's not asked more often. I'm guessing most people are using Xen still, where this is less of an issue.
Thanks Nigel.
johnny
Just curious, but on Xen if I "disable/break" the hwclock, Linux guests apparently have no problem using system time set by host (so far) and may be preventing "Time goes backwards" errors too.
e.g. In a CentOS pv guest as root:
cp /sbin/hwclock /sbin/hwclock.original echo exit 0 > /sbin/hwclock
You may have to do that again on updates.
I'm also not sure of the source of this way for Xen pv guests and have not tried it , but is the same principle:
Prevent "hwclock" from being used ================================= mv /sbin/hwclock /sbin/hwclock.dist touch /sbin/hwclock chmod +x /sbin/hwclock
Johnny Tan wrote:
Nigel Metheringham wrote:
On 21 Jul 2009, at 00:41, Johnny Tan wrote:
- Setting independent_wallclock=1 (or is that a xen-only thing)?
[Can't comment - I'm not a Xen user]
Ok, I've done a bit more research, it seems the above is a xen-only thing, so it's a no-go for KVM.
- Passing kernel parameters? If so, which? The vmware KB
mentions "notsc divider=10" -- is that vmware-specific?
This is NOT vmware-specific. In fact, it seems like divider=10 is the only thing that works. "notsc" (at least by itself) does not help.
- Just don't run ntpd on virtual guests? And let it get its
time from the host.
This does not work. On Xen, domU's apparently get their time from the physical host, so it would work for Xen. But for KVM, this does not seem to be the case.
My suggestions, from my position of complete ignorance on Xen (but I do quite a lot under VMware) is to tweak your kernel params (definitely divider, maybe some other tweaks depending on how the virtual hardware appears, but on our systems we use clocksource=acpi_pm)
Yes, divider works. I tried "clocksource=acpi_pm" -- it does not work for our KVM guests, at least not by itself. (We're using KVM-85.)
In summary: For KVM guests, pass "divider=10" as kernel option in grub.conf for your KVM guest, then use ntp to keep time (also on your KVM guest).
(p.s. However, I believe these issues are fixed with RHEL 5.4.)
Hope that this helps but you have opened a very messy can of worms here!
I figured it'd be a can of worms, but a useful one. I'm surprised it's not asked more often. I'm guessing most people are using Xen still, where this is less of an issue.
Thanks Nigel.
johnny _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
-
Ben Montanelli -
Johnny Tan -
Nigel Metheringham