[Ci-users] Install & enable NTP on Duffy nodes

On Sun, Jan 12, 2020, at 06:15, František Šumšal wrote:
> Hello!
> 
> After recent struggles with compilation under Vagrant over NFS, I'd like to ask
> if it would be possible to automatically sync all machines provided by Duffy using
> NTP.
> 
> We encountered several nasty race conditions caused by system time discrepancies
> between a Duffy host and a VM, which I managed to mitigate by installing & enabling
> chrony. Unfortunately, it's not perfect, as the time synchronization apparently takes
> some time, so the race condition is present, but it's much less probable.
> 
> Going through Jenkins logs it looks that all machines have their time significantly off:
> 
> ● chronyd.service - NTP client/server
>  Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
>  Active: active (running) since Sun 2020-01-12 16:06:53 GMT; 5h 51min left
>  Docs: man:chronyd(8)
>  man:chrony.conf(5)
> Main PID: 1992 (chronyd)
>  CGroup: /system.slice/chronyd.service
>  └─1992 /usr/sbin/chronyd
> 
> Jan 12 16:06:53 n15.pufty systemd[1]: Starting NTP client/server...
> Jan 12 16:06:53 n15.pufty chronyd[1992]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 +DEBUG)
> Jan 12 16:06:53 n15.pufty systemd[1]: Started NTP client/server.
> Jan 12 16:07:01 n15.pufty chronyd[1992]: Selected source xx.xx.xx.xx
> Jan 12 16:07:01 n15.pufty chronyd[1992]: System clock wrong by -21251.336765 seconds, adjustment started
> Jan 12 10:12:50 n15.pufty chronyd[1992]: System clock was stepped by -21251.336765 seconds
> 
> I suspect waiting for the synchronization to fully complete using chrony's CLI utilities would
> help, but having this to be handled automatically by each machine would be much better.
> 
> Thanks!
> 
> Frantisek
> 
> -- 
> PGP Key ID: 0xFB738CE27B634E4B
> 
> 
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users
> 
> 
> *Attachments:*
>  * signature.asc


Chronyd *should* be enabled by default. Unfortunately the system clocks on this hardware are not very reliable.

If your builds are sensitive to time shifts, it may make sense for you to put some guards/checks in place, but we'll look into a job that resets the system clocks on a regular basis.

--Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200112/a484413f/attachment.html>