CI-users

ci-users@lists.centos.org

420 discussions

Install & enable NTP on Duffy nodes
by František Šumšal 13 Jan '20

13 Jan '20

Hello! After recent struggles with compilation under Vagrant over NFS, I'd like to ask if it would be possible to automatically sync all machines provided by Duffy using NTP. We encountered several nasty race conditions caused by system time discrepancies between a Duffy host and a VM, which I managed to mitigate by installing & enabling chrony. Unfortunately, it's not perfect, as the time synchronization apparently takes some time, so the race condition is present, but it's much less probable. Going through Jenkins logs it looks that all machines have their time significantly off: ● chronyd.service - NTP client/server Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2020-01-12 16:06:53 GMT; 5h 51min left Docs: man:chronyd(8) man:chrony.conf(5) Main PID: 1992 (chronyd) CGroup: /system.slice/chronyd.service └─1992 /usr/sbin/chronyd Jan 12 16:06:53 n15.pufty systemd[1]: Starting NTP client/server... Jan 12 16:06:53 n15.pufty chronyd[1992]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 +DEBUG) Jan 12 16:06:53 n15.pufty systemd[1]: Started NTP client/server. Jan 12 16:07:01 n15.pufty chronyd[1992]: Selected source xx.xx.xx.xx Jan 12 16:07:01 n15.pufty chronyd[1992]: System clock wrong by -21251.336765 seconds, adjustment started Jan 12 10:12:50 n15.pufty chronyd[1992]: System clock was stepped by -21251.336765 seconds I suspect waiting for the synchronization to fully complete using chrony's CLI utilities would help, but having this to be handled automatically by each machine would be much better. Thanks! Frantisek -- PGP Key ID: 0xFB738CE27B634E4B

2 1

Duffy seems to be out of CentOS 7 nodes
by František Šumšal 03 Jan '20

03 Jan '20

Hello! Sorry to bother you during holidays, but is looks like Duffy has a trouble provisioning CentOS 7 nodes: ``` 2019-12-26 07:36:18,171 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) ... 2019-12-26 07:52:19,680 [agent-control/allocate_node] INFO: Waiting 1800 seconds before another retry 2019-12-26 08:22:19,757 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-26 08:22:19,759 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-26 08:22:20,029 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-26 08:22:20,029 [agent-control/allocate_node] INFO: Waiting 3600 seconds before another retry 2019-12-26 09:22:20,126 [agent-control/allocate_node] ERROR: Failed to allocated a node 2019-12-26 09:22:20,126 [agent-control/<module>] CRITICAL: Can't continue without a valid node ``` -- Frantisek Sumsal GPG key ID: 0xFB738CE27B634E4B

2 7

No space left on device
by Katerina Foniok 12 Dec '19

12 Dec '19

Hello guys, from yesterday evening we are experiencing failures in our jobs on ci.centos.org saying No space left on device. It's blocking our PR checks. Could you please take a look at what's happening there? Thank you! Have a nice day, Katka

3 12

Domain or ci.centos.org is down
by Petr Hracek 05 Dec '19

05 Dec '19

Hi folks, it seems like CI systems are down. Based on the pull request https://github.com/sclorg/s2i-python-container/pull/355 See log from build https://ci.centos.org/job/SCLo-container-python-rh/279/console Greetings Petr -- Petr Hracek email: phracek(a)redhat.com IRC phracek #brno #software-collections #tft #core-services #core-services-db GPG 0xA9FC9AC4 Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, Czech Republic

3 3

Nodes provided by ci.centos.org are not clean
by Katerina Foniok 02 Dec '19

02 Dec '19

Hi guys, from the last week, errors regarding node provided increased. For me, it seems like the nodes are not cleaned correctly, because a lot of stuff we use is already installed there and sometimes the folder with logs remains there too. I've summed up some errors in this issue: https://github.com/redhat-developer/rh-che/issues/1715 Thank you, have a nice day! Katka Foniok

2 1

Re: [Ci-users] Potentially mis-provisioned CentOS 8-stream nodes
by Vipul Siddharth 01 Dec '19

01 Dec '19

On Sun, Dec 1, 2019 at 8:09 PM František Šumšal <frantisek(a)sumsal.cz> wrote: > > On 12/1/19 3:04 PM, Vipul Siddharth wrote: > > Thank you for keeping me in cc, somehow I missed the email from Friday. > > Since your last email, I checked all 8-stream nodes to see if it's a > > thing with all but so far out of 15, I found just one node that has > > DNF missing (which is worse). I am trying a manual 8-Stream install to > > see if it's in a newly provisioned nodes or older/est ones (possibly > > catch the issue). > > Thank you for looking into this! While writing this email, I re-scheduled > two failed jobs which use CentOS 8 (not 8-stream), and both got a new > machine which had the same issue (missing `dnf`), so it's definitely not > isolated to CentOS 8-stream only. hmm, that's not good. CentOS 8-stream that I just tried manually had the dnf installed. WIll now try CentOS 8 and see if I can catch something. I will update here once I have figured out why this is happening. Sorry for this inconvenience. > > > All the ver=7 I checked have bash installed).. if possible, can you > > send me the ip/hostname for C7 node so that I can check it (if it > > hasn't deprovisioned) > > Unfortunately, the machine is long gone: > > ``` > Pseudo-terminal will not be allocated because stdin is not a terminal. > Warning: Permanently added 'n62.pufty,172.19.3.126' (RSA) to the list of known hosts. > bash: systemd-centos-ci/agent/bootstrap.sh: /usr/bin/bash: bad interpreter: No such file or directory > ``` > > However, so far it's been only a one time thing (let's hope it's going to > stay this way). > > > > > and the storage issue is now fixed. It was the whole central (Jenkins) > > master which was full and not the slave01. > > > > Thank you very much! > > > > On Sun, Dec 1, 2019 at 6:48 PM František Šumšal <frantisek(a)sumsal.cz> wrote: > >> > >> Hello again, > >> > >> Apart from the issue mentioned in the previous email (and which affects CentOS 8 nodes as well), > >> it looks like CentOS 7 nodes have a similar issue, but in this case it's missing `bash`. > >> > >> And to finish this beautiful weekend off, `systemd-ci-slave01` is apparently out of space: > >> > >> ``` > >> Archiving artifacts > >> ERROR: Step ‘Archive the artifacts’ aborted due to exception: > >> java.io.IOException: No space left on device > >> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > >> at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) > >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > >> at sun.nio.ch.IOUtil.write(IOUtil.java:65) > >> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) > >> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) > >> at java.nio.channels.Channels.writeFully(Channels.java:101) > >> at java.nio.channels.Channels.access$000(Channels.java:61) > >> at java.nio.channels.Channels$1.write(Channels.java:174) > >> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2315) > >> ``` > >> > >> On 11/29/19 10:56 AM, František Šumšal wrote: > >>> Hello! > >>> > >>> Today I noticed that approx. half of the CentOS 8-stream nodes our CI got its hands on today > >>> were missing `dnf`: > >>> > >>> ``` > >>> 2019-11-29 09:34:59,493 [agent-control/allocate_node] INFO: Attempting to allocate a node (8-stream x86_64) > >>> 2019-11-29 09:34:59,493 [agent-control/_execute_api_command] INFO: Duffy request URL: http://admin.ci.centos.org:8080/Node/get > >>> 2019-11-29 09:34:59,507 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): admin.ci.centos.org > >>> 2019-11-29 09:35:00,669 [agent-control/allocate_node] INFO: Successfully allocated node 'n29.pufty' (e819a934) > >>> 2019-11-29 09:35:00,669 [agent-control/<module>] INFO: PHASE 1: Setting up basic dependencies to configure CI repository > >>> ... > >>> Pseudo-terminal will not be allocated because stdin is not a terminal. > >>> Warning: Permanently added 'n29.pufty,172.19.3.93' (ECDSA) to the list of known hosts. > >>> bash: dnf: command not found > >>> ``` > >>> > >>> I'm not sure if it's already known, but it's indeed quite strange :-) > >>> > >>> Thanks! > >>> > >>> > >>> _______________________________________________ > >>> Ci-users mailing list > >>> Ci-users(a)centos.org > >>> https://lists.centos.org/mailman/listinfo/ci-users > >>> > >> > >> -- > >> PGP Key ID: 0xFB738CE27B634E4B > >> > > > > > > -- > PGP Key ID: 0xFB738CE27B634E4B > -- Vipul Siddharth He/His/Him Fedora | CentOS CI Infrastructure Team Red Hat w: vipul.dev

1 0

Re: [Ci-users] Potentially mis-provisioned CentOS 8-stream nodes
by Vipul Siddharth 01 Dec '19

01 Dec '19

Thank you for keeping me in cc, somehow I missed the email from Friday. Since your last email, I checked all 8-stream nodes to see if it's a thing with all but so far out of 15, I found just one node that has DNF missing (which is worse). I am trying a manual 8-Stream install to see if it's in a newly provisioned nodes or older/est ones (possibly catch the issue). All the ver=7 I checked have bash installed).. if possible, can you send me the ip/hostname for C7 node so that I can check it (if it hasn't deprovisioned) and the storage issue is now fixed. It was the whole central (Jenkins) master which was full and not the slave01. On Sun, Dec 1, 2019 at 6:48 PM František Šumšal <frantisek(a)sumsal.cz> wrote: > > Hello again, > > Apart from the issue mentioned in the previous email (and which affects CentOS 8 nodes as well), > it looks like CentOS 7 nodes have a similar issue, but in this case it's missing `bash`. > > And to finish this beautiful weekend off, `systemd-ci-slave01` is apparently out of space: > > ``` > Archiving artifacts > ERROR: Step ‘Archive the artifacts’ aborted due to exception: > java.io.IOException: No space left on device > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > at sun.nio.ch.IOUtil.write(IOUtil.java:65) > at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) > at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) > at java.nio.channels.Channels.writeFully(Channels.java:101) > at java.nio.channels.Channels.access$000(Channels.java:61) > at java.nio.channels.Channels$1.write(Channels.java:174) > at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2315) > ``` > > On 11/29/19 10:56 AM, František Šumšal wrote: > > Hello! > > > > Today I noticed that approx. half of the CentOS 8-stream nodes our CI got its hands on today > > were missing `dnf`: > > > > ``` > > 2019-11-29 09:34:59,493 [agent-control/allocate_node] INFO: Attempting to allocate a node (8-stream x86_64) > > 2019-11-29 09:34:59,493 [agent-control/_execute_api_command] INFO: Duffy request URL: http://admin.ci.centos.org:8080/Node/get > > 2019-11-29 09:34:59,507 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): admin.ci.centos.org > > 2019-11-29 09:35:00,669 [agent-control/allocate_node] INFO: Successfully allocated node 'n29.pufty' (e819a934) > > 2019-11-29 09:35:00,669 [agent-control/<module>] INFO: PHASE 1: Setting up basic dependencies to configure CI repository > > ... > > Pseudo-terminal will not be allocated because stdin is not a terminal. > > Warning: Permanently added 'n29.pufty,172.19.3.93' (ECDSA) to the list of known hosts. > > bash: dnf: command not found > > ``` > > > > I'm not sure if it's already known, but it's indeed quite strange :-) > > > > Thanks! > > > > > > _______________________________________________ > > Ci-users mailing list > > Ci-users(a)centos.org > > https://lists.centos.org/mailman/listinfo/ci-users > > > > -- > PGP Key ID: 0xFB738CE27B634E4B > -- Vipul Siddharth He/His/Him Fedora | CentOS CI Infrastructure Team Red Hat w: vipul.dev

1 0

Potentially mis-provisioned CentOS 8-stream nodes
by František Šumšal 29 Nov '19

29 Nov '19

Hello! Today I noticed that approx. half of the CentOS 8-stream nodes our CI got its hands on today were missing `dnf`: ``` 2019-11-29 09:34:59,493 [agent-control/allocate_node] INFO: Attempting to allocate a node (8-stream x86_64) 2019-11-29 09:34:59,493 [agent-control/_execute_api_command] INFO: Duffy request URL: http://admin.ci.centos.org:8080/Node/get 2019-11-29 09:34:59,507 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): admin.ci.centos.org 2019-11-29 09:35:00,669 [agent-control/allocate_node] INFO: Successfully allocated node 'n29.pufty' (e819a934) 2019-11-29 09:35:00,669 [agent-control/<module>] INFO: PHASE 1: Setting up basic dependencies to configure CI repository ... Pseudo-terminal will not be allocated because stdin is not a terminal. Warning: Permanently added 'n29.pufty,172.19.3.93' (ECDSA) to the list of known hosts. bash: dnf: command not found ``` I'm not sure if it's already known, but it's indeed quite strange :-) Thanks! -- Frantisek Sumsal GPG key ID: 0xFB738CE27B634E4B

1 0

Nodes are not provisioned
by Katerina Foniok 27 Nov '19

27 Nov '19

Hi guys, we are experiencing the issue with provisioning nodes on ci.centos.org. The issue started for us ~9 PM UTC yesterday. Our jobs are running on devtools-ci-slave04. Do you have any information about this? Thank you, have a great day! Katka Foniok Che QE

2 2

CentOS 8 and CentOS 8-Stream now available in CI
by Vipul Siddharth 13 Nov '19

13 Nov '19

Hello everyone, This is an email to inform you that now you can checkout CentOS 8 and CentOS 8-Stream nodes in CI. We have upgraded all the agents with the latest cico client which supports the parameter --release=8 and --release=8-stream. Here is a minimal example to get an 8/8-stream node `cico node get --release=8` `cico node get --release=8-stream` You should be able to use all other parameters like earlier. In case you see any problem with them, please let us know here or on #centos-ci@Freenode. Thank you -- Vipul Siddharth He/His/Him Fedora | CentOS CI Infrastructure Team Red Hat w: vipul.dev

8 15

← Newer
1
...
9
10
11
12
13
14
15
...
42
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users