CI-users December 2019

ci-users@lists.centos.org

6 participants
6 discussions

Duffy seems to be out of CentOS 7 nodes
by František Šumšal 03 Jan '20

03 Jan '20

Hello! Sorry to bother you during holidays, but is looks like Duffy has a trouble provisioning CentOS 7 nodes: ``` 2019-12-26 07:36:18,171 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) ... 2019-12-26 07:52:19,680 [agent-control/allocate_node] INFO: Waiting 1800 seconds before another retry 2019-12-26 08:22:19,757 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-26 08:22:19,759 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-26 08:22:20,029 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-26 08:22:20,029 [agent-control/allocate_node] INFO: Waiting 3600 seconds before another retry 2019-12-26 09:22:20,126 [agent-control/allocate_node] ERROR: Failed to allocated a node 2019-12-26 09:22:20,126 [agent-control/<module>] CRITICAL: Can't continue without a valid node ``` -- Frantisek Sumsal GPG key ID: 0xFB738CE27B634E4B

2 7

No space left on device
by Katerina Foniok 12 Dec '19

12 Dec '19

Hello guys, from yesterday evening we are experiencing failures in our jobs on ci.centos.org saying No space left on device. It's blocking our PR checks. Could you please take a look at what's happening there? Thank you! Have a nice day, Katka

3 12

Domain or ci.centos.org is down
by Petr Hracek 05 Dec '19

05 Dec '19

Hi folks, it seems like CI systems are down. Based on the pull request https://github.com/sclorg/s2i-python-container/pull/355 See log from build https://ci.centos.org/job/SCLo-container-python-rh/279/console Greetings Petr -- Petr Hracek email: phracek(a)redhat.com IRC phracek #brno #software-collections #tft #core-services #core-services-db GPG 0xA9FC9AC4 Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, Czech Republic

3 3

Nodes provided by ci.centos.org are not clean
by Katerina Foniok 02 Dec '19

02 Dec '19

Hi guys, from the last week, errors regarding node provided increased. For me, it seems like the nodes are not cleaned correctly, because a lot of stuff we use is already installed there and sometimes the folder with logs remains there too. I've summed up some errors in this issue: https://github.com/redhat-developer/rh-che/issues/1715 Thank you, have a nice day! Katka Foniok

2 1

Re: [Ci-users] Potentially mis-provisioned CentOS 8-stream nodes
by Vipul Siddharth 01 Dec '19

01 Dec '19

On Sun, Dec 1, 2019 at 8:09 PM František Šumšal <frantisek(a)sumsal.cz> wrote: > > On 12/1/19 3:04 PM, Vipul Siddharth wrote: > > Thank you for keeping me in cc, somehow I missed the email from Friday. > > Since your last email, I checked all 8-stream nodes to see if it's a > > thing with all but so far out of 15, I found just one node that has > > DNF missing (which is worse). I am trying a manual 8-Stream install to > > see if it's in a newly provisioned nodes or older/est ones (possibly > > catch the issue). > > Thank you for looking into this! While writing this email, I re-scheduled > two failed jobs which use CentOS 8 (not 8-stream), and both got a new > machine which had the same issue (missing `dnf`), so it's definitely not > isolated to CentOS 8-stream only. hmm, that's not good. CentOS 8-stream that I just tried manually had the dnf installed. WIll now try CentOS 8 and see if I can catch something. I will update here once I have figured out why this is happening. Sorry for this inconvenience. > > > All the ver=7 I checked have bash installed).. if possible, can you > > send me the ip/hostname for C7 node so that I can check it (if it > > hasn't deprovisioned) > > Unfortunately, the machine is long gone: > > ``` > Pseudo-terminal will not be allocated because stdin is not a terminal. > Warning: Permanently added 'n62.pufty,172.19.3.126' (RSA) to the list of known hosts. > bash: systemd-centos-ci/agent/bootstrap.sh: /usr/bin/bash: bad interpreter: No such file or directory > ``` > > However, so far it's been only a one time thing (let's hope it's going to > stay this way). > > > > > and the storage issue is now fixed. It was the whole central (Jenkins) > > master which was full and not the slave01. > > > > Thank you very much! > > > > On Sun, Dec 1, 2019 at 6:48 PM František Šumšal <frantisek(a)sumsal.cz> wrote: > >> > >> Hello again, > >> > >> Apart from the issue mentioned in the previous email (and which affects CentOS 8 nodes as well), > >> it looks like CentOS 7 nodes have a similar issue, but in this case it's missing `bash`. > >> > >> And to finish this beautiful weekend off, `systemd-ci-slave01` is apparently out of space: > >> > >> ``` > >> Archiving artifacts > >> ERROR: Step ‘Archive the artifacts’ aborted due to exception: > >> java.io.IOException: No space left on device > >> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > >> at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) > >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > >> at sun.nio.ch.IOUtil.write(IOUtil.java:65) > >> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) > >> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) > >> at java.nio.channels.Channels.writeFully(Channels.java:101) > >> at java.nio.channels.Channels.access$000(Channels.java:61) > >> at java.nio.channels.Channels$1.write(Channels.java:174) > >> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2315) > >> ``` > >> > >> On 11/29/19 10:56 AM, František Šumšal wrote: > >>> Hello! > >>> > >>> Today I noticed that approx. half of the CentOS 8-stream nodes our CI got its hands on today > >>> were missing `dnf`: > >>> > >>> ``` > >>> 2019-11-29 09:34:59,493 [agent-control/allocate_node] INFO: Attempting to allocate a node (8-stream x86_64) > >>> 2019-11-29 09:34:59,493 [agent-control/_execute_api_command] INFO: Duffy request URL: http://admin.ci.centos.org:8080/Node/get > >>> 2019-11-29 09:34:59,507 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): admin.ci.centos.org > >>> 2019-11-29 09:35:00,669 [agent-control/allocate_node] INFO: Successfully allocated node 'n29.pufty' (e819a934) > >>> 2019-11-29 09:35:00,669 [agent-control/<module>] INFO: PHASE 1: Setting up basic dependencies to configure CI repository > >>> ... > >>> Pseudo-terminal will not be allocated because stdin is not a terminal. > >>> Warning: Permanently added 'n29.pufty,172.19.3.93' (ECDSA) to the list of known hosts. > >>> bash: dnf: command not found > >>> ``` > >>> > >>> I'm not sure if it's already known, but it's indeed quite strange :-) > >>> > >>> Thanks! > >>> > >>> > >>> _______________________________________________ > >>> Ci-users mailing list > >>> Ci-users(a)centos.org > >>> https://lists.centos.org/mailman/listinfo/ci-users > >>> > >> > >> -- > >> PGP Key ID: 0xFB738CE27B634E4B > >> > > > > > > -- > PGP Key ID: 0xFB738CE27B634E4B > -- Vipul Siddharth He/His/Him Fedora | CentOS CI Infrastructure Team Red Hat w: vipul.dev

1 0

Re: [Ci-users] Potentially mis-provisioned CentOS 8-stream nodes
by Vipul Siddharth 01 Dec '19

01 Dec '19

Thank you for keeping me in cc, somehow I missed the email from Friday. Since your last email, I checked all 8-stream nodes to see if it's a thing with all but so far out of 15, I found just one node that has DNF missing (which is worse). I am trying a manual 8-Stream install to see if it's in a newly provisioned nodes or older/est ones (possibly catch the issue). All the ver=7 I checked have bash installed).. if possible, can you send me the ip/hostname for C7 node so that I can check it (if it hasn't deprovisioned) and the storage issue is now fixed. It was the whole central (Jenkins) master which was full and not the slave01. On Sun, Dec 1, 2019 at 6:48 PM František Šumšal <frantisek(a)sumsal.cz> wrote: > > Hello again, > > Apart from the issue mentioned in the previous email (and which affects CentOS 8 nodes as well), > it looks like CentOS 7 nodes have a similar issue, but in this case it's missing `bash`. > > And to finish this beautiful weekend off, `systemd-ci-slave01` is apparently out of space: > > ``` > Archiving artifacts > ERROR: Step ‘Archive the artifacts’ aborted due to exception: > java.io.IOException: No space left on device > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > at sun.nio.ch.IOUtil.write(IOUtil.java:65) > at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) > at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) > at java.nio.channels.Channels.writeFully(Channels.java:101) > at java.nio.channels.Channels.access$000(Channels.java:61) > at java.nio.channels.Channels$1.write(Channels.java:174) > at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2315) > ``` > > On 11/29/19 10:56 AM, František Šumšal wrote: > > Hello! > > > > Today I noticed that approx. half of the CentOS 8-stream nodes our CI got its hands on today > > were missing `dnf`: > > > > ``` > > 2019-11-29 09:34:59,493 [agent-control/allocate_node] INFO: Attempting to allocate a node (8-stream x86_64) > > 2019-11-29 09:34:59,493 [agent-control/_execute_api_command] INFO: Duffy request URL: http://admin.ci.centos.org:8080/Node/get > > 2019-11-29 09:34:59,507 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): admin.ci.centos.org > > 2019-11-29 09:35:00,669 [agent-control/allocate_node] INFO: Successfully allocated node 'n29.pufty' (e819a934) > > 2019-11-29 09:35:00,669 [agent-control/<module>] INFO: PHASE 1: Setting up basic dependencies to configure CI repository > > ... > > Pseudo-terminal will not be allocated because stdin is not a terminal. > > Warning: Permanently added 'n29.pufty,172.19.3.93' (ECDSA) to the list of known hosts. > > bash: dnf: command not found > > ``` > > > > I'm not sure if it's already known, but it's indeed quite strange :-) > > > > Thanks! > > > > > > _______________________________________________ > > Ci-users mailing list > > Ci-users(a)centos.org > > https://lists.centos.org/mailman/listinfo/ci-users > > > > -- > PGP Key ID: 0xFB738CE27B634E4B > -- Vipul Siddharth He/His/Him Fedora | CentOS CI Infrastructure Team Red Hat w: vipul.dev

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users December 2019