[Ci-users] Potentially mis-provisioned CentOS 8-stream nodes

On Sun, Dec 1, 2019 at 8:09 PM František Šumšal <frantisek at sumsal.cz> wrote:
>
> On 12/1/19 3:04 PM, Vipul Siddharth wrote:
> > Thank you for keeping me in cc, somehow I missed the email from Friday.
> > Since your last email, I checked all 8-stream nodes to see if it's a
> > thing with all but so far out of 15, I found just one node that has
> > DNF missing (which is worse). I am trying a manual 8-Stream install to
> > see if it's in a newly provisioned nodes or older/est ones (possibly
> > catch the issue).
>
> Thank you for looking into this! While writing this email, I re-scheduled
> two failed jobs which use CentOS 8 (not 8-stream), and both got a new
> machine which had the same issue (missing `dnf`), so it's definitely not
> isolated to CentOS 8-stream only.
hmm, that's not good. CentOS 8-stream that I just tried manually had
the dnf installed.
WIll now try CentOS 8 and see if I can catch something.
I will update here once I have figured out why this is happening.
Sorry for this inconvenience.
>
> > All the ver=7 I checked have bash installed).. if possible, can you
> > send me the ip/hostname for C7 node so that I can check it (if it
> > hasn't deprovisioned)
>
> Unfortunately, the machine is long gone:
>
> ```
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> Warning: Permanently added 'n62.pufty,172.19.3.126' (RSA) to the list of known hosts.
> bash: systemd-centos-ci/agent/bootstrap.sh: /usr/bin/bash: bad interpreter: No such file or directory
> ```
>
> However, so far it's been only a one time thing (let's hope it's going to
> stay this way).
>
> >
> > and the storage issue is now fixed. It was the whole central (Jenkins)
> > master which was full and not the slave01.
> >
>
> Thank you very much!
>
>
> > On Sun, Dec 1, 2019 at 6:48 PM František Šumšal <frantisek at sumsal.cz> wrote:
> >>
> >> Hello again,
> >>
> >> Apart from the issue mentioned in the previous email (and which affects CentOS 8 nodes as well),
> >> it looks like CentOS 7 nodes have a similar issue, but in this case it's missing `bash`.
> >>
> >> And to finish this beautiful weekend off, `systemd-ci-slave01` is apparently out of space:
> >>
> >> ```
> >> Archiving artifacts
> >> ERROR: Step ‘Archive the artifacts’ aborted due to exception:
> >> java.io.IOException: No space left on device
> >>         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> >>         at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
> >>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> >>         at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> >>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
> >>         at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
> >>         at java.nio.channels.Channels.writeFully(Channels.java:101)
> >>         at java.nio.channels.Channels.access$000(Channels.java:61)
> >>         at java.nio.channels.Channels$1.write(Channels.java:174)
> >>         at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2315)
> >> ```
> >>
> >> On 11/29/19 10:56 AM, František Šumšal wrote:
> >>> Hello!
> >>>
> >>> Today I noticed that approx. half of the CentOS 8-stream nodes our CI got its hands on today
> >>> were missing `dnf`:
> >>>
> >>> ```
> >>> 2019-11-29 09:34:59,493 [agent-control/allocate_node] INFO: Attempting to allocate a node (8-stream x86_64)
> >>> 2019-11-29 09:34:59,493 [agent-control/_execute_api_command] INFO: Duffy request URL: http://admin.ci.centos.org:8080/Node/get
> >>> 2019-11-29 09:34:59,507 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): admin.ci.centos.org
> >>> 2019-11-29 09:35:00,669 [agent-control/allocate_node] INFO: Successfully allocated node 'n29.pufty' (e819a934)
> >>> 2019-11-29 09:35:00,669 [agent-control/<module>] INFO: PHASE 1: Setting up basic dependencies to configure CI repository
> >>> ...
> >>> Pseudo-terminal will not be allocated because stdin is not a terminal.
> >>> Warning: Permanently added 'n29.pufty,172.19.3.93' (ECDSA) to the list of known hosts.
> >>> bash: dnf: command not found
> >>> ```
> >>>
> >>> I'm not sure if it's already known, but it's indeed quite strange :-)
> >>>
> >>> Thanks!
> >>>
> >>>
> >>> _______________________________________________
> >>> Ci-users mailing list
> >>> Ci-users at centos.org
> >>> https://lists.centos.org/mailman/listinfo/ci-users
> >>>
> >>
> >> --
> >> PGP Key ID: 0xFB738CE27B634E4B
> >>
> >
> >
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev