CI-users April 2020

ci-users@lists.centos.org

8 participants
7 discussions

CI state and efforts
by Vipul Siddharth 01 May '20

01 May '20

Greetings everyone This email is to update you all on some of the efforts we are about to put/putting in place within the CentOS CI Infrastructure. Some of the things have been long due and we realize that. For a couple of years, our efforts have mainly been under keeping the fires away and general maintenance (adding new releases, arches, making things in the background a bit sophisticated, etc). Right now we have started working on some things that you would notice and will impact you (in a better way hopefully!) and we will be keeping you updated. * Monitoring - Done * A couple of months ago, All I had was basic ansible scripts to show me data, and that was my way of monitoring things. As one would expect, I often missed things, and email on the list would pop up with `insufficient nodes` or `no space on Jenkins master`. While we tried to be relatively quick on the resolution, we needed a better way. In early February, Fabian (Thank you) set up a monitoring system (added ci.centos.org to CentOS Infra Zabbix instance) which in turn reduced such incidence by a lot. I hope you have not noticed those emails/complaints in the last 2-3 months. We will be adding monitoring to more areas as we go. * OpenShift cluster upgrade - In Progress * https://console.apps.ci.centos.org:8443/ is using openshift version 3.6 which has been outdated for a while. We are upgrading the cluster to OpenShift 4.3 which handles some of the group permissions in a better way. Apart from the shiny new deployment which has a bunch of improvements/features, it will allow us to give admin access to more members easily which in turn will decrease the wait time for new accounts/users (it won't wait on a single person to do all the work). * Deployment of OpenNebula for VM checkouts through cico - Upcoming * We will be deploying OpenNebula and will extend duffy api to support VMs in the check out process. This will be useful for the jobs who don't need bare metals to do their tests and need slightly less resources. This will also enable us to support Fedora as an on-demand offering distro (in vm) that can be used to run your test. Ideally, it would look like `cico node get --type vm` or `cico node get --type vm --distro fedora -- version rawhide`. This needs more grooming and planning before we start planning exact images/distros and ways but I am hopeful that it will make your experience better. We are hoping to update you all with the things going on in the background pretty frequently and the updates will show up as weekly emails `progress of $initiatives_mentioned_above` and "probably" monthly email of some matrices of things that happened in that month (not sure yet). Once we have upgraded the openshift cluster, we will need your help in migrating the jobs from legacy `ci.centos.org` to openshift. You will get a namespace where you can control all things related. This will introduce easier workflow and management for admins and more freedom to you, tenants! If you want to discuss things more, feel free to contact us or catch us at #centos-ci@Freenode Thank you! Be safe -- @siddharthvipul He/His/Him Fedora | CentOS CI Infrastructure Team Red Hat

2 1

VM migration in progress
by Fabian Arrotin 30 Apr '20

30 Apr '20

Hi all, Due to an upgrade/reorg of the infra in CentOS CI , we'll have to move the VMs actually hosting most of the jenkins slaves connected to ci.centos.org jenkins master. That means that when VM is moved/migrated, you'll see a temporary jenkins slave disconnection from master and then reconnect to process jobs in the queue. As we'll work on those migrations based on various factors, we can't give you an estimate about when and how long it will take for your specific VM but you're now aware that one explains the other (this mail being a FYI) -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab

1 1

[Resolved] Unplanned outage: apps.ci.centos.org
by Brian Stinson 28 Apr '20

28 Apr '20

Hi Folks, One of the servers hosting physical volumes for apps.ci.centos.org was configured to automatically update to CentOS 7.8.2003 and we lost NFS services for a little bit. I rebooted some of the nodes in the cluster, and restarted the NFS server. You may notice some pod flapping for the next bit while things settle down, but things seem to be trending toward normal. If you have questions, please let me know. Apologies for the inconvenience. -- Brian Stinson CentOS Infrastructure Team

1 0

cannot get active node
by Vladimir Benes 21 Apr '20

21 Apr '20

Hi all, any idea where is the issue? by request for 8-stream box I am getting an error: [networkmanager@slave03 ~]$ cico node get --release 8-stream Starting new HTTP connection (1): admin.ci.centos.org Resetting dropped connection: admin.ci.centos.org The requested operation failed as no inventory is available. Thanks, Vladimir

3 3

SCL containers failed with message: nss_wrapper package is missing.
by Petr Hracek 20 Apr '20

20 Apr '20

Dear CI Users, RHSCL team triggers nightly build testing of RHSCL images for CentOS. Unfortunately it failed on: ```bash Step 6/12 : RUN yum install -y centos-release-scl-rh && ( [ "rh-${NAME}${NODEJS_VERSION}" != "${NODEJS_SCL}" ] && yum remove -y ${NODEJS_SCL}\* || : ) && INSTALL_PKGS="rh-nodejs${NODEJS_VERSION} rh-nodejs${NODEJS_VERSION}-npm rh-nodejs${NODEJS_VERSION}-nodejs-nodemon nss_wrapper" && ln -s /usr/lib/node_modules/nodemon/bin/nodemon.js /usr/bin/nodemon && yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS && rpm -V $INSTALL_PKGS && yum -y clean all --enablerepo='*' ---> Running in 73e8e1473359 [91m [0mLoaded plugins: fastestmirror, ovl Determining fastest mirrors * base: mirrors.greenmountainaccess.net * centos-sclo-rh: mirror.umd.edu * centos-sclo-sclo: repo.miserver.it.umich.edu * extras: mirror.es.its.nyu.edu * updates: mirror.vtti.vt.edu Package centos-release-scl-rh-2-3.el7.centos.noarch already installed and latest version Nothing to do Loaded plugins: fastestmirror, ovl Loading mirror speeds from cached hostfile * base: mirrors.greenmountainaccess.net * centos-sclo-rh: mirror.umd.edu * centos-sclo-sclo: repo.miserver.it.umich.edu * extras: mirror.es.its.nyu.edu * updates: mirror.vtti.vt.edu Package rh-nodejs10-npm-6.13.4-10.19.0.1.el7.x86_64 already installed and latest version No package nss_wrapper available. Resolving Dependencies --> Running transaction check ---> Package rh-nodejs10.x86_64 0:3.2-3.el7 will be installed ---> Package rh-nodejs10-nodejs-nodemon.noarch 0:1.18.3-3.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: rh-nodejs10 x86_64 3.2-3.el7 centos-sclo-rh 7.0 k rh-nodejs10-nodejs-nodemon noarch 1.18.3-3.el7 centos-sclo-rh 941 k Transaction Summary ================================================================================ Install 2 Packages Total download size: 948 k Installed size: 4.0 M Downloading packages: -------------------------------------------------------------------------------- Total 2.3 MB/s | 948 kB 00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : rh-nodejs10-3.2-3.el7.x86_64 1/2 Installing : rh-nodejs10-nodejs-nodemon-1.18.3-3.el7.noarch 2/2 Verifying : rh-nodejs10-nodejs-nodemon-1.18.3-3.el7.noarch 1/2 Verifying : rh-nodejs10-3.2-3.el7.x86_64 2/2 Installed: rh-nodejs10.x86_64 0:3.2-3.el7 rh-nodejs10-nodejs-nodemon.noarch 0:1.18.3-3.el7 Complete! package nss_wrapper is not installed The command '/bin/sh -c yum install -y centos-release-scl-rh && ( [ "rh-${NAME}${NODEJS_VERSION}" != "${NODEJS_SCL}" ] && yum remove -y ${NODEJS_SCL}\* || : ) && INSTALL_PKGS="rh-nodejs${NODEJS_VERSION} rh-nodejs${NODEJS_VERSION}-npm rh-nodejs${NODEJS_VERSION}-nodejs-nodemon nss_wrapper" && ln -s /usr/lib/node_modules/nodemon/bin/nodemon.js /usr/bin/nodemon && yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS && rpm -V $INSTALL_PKGS && yum -y clean all --enablerepo='*'' returned a non-zero code: 1 make[1]: *** [10] Error 1 make[1]: Leaving directory `/tmp/daily_scl_tests/s2i-nodejs-container' ``` Do you know, where it could be a problem? Did you remove nss_wrapper from CentOS repositories? -- Petr Hracek email: phracek(a)redhat.com IRC phracek #brno #software-collections #tft #core-services #core-services-db GPG 0xA9FC9AC4 Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, Czech Republic

3 3

annobin not matching gcc
by Vladimir Benes 16 Apr '20

16 Apr '20

Hi all, any idea why is annobin not matching gcc in 8-stream? NetworkManager compilation ends like this: checking for gcc... gcc checking whether the C compiler works... no configure: error: in `/root/nm- build/NetworkManager/contrib/fedora/rpm/NetworkManager.20200416- 093131.EsV6WT/BUILD/NetworkManager-1.25.0': configure: error: C compiler cannot create executables See `config.log' for more details error: Bad exit status from /var/tmp/rpm-tmp.mwThzy (%build) downgrading annobin fixes the issue. Can we somehow make sure these two are in sync all the time? Thanks, Vladimir

1 0

Issues with scl
by Katerina Foniok 15 Apr '20

15 Apr '20

Hello guys, we are often having problems with Jenkins job [1] where sometimes we are not able to use scl command. It just says `scl: command not found`. We are not sure how we could solve this problem or if it is some problem with nodes and you could help us with that? The job is running on devtools-ci-slave04. Thank you for your time. Have a nice day, Katka [1] https://ci.centos.org/job/devtools-rh-che-periodic-prod-1a/2340/console

3 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users April 2020