Greetings everyone This email is to update you all on some of the efforts we are about to put/putting in place within the CentOS CI Infrastructure. Some of the things have been long due and we realize that. For a couple of years, our efforts have mainly been under keeping the fires away and general maintenance (adding new releases, arches, making things in the background a bit sophisticated, etc). Right now we have started working on some things that you would notice and will impact you (in a better way hopefully!) and we will be keeping you updated.
* Monitoring - Done * A couple of months ago, All I had was basic ansible scripts to show me data, and that was my way of monitoring things. As one would expect, I often missed things, and email on the list would pop up with `insufficient nodes` or `no space on Jenkins master`. While we tried to be relatively quick on the resolution, we needed a better way. In early February, Fabian (Thank you) set up a monitoring system (added ci.centos.org to CentOS Infra Zabbix instance) which in turn reduced such incidence by a lot. I hope you have not noticed those emails/complaints in the last 2-3 months. We will be adding monitoring to more areas as we go.
* OpenShift cluster upgrade - In Progress * https://console.apps.ci.centos.org:8443/ is using openshift version 3.6 which has been outdated for a while. We are upgrading the cluster to OpenShift 4.3 which handles some of the group permissions in a better way. Apart from the shiny new deployment which has a bunch of improvements/features, it will allow us to give admin access to more members easily which in turn will decrease the wait time for new accounts/users (it won't wait on a single person to do all the work).
* Deployment of OpenNebula for VM checkouts through cico - Upcoming * We will be deploying OpenNebula and will extend duffy api to support VMs in the check out process. This will be useful for the jobs who don't need bare metals to do their tests and need slightly less resources. This will also enable us to support Fedora as an on-demand offering distro (in vm) that can be used to run your test. Ideally, it would look like `cico node get --type vm` or `cico node get --type vm --distro fedora -- version rawhide`. This needs more grooming and planning before we start planning exact images/distros and ways but I am hopeful that it will make your experience better.
We are hoping to update you all with the things going on in the background pretty frequently and the updates will show up as weekly emails `progress of $initiatives_mentioned_above` and "probably" monthly email of some matrices of things that happened in that month (not sure yet). Once we have upgraded the openshift cluster, we will need your help in migrating the jobs from legacy `ci.centos.org` to openshift. You will get a namespace where you can control all things related. This will introduce easier workflow and management for admins and more freedom to you, tenants! If you want to discuss things more, feel free to contact us or catch us at #centos-ci@Freenode
Thank you! Be safe