Greetings everyone
This email is to update you all on some of the efforts we are about to
put/putting in place within the CentOS CI Infrastructure.
Some of the things have been long due and we realize that. For a
couple of years, our efforts have mainly been under keeping the fires
away and general maintenance (adding new releases, arches, making
things in the background a bit sophisticated, etc). Right now we have
started working on some things that you would notice and will impact
you (in a better way hopefully!) and we will be keeping you updated.
* Monitoring - Done
* A couple of months ago, All I had was basic ansible scripts to
show me data, and that was my way of monitoring things. As one would
expect, I often missed things, and email on the list would pop up with
`insufficient nodes` or `no space on Jenkins master`. While we tried
to be relatively quick on the resolution, we needed a better way. In
early February, Fabian (Thank you) set up a monitoring system (added
ci.centos.org to CentOS Infra Zabbix instance) which in turn reduced
such incidence by a lot. I hope you have not noticed those
emails/complaints in the last 2-3 months. We will be adding monitoring
to more areas as we go.
* OpenShift cluster upgrade - In Progress
* https://console.apps.ci.centos.org:8443/ is using openshift
version 3.6 which has been outdated for a while. We are upgrading the
cluster to OpenShift 4.3 which handles some of the group permissions
in a better way. Apart from the shiny new deployment which has a bunch
of improvements/features, it will allow us to give admin access to
more members easily which in turn will decrease the wait time for new
accounts/users (it won't wait on a single person to do all the work).
* Deployment of OpenNebula for VM checkouts through cico - Upcoming
* We will be deploying OpenNebula and will extend duffy api to
support VMs in the check out process. This will be useful for the jobs
who don't need bare metals to do their tests and need slightly less
resources. This will also enable us to support Fedora as an on-demand
offering distro (in vm) that can be used to run your test. Ideally, it
would look like `cico node get --type vm` or `cico node get --type vm
--distro fedora -- version rawhide`. This needs more grooming and
planning before we start planning exact images/distros and ways but I
am hopeful that it will make your experience better.
We are hoping to update you all with the things going on in the
background pretty frequently and the updates will show up as weekly
emails `progress of $initiatives_mentioned_above` and "probably"
monthly email of some matrices of things that happened in that month
(not sure yet).
Once we have upgraded the openshift cluster, we will need your help in
migrating the jobs from legacy `ci.centos.org` to openshift. You will
get a namespace where you can control all things related. This will
introduce easier workflow and management for admins and more freedom
to you, tenants!
If you want to discuss things more, feel free to contact us or catch
us at #centos-ci at Freenode
Thank you!
Be safe
--
@siddharthvipul
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat