Hi all (especially SIG members/contributors) ,
As announced by Jim some time ago, we wanted to redesign the current signing process that is in place for https://cbs.centos.org for CentOS 6 and 7 (so quite some years now)
The goal is to automate as much as possible the workflow, and working for all releases (yeah for CentOS 8 and Stream)
Thomas and myself have worked on the following idea and we're now happy with the results (in our Dev environment) : - when someone builds a pkg, and that it's tagged to -testing, koji sends a notification (through koji callback - https://fedoraproject.org/wiki/Koji/WritingKojiCode#Event_Plugin) to a bus - signing machine listens to the bus, process the tag and push directly to buildlogs - when someone tag-build said pkg to -release, the node signs it with correct gpg key id (from the SIG), push generated repository (including for debuginfo/src.rpm packages) to mirror CDN
As we were working on such automation, we also confirm that if one uses "untag-build", the same rule applies : it triggers automatically koji to reprocess the tag/repo and so removes the pkg from mirror (we got some requests for that in the past)
The process so would work 24/7, and also have monitoring built-in (so that we can detect - the infra team - when something doesn't work, and so we can still process the tag again)
We recently upgraded cbs.centos.org to newer koji version that permits some of the needed operations for this to happen.
At this stage, we are just missing the SIG GPG private keys, but we were promised to have those this week.
We don't have any ETA yet for the migration to new signing system, but we were thinking about such plan:
- deploy and configure (automatically) new signing node (this week) - import GPG keys in the new system and validate that we can sign (when we get those) - changes some flags/parameters on tags on cbs.centos.org (this week)
Once we have this in place, we'd like to start validating the signing process but *not* pushing to mirror CDN (yet). Same for the bus : we'd like to process all tags in advance as we'll have to import RPM signatures back in koji/cbs.
After full validation, we announce that new system would be fully in place, and kojihub restarted with the notification plugin. From that point, the system would be fully operational and so when SIG members would build/tag pkgs, that would trigger the push (and signing if needed for -release tag)
Worth knowing that with the update to some components in the chain, the directory layout will change a little bit (like people are already used to if using Fedora and Epel) : the Packages will be splitted by first letter. Here is an example from our testing :
└── storage └── x86_64 └── gluster-7 ├── Packages │ ├── g │ │ ├── glusterfs-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-api-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-api-devel-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-cli-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-client-xlators-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-cloudsync-plugins-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-devel-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-events-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-extra-xlators-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-fuse-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-geo-replication-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-libs-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-rdma-7.2-1.el7.x86_64.rpm │ │ ├── glusterfs-resource-agents-7.2-1.el7.noarch.rpm │ │ ├── glusterfs-server-7.2-1.el7.x86_64.rpm │ │ └── glusterfs-thin-arbiter-7.2-1.el7.x86_64.rpm │ ├── p │ │ └── python2-gluster-7.2-1.el7.x86_64.rpm │ └── u │ ├── userspace-rcu-0.7.16-3.el7.x86_64.rpm │ └── userspace-rcu-devel-0.7.16-3.el7.x86_64.rpm └── repodata ├── 047503df3313d596e14a1a0d097b6c2261b26690000ecd4260d212a6f5e5632d-other.sqlite.bz2 ├── 34c1f5dd954099816d6cce289e52f96e61cc2e6430e13072194e35c063b0e3da-primary.xml.gz ├── 6293e07e4c5a7acc2c610beb5acb84b996329ef3c6d61c03e375f2afc6a71536-filelists.sqlite.bz2 ├── 6f89e8b43fb7e384f86f041c4de84d9afeb15dca981559b15230cb48887095db-other.xml.gz ├── bef236563d185417fbda48e3b4dd1d62714d1af2b7a7d6dc1a7213af5de1aa9b-primary.sqlite.bz2 ├── fb66c148e8e2738bd5d19b88d1b3507b0383aaab0938de3809b399babb1a0b44-filelists.xml.gz ├── repomd.xml └── repomd.xml.asc
From a usage perspective that will not change the way pkgs can be consumed, as repodata will always be at the right place, but in the case of SCLs that will mean no subdirectory like before, but instead just same "first letter" name (while repodata will be global)
So in the current stage, we'd like to deploy and fully validate the signing setup and we'll announce on this list when we'll fully switch to new system.
Also worth noting that with the new system in place, in theory we'll *not* use the old cbs-content-control git repo (as we'll query automatically koji for tags, enabled architectures, etc) so tagging to -testing and -release should trigger automatically the "push to mirrors" step
If you have questions, comments, feel free to answer in this thread and/or #centos-devel on irc.freenode.net
Kind Regards,
Thomas and Fabian