[CentOS-devel] New CBS/SIG signing process

Tue Mar 10 15:17:42 UTC 2020
Fabian Arrotin <arrfab at centos.org>

Hi all (especially SIG members/contributors) ,

As announced by Jim some time ago, we wanted to redesign the current
signing process that is in place for https://cbs.centos.org for CentOS 6
and 7 (so quite some years now)

The goal is to automate as much as possible the workflow, and working
for all releases (yeah for CentOS 8 and Stream)

Thomas and myself have worked on the following idea and we're now happy
with the results (in our Dev environment) :
- when someone builds a pkg, and that it's tagged to -testing, koji
sends a notification (through koji callback -
https://fedoraproject.org/wiki/Koji/WritingKojiCode#Event_Plugin) to a bus
- signing machine listens to the bus, process the tag and push directly
to buildlogs
- when someone tag-build said pkg to -release, the node signs it with
correct gpg key id (from the SIG), push generated repository (including
for debuginfo/src.rpm packages) to mirror CDN

As we were working on such automation, we also confirm that if one uses
"untag-build", the same rule applies : it triggers automatically koji to
reprocess the tag/repo and so removes the pkg from mirror (we got some
requests for that in the past)

The process so would work 24/7, and also have monitoring built-in (so
that we can detect - the infra team - when something doesn't work, and
so we can still process the tag again)

We recently upgraded cbs.centos.org to newer koji version that permits
some of the needed operations for this to happen.

At this stage, we are just missing the SIG GPG private keys, but we were
promised to have those this week.

We don't have any ETA yet for the migration to new signing system, but
we were thinking about such plan:

- deploy and configure (automatically) new signing node (this week)
- import GPG keys in the new system and validate that we can sign (when
we get those)
- changes some flags/parameters on tags on cbs.centos.org (this week)

Once we have this in place, we'd like to start validating the signing
process but *not* pushing to mirror CDN (yet). Same for the bus : we'd
like to process all tags in advance as we'll have to import RPM
signatures back in koji/cbs.

After full validation, we announce that new system would be fully in
place, and kojihub restarted with the notification plugin.
From that point, the system would be fully operational and so when SIG
members would build/tag pkgs, that would trigger the push (and signing
if needed for -release tag)

Worth knowing that with the update to some components in the chain, the
directory layout will change a little bit (like people are already used
to if using Fedora and Epel) : the Packages will be splitted by first
letter.
Here is an example from our testing :

 └── storage
        └── x86_64
            └── gluster-7
                ├── Packages
                │   ├── g
                │   │   ├── glusterfs-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-api-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-api-devel-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-cli-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-client-xlators-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-cloudsync-plugins-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-devel-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-events-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-extra-xlators-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-fuse-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-geo-replication-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-libs-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-rdma-7.2-1.el7.x86_64.rpm
                │   │   ├── glusterfs-resource-agents-7.2-1.el7.noarch.rpm
                │   │   ├── glusterfs-server-7.2-1.el7.x86_64.rpm
                │   │   └── glusterfs-thin-arbiter-7.2-1.el7.x86_64.rpm
                │   ├── p
                │   │   └── python2-gluster-7.2-1.el7.x86_64.rpm
                │   └── u
                │       ├── userspace-rcu-0.7.16-3.el7.x86_64.rpm
                │       └── userspace-rcu-devel-0.7.16-3.el7.x86_64.rpm
                └── repodata
                    ├──
047503df3313d596e14a1a0d097b6c2261b26690000ecd4260d212a6f5e5632d-other.sqlite.bz2
                    ├──
34c1f5dd954099816d6cce289e52f96e61cc2e6430e13072194e35c063b0e3da-primary.xml.gz
                    ├──
6293e07e4c5a7acc2c610beb5acb84b996329ef3c6d61c03e375f2afc6a71536-filelists.sqlite.bz2
                    ├──
6f89e8b43fb7e384f86f041c4de84d9afeb15dca981559b15230cb48887095db-other.xml.gz
                    ├──
bef236563d185417fbda48e3b4dd1d62714d1af2b7a7d6dc1a7213af5de1aa9b-primary.sqlite.bz2
                    ├──
fb66c148e8e2738bd5d19b88d1b3507b0383aaab0938de3809b399babb1a0b44-filelists.xml.gz
                    ├── repomd.xml
                    └── repomd.xml.asc

From a usage perspective that will not change the way pkgs can be
consumed, as repodata will always be at the right place, but in the case
of SCLs that will mean no subdirectory like before, but instead just
same "first letter" name (while repodata will be global)

So in the current stage, we'd like to deploy and fully validate the
signing setup and we'll announce on this list when we'll fully switch to
new system.

Also worth noting that with the new system in place, in theory we'll
*not* use the old cbs-content-control git repo (as we'll query
automatically koji for tags, enabled architectures, etc) so tagging to
-testing and -release should trigger automatically the "push to mirrors"
step

If you have questions, comments, feel free to answer in this thread
and/or #centos-devel on irc.freenode.net

Kind Regards,

Thomas and Fabian

-- 
Fabian Arrotin
The CentOS Project | https://www.centos.org
gpg key: 17F3B7A1 | twitter: @arrfab

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20200310/aa8d667f/attachment-0006.sig>