On Tue, Mar 10, 2020 at 4:18 PM Fabian Arrotin <arrfab at centos.org> wrote: > Hi all (especially SIG members/contributors) , > > As announced by Jim some time ago, we wanted to redesign the current > signing process that is in place for https://cbs.centos.org for CentOS 6 > and 7 (so quite some years now) > > The goal is to automate as much as possible the workflow, and working > for all releases (yeah for CentOS 8 and Stream) > > Thomas and myself have worked on the following idea and we're now happy > with the results (in our Dev environment) : > - when someone builds a pkg, and that it's tagged to -testing, koji > sends a notification (through koji callback - > https://fedoraproject.org/wiki/Koji/WritingKojiCode#Event_Plugin) to a bus > - signing machine listens to the bus, process the tag and push directly > to buildlogs > - when someone tag-build said pkg to -release, the node signs it with > correct gpg key id (from the SIG), push generated repository (including > for debuginfo/src.rpm packages) to mirror CDN > > Any recommendation for eficient bulk tagging. In some cases we (CloudSIG) tag more that 300 builds together, will kojiclient multicall api feature help the new system to do the sign/push more efficiently (btw, we are already using that multicall feature when tagging). > As we were working on such automation, we also confirm that if one uses > "untag-build", the same rule applies : it triggers automatically koji to > reprocess the tag/repo and so removes the pkg from mirror (we got some > requests for that in the past) > > The process so would work 24/7, and also have monitoring built-in (so > that we can detect - the infra team - when something doesn't work, and > so we can still process the tag again) > > We recently upgraded cbs.centos.org to newer koji version that permits > some of the needed operations for this to happen. > > At this stage, we are just missing the SIG GPG private keys, but we were > promised to have those this week. > > We don't have any ETA yet for the migration to new signing system, but > we were thinking about such plan: > > - deploy and configure (automatically) new signing node (this week) > - import GPG keys in the new system and validate that we can sign (when > we get those) > - changes some flags/parameters on tags on cbs.centos.org (this week) > > Currently, there are builds in -testing and -release tags which are not signed nor shipped (both, centos8 and centos7 multiarch, i.e.), will this pending builds be processed somehow when moving to the new system, or an untag/tag will be needed? > Once we have this in place, we'd like to start validating the signing > process but *not* pushing to mirror CDN (yet). Same for the bus : we'd > like to process all tags in advance as we'll have to import RPM > signatures back in koji/cbs. > > After full validation, we announce that new system would be fully in > place, and kojihub restarted with the notification plugin. > From that point, the system would be fully operational and so when SIG > members would build/tag pkgs, that would trigger the push (and signing > if needed for -release tag) > > Worth knowing that with the update to some components in the chain, the > directory layout will change a little bit (like people are already used > to if using Fedora and Epel) : the Packages will be splitted by first > letter. > Here is an example from our testing : > > └── storage > └── x86_64 > └── gluster-7 > ├── Packages > │ ├── g > │ │ ├── glusterfs-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-api-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-api-devel-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-cli-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-client-xlators-7.2-1.el7.x86_64.rpm > │ │ ├── > glusterfs-cloudsync-plugins-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-devel-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-events-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-extra-xlators-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-fuse-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-geo-replication-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-libs-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-rdma-7.2-1.el7.x86_64.rpm > │ │ ├── glusterfs-resource-agents-7.2-1.el7.noarch.rpm > │ │ ├── glusterfs-server-7.2-1.el7.x86_64.rpm > │ │ └── glusterfs-thin-arbiter-7.2-1.el7.x86_64.rpm > │ ├── p > │ │ └── python2-gluster-7.2-1.el7.x86_64.rpm > │ └── u > │ ├── userspace-rcu-0.7.16-3.el7.x86_64.rpm > │ └── userspace-rcu-devel-0.7.16-3.el7.x86_64.rpm > └── repodata > ├── > > 047503df3313d596e14a1a0d097b6c2261b26690000ecd4260d212a6f5e5632d-other.sqlite.bz2 > ├── > > 34c1f5dd954099816d6cce289e52f96e61cc2e6430e13072194e35c063b0e3da-primary.xml.gz > ├── > > 6293e07e4c5a7acc2c610beb5acb84b996329ef3c6d61c03e375f2afc6a71536-filelists.sqlite.bz2 > ├── > > 6f89e8b43fb7e384f86f041c4de84d9afeb15dca981559b15230cb48887095db-other.xml.gz > ├── > > bef236563d185417fbda48e3b4dd1d62714d1af2b7a7d6dc1a7213af5de1aa9b-primary.sqlite.bz2 > ├── > > fb66c148e8e2738bd5d19b88d1b3507b0383aaab0938de3809b399babb1a0b44-filelists.xml.gz > ├── repomd.xml > └── repomd.xml.asc > > From a usage perspective that will not change the way pkgs can be > consumed, as repodata will always be at the right place, but in the case > of SCLs that will mean no subdirectory like before, but instead just > same "first letter" name (while repodata will be global) > > So in the current stage, we'd like to deploy and fully validate the > signing setup and we'll announce on this list when we'll fully switch to > new system. > > Also worth noting that with the new system in place, in theory we'll > *not* use the old cbs-content-control git repo (as we'll query > automatically koji for tags, enabled architectures, etc) so tagging to > -testing and -release should trigger automatically the "push to mirrors" > step > > If you have questions, comments, feel free to answer in this thread > and/or #centos-devel on irc.freenode.net > > Kind Regards, > > Thomas and Fabian > > -- > Fabian Arrotin > The CentOS Project | https://www.centos.org > gpg key: 17F3B7A1 | twitter: @arrfab > > _______________________________________________ > CentOS-devel mailing list > CentOS-devel at centos.org > https://lists.centos.org/mailman/listinfo/centos-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20200312/ef538f20/attachment-0005.html>