[CentOS-devel] New CBS/SIG signing process

Thu Mar 12 09:35:38 UTC 2020
Alfredo Moralejo Alonso <amoralej at redhat.com>

On Tue, Mar 10, 2020 at 4:18 PM Fabian Arrotin <arrfab at centos.org> wrote:

> Hi all (especially SIG members/contributors) ,
>
> As announced by Jim some time ago, we wanted to redesign the current
> signing process that is in place for https://cbs.centos.org for CentOS 6
> and 7 (so quite some years now)
>
> The goal is to automate as much as possible the workflow, and working
> for all releases (yeah for CentOS 8 and Stream)
>
> Thomas and myself have worked on the following idea and we're now happy
> with the results (in our Dev environment) :
> - when someone builds a pkg, and that it's tagged to -testing, koji
> sends a notification (through koji callback -
> https://fedoraproject.org/wiki/Koji/WritingKojiCode#Event_Plugin) to a bus
> - signing machine listens to the bus, process the tag and push directly
> to buildlogs
> - when someone tag-build said pkg to -release, the node signs it with
> correct gpg key id (from the SIG), push generated repository (including
> for debuginfo/src.rpm packages) to mirror CDN
>
>
Any recommendation for eficient bulk tagging. In some cases we (CloudSIG)
tag more that 300 builds together, will kojiclient multicall api feature
help the new system to do the sign/push more efficiently (btw, we are
already using that multicall feature when tagging).


> As we were working on such automation, we also confirm that if one uses
> "untag-build", the same rule applies : it triggers automatically koji to
> reprocess the tag/repo and so removes the pkg from mirror (we got some
> requests for that in the past)
>
> The process so would work 24/7, and also have monitoring built-in (so
> that we can detect - the infra team - when something doesn't work, and
> so we can still process the tag again)
>
> We recently upgraded cbs.centos.org to newer koji version that permits
> some of the needed operations for this to happen.
>
> At this stage, we are just missing the SIG GPG private keys, but we were
> promised to have those this week.
>
> We don't have any ETA yet for the migration to new signing system, but
> we were thinking about such plan:
>
> - deploy and configure (automatically) new signing node (this week)
> - import GPG keys in the new system and validate that we can sign (when
> we get those)
> - changes some flags/parameters on tags on cbs.centos.org (this week)
>
>
Currently, there are builds in -testing and -release tags which are not
signed nor shipped (both, centos8 and centos7 multiarch, i.e.), will this
pending builds be processed somehow when moving to the new system, or an
untag/tag will be needed?



> Once we have this in place, we'd like to start validating the signing
> process but *not* pushing to mirror CDN (yet). Same for the bus : we'd
> like to process all tags in advance as we'll have to import RPM
> signatures back in koji/cbs.
>
> After full validation, we announce that new system would be fully in
> place, and kojihub restarted with the notification plugin.
> From that point, the system would be fully operational and so when SIG
> members would build/tag pkgs, that would trigger the push (and signing
> if needed for -release tag)
>
> Worth knowing that with the update to some components in the chain, the
> directory layout will change a little bit (like people are already used
> to if using Fedora and Epel) : the Packages will be splitted by first
> letter.
> Here is an example from our testing :
>
>  └── storage
>         └── x86_64
>             └── gluster-7
>                 ├── Packages
>                 │   ├── g
>                 │   │   ├── glusterfs-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-api-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-api-devel-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-cli-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-client-xlators-7.2-1.el7.x86_64.rpm
>                 │   │   ├──
> glusterfs-cloudsync-plugins-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-devel-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-events-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-extra-xlators-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-fuse-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-geo-replication-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-libs-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-rdma-7.2-1.el7.x86_64.rpm
>                 │   │   ├── glusterfs-resource-agents-7.2-1.el7.noarch.rpm
>                 │   │   ├── glusterfs-server-7.2-1.el7.x86_64.rpm
>                 │   │   └── glusterfs-thin-arbiter-7.2-1.el7.x86_64.rpm
>                 │   ├── p
>                 │   │   └── python2-gluster-7.2-1.el7.x86_64.rpm
>                 │   └── u
>                 │       ├── userspace-rcu-0.7.16-3.el7.x86_64.rpm
>                 │       └── userspace-rcu-devel-0.7.16-3.el7.x86_64.rpm
>                 └── repodata
>                     ├──
>
> 047503df3313d596e14a1a0d097b6c2261b26690000ecd4260d212a6f5e5632d-other.sqlite.bz2
>                     ├──
>
> 34c1f5dd954099816d6cce289e52f96e61cc2e6430e13072194e35c063b0e3da-primary.xml.gz
>                     ├──
>
> 6293e07e4c5a7acc2c610beb5acb84b996329ef3c6d61c03e375f2afc6a71536-filelists.sqlite.bz2
>                     ├──
>
> 6f89e8b43fb7e384f86f041c4de84d9afeb15dca981559b15230cb48887095db-other.xml.gz
>                     ├──
>
> bef236563d185417fbda48e3b4dd1d62714d1af2b7a7d6dc1a7213af5de1aa9b-primary.sqlite.bz2
>                     ├──
>
> fb66c148e8e2738bd5d19b88d1b3507b0383aaab0938de3809b399babb1a0b44-filelists.xml.gz
>                     ├── repomd.xml
>                     └── repomd.xml.asc
>
> From a usage perspective that will not change the way pkgs can be
> consumed, as repodata will always be at the right place, but in the case
> of SCLs that will mean no subdirectory like before, but instead just
> same "first letter" name (while repodata will be global)
>
> So in the current stage, we'd like to deploy and fully validate the
> signing setup and we'll announce on this list when we'll fully switch to
> new system.
>
> Also worth noting that with the new system in place, in theory we'll
> *not* use the old cbs-content-control git repo (as we'll query
> automatically koji for tags, enabled architectures, etc) so tagging to
> -testing and -release should trigger automatically the "push to mirrors"
> step
>
> If you have questions, comments, feel free to answer in this thread
> and/or #centos-devel on irc.freenode.net
>
> Kind Regards,
>
> Thomas and Fabian
>
> --
> Fabian Arrotin
> The CentOS Project | https://www.centos.org
> gpg key: 17F3B7A1 | twitter: @arrfab
>
> _______________________________________________
> CentOS-devel mailing list
> CentOS-devel at centos.org
> https://lists.centos.org/mailman/listinfo/centos-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20200312/ef538f20/attachment-0007.html>