<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 10, 2020 at 4:18 PM Fabian Arrotin <<a href="mailto:arrfab@centos.org">arrfab@centos.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi all (especially SIG members/contributors) ,<br>
<br>
As announced by Jim some time ago, we wanted to redesign the current<br>
signing process that is in place for <a href="https://cbs.centos.org" rel="noreferrer" target="_blank">https://cbs.centos.org</a> for CentOS 6<br>
and 7 (so quite some years now)<br>
<br>
The goal is to automate as much as possible the workflow, and working<br>
for all releases (yeah for CentOS 8 and Stream)<br>
<br>
Thomas and myself have worked on the following idea and we're now happy<br>
with the results (in our Dev environment) :<br>
- when someone builds a pkg, and that it's tagged to -testing, koji<br>
sends a notification (through koji callback -<br>
<a href="https://fedoraproject.org/wiki/Koji/WritingKojiCode#Event_Plugin" rel="noreferrer" target="_blank">https://fedoraproject.org/wiki/Koji/WritingKojiCode#Event_Plugin</a>) to a bus<br>
- signing machine listens to the bus, process the tag and push directly<br>
to buildlogs<br>
- when someone tag-build said pkg to -release, the node signs it with<br>
correct gpg key id (from the SIG), push generated repository (including<br>
for debuginfo/src.rpm packages) to mirror CDN<br>
<br></blockquote><div><br></div><div>Any recommendation for eficient bulk tagging. In some cases we (CloudSIG) tag more that 300 builds together, will kojiclient multicall api feature help the new system to do the sign/push more efficiently (btw, we are already using that multicall feature when tagging). <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
As we were working on such automation, we also confirm that if one uses<br>
"untag-build", the same rule applies : it triggers automatically koji to<br>
reprocess the tag/repo and so removes the pkg from mirror (we got some<br>
requests for that in the past)<br>
<br>
The process so would work 24/7, and also have monitoring built-in (so<br>
that we can detect - the infra team - when something doesn't work, and<br>
so we can still process the tag again)<br>
<br>
We recently upgraded <a href="http://cbs.centos.org" rel="noreferrer" target="_blank">cbs.centos.org</a> to newer koji version that permits<br>
some of the needed operations for this to happen.<br>
<br>
At this stage, we are just missing the SIG GPG private keys, but we were<br>
promised to have those this week.<br>
<br>
We don't have any ETA yet for the migration to new signing system, but<br>
we were thinking about such plan:<br>
<br>
- deploy and configure (automatically) new signing node (this week)<br>
- import GPG keys in the new system and validate that we can sign (when<br>
we get those)<br>
- changes some flags/parameters on tags on <a href="http://cbs.centos.org" rel="noreferrer" target="_blank">cbs.centos.org</a> (this week)<br>
<br></blockquote><div><br></div><div>Currently, there are builds in -testing and -release tags which are not signed nor shipped (both, centos8 and centos7 multiarch, i.e.), will this pending builds be processed somehow when moving to the new system, or an untag/tag will be needed?<br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Once we have this in place, we'd like to start validating the signing<br>
process but *not* pushing to mirror CDN (yet). Same for the bus : we'd<br>
like to process all tags in advance as we'll have to import RPM<br>
signatures back in koji/cbs.<br>
<br>
After full validation, we announce that new system would be fully in<br>
place, and kojihub restarted with the notification plugin.<br>
>From that point, the system would be fully operational and so when SIG<br>
members would build/tag pkgs, that would trigger the push (and signing<br>
if needed for -release tag)<br>
<br>
Worth knowing that with the update to some components in the chain, the<br>
directory layout will change a little bit (like people are already used<br>
to if using Fedora and Epel) : the Packages will be splitted by first<br>
letter.<br>
Here is an example from our testing :<br>
<br>
└── storage<br>
└── x86_64<br>
└── gluster-7<br>
├── Packages<br>
│ ├── g<br>
│ │ ├── glusterfs-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-api-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-api-devel-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-cli-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-client-xlators-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-cloudsync-plugins-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-devel-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-events-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-extra-xlators-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-fuse-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-geo-replication-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-libs-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-rdma-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-resource-agents-7.2-1.el7.noarch.rpm<br>
│ │ ├── glusterfs-server-7.2-1.el7.x86_64.rpm<br>
│ │ └── glusterfs-thin-arbiter-7.2-1.el7.x86_64.rpm<br>
│ ├── p<br>
│ │ └── python2-gluster-7.2-1.el7.x86_64.rpm<br>
│ └── u<br>
│ ├── userspace-rcu-0.7.16-3.el7.x86_64.rpm<br>
│ └── userspace-rcu-devel-0.7.16-3.el7.x86_64.rpm<br>
└── repodata<br>
├──<br>
047503df3313d596e14a1a0d097b6c2261b26690000ecd4260d212a6f5e5632d-other.sqlite.bz2<br>
├──<br>
34c1f5dd954099816d6cce289e52f96e61cc2e6430e13072194e35c063b0e3da-primary.xml.gz<br>
├──<br>
6293e07e4c5a7acc2c610beb5acb84b996329ef3c6d61c03e375f2afc6a71536-filelists.sqlite.bz2<br>
├──<br>
6f89e8b43fb7e384f86f041c4de84d9afeb15dca981559b15230cb48887095db-other.xml.gz<br>
├──<br>
bef236563d185417fbda48e3b4dd1d62714d1af2b7a7d6dc1a7213af5de1aa9b-primary.sqlite.bz2<br>
├──<br>
fb66c148e8e2738bd5d19b88d1b3507b0383aaab0938de3809b399babb1a0b44-filelists.xml.gz<br>
├── repomd.xml<br>
└── repomd.xml.asc<br>
<br>
>From a usage perspective that will not change the way pkgs can be<br>
consumed, as repodata will always be at the right place, but in the case<br>
of SCLs that will mean no subdirectory like before, but instead just<br>
same "first letter" name (while repodata will be global)<br>
<br>
So in the current stage, we'd like to deploy and fully validate the<br>
signing setup and we'll announce on this list when we'll fully switch to<br>
new system.<br>
<br>
Also worth noting that with the new system in place, in theory we'll<br>
*not* use the old cbs-content-control git repo (as we'll query<br>
automatically koji for tags, enabled architectures, etc) so tagging to<br>
-testing and -release should trigger automatically the "push to mirrors"<br>
step<br>
<br>
If you have questions, comments, feel free to answer in this thread<br>
and/or #centos-devel on <a href="http://irc.freenode.net" rel="noreferrer" target="_blank">irc.freenode.net</a><br>
<br>
Kind Regards,<br>
<br>
Thomas and Fabian<br>
<br>
-- <br>
Fabian Arrotin<br>
The CentOS Project | <a href="https://www.centos.org" rel="noreferrer" target="_blank">https://www.centos.org</a><br>
gpg key: 17F3B7A1 | twitter: @arrfab<br>
<br>
_______________________________________________<br>
CentOS-devel mailing list<br>
<a href="mailto:CentOS-devel@centos.org" target="_blank">CentOS-devel@centos.org</a><br>
<a href="https://lists.centos.org/mailman/listinfo/centos-devel" rel="noreferrer" target="_blank">https://lists.centos.org/mailman/listinfo/centos-devel</a><br>
</blockquote></div></div>