<div dir="ltr">Thanks for the update, looking forward to this!<br><div>Also thanks for your hard work on this!</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mar 10 mar 2020 alle ore 16:18 Fabian Arrotin <<a href="mailto:arrfab@centos.org">arrfab@centos.org</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi all (especially SIG members/contributors) ,<br>
<br>
As announced by Jim some time ago, we wanted to redesign the current<br>
signing process that is in place for <a href="https://cbs.centos.org" rel="noreferrer" target="_blank">https://cbs.centos.org</a> for CentOS 6<br>
and 7 (so quite some years now)<br>
<br>
The goal is to automate as much as possible the workflow, and working<br>
for all releases (yeah for CentOS 8 and Stream)<br>
<br>
Thomas and myself have worked on the following idea and we're now happy<br>
with the results (in our Dev environment) :<br>
- when someone builds a pkg, and that it's tagged to -testing, koji<br>
sends a notification (through koji callback -<br>
<a href="https://fedoraproject.org/wiki/Koji/WritingKojiCode#Event_Plugin" rel="noreferrer" target="_blank">https://fedoraproject.org/wiki/Koji/WritingKojiCode#Event_Plugin</a>) to a bus<br>
- signing machine listens to the bus, process the tag and push directly<br>
to buildlogs<br>
- when someone tag-build said pkg to -release, the node signs it with<br>
correct gpg key id (from the SIG), push generated repository (including<br>
for debuginfo/src.rpm packages) to mirror CDN<br>
<br>
As we were working on such automation, we also confirm that if one uses<br>
"untag-build", the same rule applies : it triggers automatically koji to<br>
reprocess the tag/repo and so removes the pkg from mirror (we got some<br>
requests for that in the past)<br>
<br>
The process so would work 24/7, and also have monitoring built-in (so<br>
that we can detect - the infra team - when something doesn't work, and<br>
so we can still process the tag again)<br>
<br>
We recently upgraded <a href="http://cbs.centos.org" rel="noreferrer" target="_blank">cbs.centos.org</a> to newer koji version that permits<br>
some of the needed operations for this to happen.<br>
<br>
At this stage, we are just missing the SIG GPG private keys, but we were<br>
promised to have those this week.<br>
<br>
We don't have any ETA yet for the migration to new signing system, but<br>
we were thinking about such plan:<br>
<br>
- deploy and configure (automatically) new signing node (this week)<br>
- import GPG keys in the new system and validate that we can sign (when<br>
we get those)<br>
- changes some flags/parameters on tags on <a href="http://cbs.centos.org" rel="noreferrer" target="_blank">cbs.centos.org</a> (this week)<br>
<br>
Once we have this in place, we'd like to start validating the signing<br>
process but *not* pushing to mirror CDN (yet). Same for the bus : we'd<br>
like to process all tags in advance as we'll have to import RPM<br>
signatures back in koji/cbs.<br>
<br>
After full validation, we announce that new system would be fully in<br>
place, and kojihub restarted with the notification plugin.<br>
>From that point, the system would be fully operational and so when SIG<br>
members would build/tag pkgs, that would trigger the push (and signing<br>
if needed for -release tag)<br>
<br>
Worth knowing that with the update to some components in the chain, the<br>
directory layout will change a little bit (like people are already used<br>
to if using Fedora and Epel) : the Packages will be splitted by first<br>
letter.<br>
Here is an example from our testing :<br>
<br>
└── storage<br>
└── x86_64<br>
└── gluster-7<br>
├── Packages<br>
│ ├── g<br>
│ │ ├── glusterfs-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-api-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-api-devel-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-cli-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-client-xlators-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-cloudsync-plugins-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-devel-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-events-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-extra-xlators-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-fuse-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-geo-replication-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-libs-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-rdma-7.2-1.el7.x86_64.rpm<br>
│ │ ├── glusterfs-resource-agents-7.2-1.el7.noarch.rpm<br>
│ │ ├── glusterfs-server-7.2-1.el7.x86_64.rpm<br>
│ │ └── glusterfs-thin-arbiter-7.2-1.el7.x86_64.rpm<br>
│ ├── p<br>
│ │ └── python2-gluster-7.2-1.el7.x86_64.rpm<br>
│ └── u<br>
│ ├── userspace-rcu-0.7.16-3.el7.x86_64.rpm<br>
│ └── userspace-rcu-devel-0.7.16-3.el7.x86_64.rpm<br>
└── repodata<br>
├──<br>
047503df3313d596e14a1a0d097b6c2261b26690000ecd4260d212a6f5e5632d-other.sqlite.bz2<br>
├──<br>
34c1f5dd954099816d6cce289e52f96e61cc2e6430e13072194e35c063b0e3da-primary.xml.gz<br>
├──<br>
6293e07e4c5a7acc2c610beb5acb84b996329ef3c6d61c03e375f2afc6a71536-filelists.sqlite.bz2<br>
├──<br>
6f89e8b43fb7e384f86f041c4de84d9afeb15dca981559b15230cb48887095db-other.xml.gz<br>
├──<br>
bef236563d185417fbda48e3b4dd1d62714d1af2b7a7d6dc1a7213af5de1aa9b-primary.sqlite.bz2<br>
├──<br>
fb66c148e8e2738bd5d19b88d1b3507b0383aaab0938de3809b399babb1a0b44-filelists.xml.gz<br>
├── repomd.xml<br>
└── repomd.xml.asc<br>
<br>
>From a usage perspective that will not change the way pkgs can be<br>
consumed, as repodata will always be at the right place, but in the case<br>
of SCLs that will mean no subdirectory like before, but instead just<br>
same "first letter" name (while repodata will be global)<br>
<br>
So in the current stage, we'd like to deploy and fully validate the<br>
signing setup and we'll announce on this list when we'll fully switch to<br>
new system.<br>
<br>
Also worth noting that with the new system in place, in theory we'll<br>
*not* use the old cbs-content-control git repo (as we'll query<br>
automatically koji for tags, enabled architectures, etc) so tagging to<br>
-testing and -release should trigger automatically the "push to mirrors"<br>
step<br>
<br>
If you have questions, comments, feel free to answer in this thread<br>
and/or #centos-devel on <a href="http://irc.freenode.net" rel="noreferrer" target="_blank">irc.freenode.net</a><br>
<br>
Kind Regards,<br>
<br>
Thomas and Fabian<br>
<br>
-- <br>
Fabian Arrotin<br>
The CentOS Project | <a href="https://www.centos.org" rel="noreferrer" target="_blank">https://www.centos.org</a><br>
gpg key: 17F3B7A1 | twitter: @arrfab<br>
<br>
_______________________________________________<br>
CentOS-devel mailing list<br>
<a href="mailto:CentOS-devel@centos.org" target="_blank">CentOS-devel@centos.org</a><br>
<a href="https://lists.centos.org/mailman/listinfo/centos-devel" rel="noreferrer" target="_blank">https://lists.centos.org/mailman/listinfo/centos-devel</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:capitalize"><span>Sandro</span> <span>Bonazzola</span><span style="text-transform:uppercase;color:rgb(170,170,170);margin:0px"></span></p><p style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:12px;margin:0px;text-transform:capitalize"><span>MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV</span></p><p style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;margin:0px 0px 4px;font-size:12px"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>EMEA</span></a></p><div style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:medium;margin-bottom:4px"></div><p style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;margin:0px;font-size:12px"><span style="margin:0px;padding:0px"><a href="mailto:sbonazzo@redhat.com" style="color:rgb(0,0,0);margin:0px" target="_blank">sbonazzo@redhat.com</a> </span></p><div style="margin-top:12px"><table border="0"><tbody><tr><td width="100px"><a href="https://www.redhat.com/" target="_blank"><font color="#000000" face="RedHatText, sans-serif" size="3"><img src="https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/f5445ae0c9ddafd5b2f1836854d7416a/Logo-RedHat-Email.png" width="90" height="auto"></font></a></td></tr></tbody></table><font color="#000000" face="arial, sans-serif" size="1"><b><a href="https://mojo.redhat.com/docs/DOC-1199578" target="_blank">Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.</a></b></font><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>