Hello,
Fabian suggested to post about https://pagure.io/centos-infra/issue/306 so we can have a wider discussion around it. That ticket is specifically for the Hyperscale SIG, but I think this is something that could be generally useful.
Within Hyperscale, we have a need to build a few packages that have dependencies coming from EPEL. While we could rebuild those dependencies within the SIG, that seems both wasteful and undesirable, as it would lead to needless deviation and potentially discourage folks from maintaining stuff in EPEL (where it has a chance to benefit the most people). So, our conclusion was that it would be great to just depend directly on EPEL and have it added to our build targets in CBS so we can build against it.
One concern with doing this is that EPEL is kind of a moving target. This doesn't really apply to Hyperscale (as we track CentOS Stream), but it could be relevant for SIGs tracking CentOS Linux. Still, this would just be a non-default option for a SIG to add to their build tags, so I don't think this will be much of a problem in practice.
From my chat with Fabian I understand this was raised in the past and there had been policy concerns around it. I'd love to get more context around this and see if we can get them addressed. Alternatively, if there are no concerns I'd be happy to entertain suggestions on how to actually do this :) Thanks!
Cheers Davide
On 28/04/2021 22:09, Davide Cavalca via CentOS-devel wrote:
Hello,
Fabian suggested to post about https://pagure.io/centos-infra/issue/306 so we can have a wider discussion around it. That ticket is specifically for the Hyperscale SIG, but I think this is something that could be generally useful.
Within Hyperscale, we have a need to build a few packages that have dependencies coming from EPEL. While we could rebuild those dependencies within the SIG, that seems both wasteful and undesirable, as it would lead to needless deviation and potentially discourage folks from maintaining stuff in EPEL (where it has a chance to benefit the most people). So, our conclusion was that it would be great to just depend directly on EPEL and have it added to our build targets in CBS so we can build against it.
One concern with doing this is that EPEL is kind of a moving target. This doesn't really apply to Hyperscale (as we track CentOS Stream), but it could be relevant for SIGs tracking CentOS Linux. Still, this would just be a non-default option for a SIG to add to their build tags, so I don't think this will be much of a problem in practice.
From my chat with Fabian I understand this was raised in the past and there had been policy concerns around it. I'd love to get more context around this and see if we can get them addressed. Alternatively, if there are no concerns I'd be happy to entertain suggestions on how to actually do this :) Thanks!
Cheers Davide
Just some links to previous threads about this :
https://lists.centos.org/pipermail/centos-devel/2015-September/050896.html
https://lists.centos.org/pipermail/centos-devel/2016-April/051678.html
https://lists.centos.org/pipermail/centos-devel/2019-July/054614.html
So worth reading the three threads and then revisit it maybe ? :)
On Wed, 2021-04-28 at 23:15 +0200, Fabian Arrotin wrote:
Just some links to previous threads about this :
(cut)
So worth reading the three threads and then revisit it maybe ? :)
Thanks Fabian, this is really helpful. I actually think the proposal you had made back then in https://lists.centos.org/pipermail/centos-devel/2019-July/073860.html would work well for this, and addresses most of the concerns I've seen raised throughout those threads.
Specifically: - it does not require rebuilding EPEL onto CBS (which would defeat the entire purpose of this exercise) - it gives SIGs autonomy of decision on whether to consume content from EPEL, rebuild it or ignore it, on a per-package basis - because EPEL is continuously imported and versions that are consumed are tagged, this also solves the "yum downgrade" problem -- even if EPEL upstream moves on, older versions that were consumed in CBS would remain available (unless one untags them explicitly) - OTOH, if a SIG does want to track the latest versions available in EPEL, that is also possible (e.g. Hyperscale would like this as it would expose potential breakage early on)
One potential issue I don't have a clear sense of is the matter of conflicting NVRs that was raised in https://lists.centos.org/pipermail/centos-devel/2019-July/073863.html . This is not a concern for Hyperscale as we set a custom disttag, but it could be for other SIGs that don't do this and also rebuild packages present in EPEL with the same NVR.
Cheers Davide
On 29/04/2021 00:07, Davide Cavalca via CentOS-devel wrote:
On Wed, 2021-04-28 at 23:15 +0200, Fabian Arrotin wrote:
Just some links to previous threads about this :
(cut)
So worth reading the three threads and then revisit it maybe ? :)
Thanks Fabian, this is really helpful. I actually think the proposal you had made back then in https://lists.centos.org/pipermail/centos-devel/2019-July/073860.html would work well for this, and addresses most of the concerns I've seen raised throughout those threads.
Great, I'd like to have other SIGs opinions too but that's a good start ! :-)
Specifically:
- it does not require rebuilding EPEL onto CBS (which would defeat the
entire purpose of this exercise)
Yes ! :-)
- it gives SIGs autonomy of decision on whether to consume content from
EPEL, rebuild it or ignore it, on a per-package basis
That was indeed the idea : each SIG can tag-build for -testing/-release the version they want
- because EPEL is continuously imported and versions that are consumed
are tagged, this also solves the "yum downgrade" problem -- even if EPEL upstream moves on, older versions that were consumed in CBS would remain available (unless one untags them explicitly)
Yes, idea (as explained below) would be to create internal tag just for this and people can tag-build from that tag in their SIG tag[s]
- OTOH, if a SIG does want to track the latest versions available in
EPEL, that is also possible (e.g. Hyperscale would like this as it would expose potential breakage early on)
One potential issue I don't have a clear sense of is the matter of conflicting NVRs that was raised in https://lists.centos.org/pipermail/centos-devel/2019-July/073863.html . This is not a concern for Hyperscale as we set a custom disttag, but it could be for other SIGs that don't do this and also rebuild packages present in EPEL with the same NVR.
Reason why we'd have a consensus on doing this and properly communicated. If we decide to just import existing builds into cbs (can be an internal tag that will never be pushed out for obvious reasons), that will work for pkgs with different ENVR, but not for existing ones. So if SIGs applied some patches before rebuilding an epel pkg, they'd be more than encouraged to do this at the EPEL level. For the others, they can just carry on with existing pkgs and next time, just 'tag-build' the import EPEL packages to satisfy Requires: or BuildRequires:
Yeah, I think this plan sounds fine. It would require making a import script, that imports as things build in epel, but that should be do-able.
kevin
On 29/04/2021 23:07, Kevin Fenzi wrote:
Yeah, I think this plan sounds fine. It would require making a import script, that imports as things build in epel, but that should be do-able.
kevin
I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs.
*but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}).
Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key
Is that the workflow that people wanted to see ? It's true that it would be easy to consume, and even cherry-pick which ENVR of a pkg to have in a repo (so not be forced to upgrade to a newer epel pkg).
If so, can we have +1 from Fedora infra/Fesco about just importing epel pkgs in our koji (to not have to rebuild everything) and so also ship pkgs out (but signed again with SIG gpg pub key for repoclosure in their own repo)
Searching for feedback to make progress on this request and not let it fall in a hole like last time :)
On Wed, May 5, 2021 at 7:59 AM Fabian Arrotin arrfab@centos.org wrote:
On 29/04/2021 23:07, Kevin Fenzi wrote:
Yeah, I think this plan sounds fine. It would require making a import script, that imports as things build in epel, but that should be do-able.
kevin
I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs.
*but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}).
Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key
Is that the workflow that people wanted to see ? It's true that it would be easy to consume, and even cherry-pick which ENVR of a pkg to have in a repo (so not be forced to upgrade to a newer epel pkg).
If so, can we have +1 from Fedora infra/Fesco about just importing epel pkgs in our koji (to not have to rebuild everything) and so also ship pkgs out (but signed again with SIG gpg pub key for repoclosure in their own repo)
Searching for feedback to make progress on this request and not let it fall in a hole like last time :)
I would probably suggest instead that we make release packages depend on epel-release. The package is already shipped in CentOS, so we can have SIG release packages also depend on it. For example, Davide and I are prepared already to make Hyperscale's SIG release package depend on epel-release.
On 05/05/2021 14:04, Neal Gompa wrote:
On Wed, May 5, 2021 at 7:59 AM Fabian Arrotin arrfab@centos.org wrote:
On 29/04/2021 23:07, Kevin Fenzi wrote:
Yeah, I think this plan sounds fine. It would require making a import script, that imports as things build in epel, but that should be do-able.
kevin
I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs.
*but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}).
Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key
Is that the workflow that people wanted to see ? It's true that it would be easy to consume, and even cherry-pick which ENVR of a pkg to have in a repo (so not be forced to upgrade to a newer epel pkg).
If so, can we have +1 from Fedora infra/Fesco about just importing epel pkgs in our koji (to not have to rebuild everything) and so also ship pkgs out (but signed again with SIG gpg pub key for repoclosure in their own repo)
Searching for feedback to make progress on this request and not let it fall in a hole like last time :)
I would probably suggest instead that we make release packages depend on epel-release. The package is already shipped in CentOS, so we can have SIG release packages also depend on it. For example, Davide and I are prepared already to make Hyperscale's SIG release package depend on epel-release.
That would probably work for hyperscale SIG, if you just rely on epel-release to have all Requires: pkgs available at client side .. That doesn't solve the other request about SIGs just willing to import one (or a few) pkg[s] to a specific version for their own SIG and still redistribute it , instead of just rebuilding it (like they do now) .. so original question still apply : can Fesco/Fedora allow this ? :)
PS : due to previous infra policy, myself I had to spend time rebuilding pkgs from epel7/8 to have these available for infra tags , while for a majority of the cases, just tagging an existing epel build would be enough and so win of time ..
On Wed, May 5, 2021 at 8:23 AM Fabian Arrotin arrfab@centos.org wrote:
On 05/05/2021 14:04, Neal Gompa wrote:
On Wed, May 5, 2021 at 7:59 AM Fabian Arrotin arrfab@centos.org wrote:
On 29/04/2021 23:07, Kevin Fenzi wrote:
Yeah, I think this plan sounds fine. It would require making a import script, that imports as things build in epel, but that should be do-able.
kevin
I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs.
*but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}).
Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key
Is that the workflow that people wanted to see ? It's true that it would be easy to consume, and even cherry-pick which ENVR of a pkg to have in a repo (so not be forced to upgrade to a newer epel pkg).
If so, can we have +1 from Fedora infra/Fesco about just importing epel pkgs in our koji (to not have to rebuild everything) and so also ship pkgs out (but signed again with SIG gpg pub key for repoclosure in their own repo)
Searching for feedback to make progress on this request and not let it fall in a hole like last time :)
I would probably suggest instead that we make release packages depend on epel-release. The package is already shipped in CentOS, so we can have SIG release packages also depend on it. For example, Davide and I are prepared already to make Hyperscale's SIG release package depend on epel-release.
That would probably work for hyperscale SIG, if you just rely on epel-release to have all Requires: pkgs available at client side .. That doesn't solve the other request about SIGs just willing to import one (or a few) pkg[s] to a specific version for their own SIG and still redistribute it , instead of just rebuilding it (like they do now) .. so original question still apply : can Fesco/Fedora allow this ? :)
PS : due to previous infra policy, myself I had to spend time rebuilding pkgs from epel7/8 to have these available for infra tags , while for a majority of the cases, just tagging an existing epel build would be enough and so win of time ..
I don't think this is a good solution, but nothing stops you from importing them and resigning them for your own distribution.
I think you're overthinking this and supporting a use-case that SIGs should not be allowed to do. They already can't do this for CentOS base, it should be treated similarly for EPEL, in my opinion.
The general user expectation is that CentOS content should *all* be compatible with EPEL and vice versa. Breaking that expectation creates a source of agony for users.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, May 5, 2021 at 7:59 AM Fabian Arrotin arrfab@centos.org wrote:
On 29/04/2021 23:07, Kevin Fenzi wrote:
Yeah, I think this plan sounds fine. It would require making a import script, that imports as things build in epel, but that should be do-able.
kevin
I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs.
*but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}).
Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key
Is that the workflow that people wanted to see ? It's true that it would be easy to consume, and even cherry-pick which ENVR of a pkg to have in a repo (so not be forced to upgrade to a newer epel pkg).
I'm not sure I see a need to copy+sign+mirror EPEL packages for the SIGs.
I'd rather see people get them from EPEL directly.
--
Kaleb
On 5/5/21 7:30 AM, Kaleb Keithley wrote:
On Wed, May 5, 2021 at 7:59 AM Fabian Arrotin <arrfab@centos.org mailto:arrfab@centos.org> wrote:
On 29/04/2021 23:07, Kevin Fenzi wrote: > Yeah, I think this plan sounds fine. It would require making a import > script, that imports as things build in epel, but that should be > do-able. > > kevin > I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs. *but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}). Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key Is that the workflow that people wanted to see ? It's true that it would be easy to consume, and even cherry-pick which ENVR of a pkg to have in a repo (so not be forced to upgrade to a newer epel pkg).I'm not sure I see a need to copy+sign+mirror EPEL packages for the SIGs.
I'd rather see people get them from EPEL directly.
That is fine .. IF .. you don't need repoclosure on the repo+centos.
I also think it is OK. I guess if you are building against EPEL .. you need to include epel-release as a require for your package .. otherwise .. people with centos installed can not install items from your SIG until they set up EPEL.
That is the whole purpose of including them in the SIG.
If people get the repos needed enabled, I have no issues building directly against EPEL and requiring epel-release in your <SIG>-release package.
On Wed, 2021-05-05 at 13:59 +0200, Fabian Arrotin wrote:
I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs.
*but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}).
Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key
I can think of one downside of this: it would result in packages with the same ENVR, but different signatures and checksums. I know this would be a problem for FB (due to how some of our internal tooling works), but I'm not sure what other side effects it could bring. If we go down this path, would it be possible to *not* resign the packages, and just leave them signed with the EPEL key?
Cheers Davide
On Wed, May 05, 2021 at 03:21:31PM +0000, Davide Cavalca via CentOS-devel wrote:
On Wed, 2021-05-05 at 13:59 +0200, Fabian Arrotin wrote:
I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs.
*but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}).
Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key
I can think of one downside of this: it would result in packages with the same ENVR, but different signatures and checksums. I know this would be a problem for FB (due to how some of our internal tooling works), but I'm not sure what other side effects it could bring. If we go down this path, would it be possible to *not* resign the packages, and just leave them signed with the EPEL key?
There is a koji import-sig call. So, in theory, the scripting could just import the signatures from fedora koji and then cbs could write out needed signed copies for whatever reason.
I think that will require downloading/calling fedora koji directly tho, as the detached signatures are not in the download/repo, only on the koji hub.
Should be possible from a technical side though I would think...
kevin
On 05/05/2021 17:21, Davide Cavalca via CentOS-devel wrote:
On Wed, 2021-05-05 at 13:59 +0200, Fabian Arrotin wrote:
I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs.
*but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}).
Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key
I can think of one downside of this: it would result in packages with the same ENVR, but different signatures and checksums. I know this would be a problem for FB (due to how some of our internal tooling works), but I'm not sure what other side effects it could bring. If we go down this path, would it be possible to *not* resign the packages, and just leave them signed with the EPEL key?
Well, pulling/rsync EPEL signed pkgs and import in cbs is "easy" but yeah, the current signing pipeline would just (as it was designed for that particular case) sign pkgs in a tags with the SIG gpg key, and not have "exceptions" So if that's considered an issue to have epel pkgs signed again with SIG gpg key in *their* repositories, we should revisit the original RFE.
The other solution is then : use EPEL as external repo in koji so that pkgs depending on (Build)Requires: at build time would find pkgs and so build .. but that would mean : - such SIG would probably have a dep on epel-release if other EPEL pkgs are needed at runtime (probably the case if it was needed also at buildtime) - no way for SIG to stick to a particular ENVR (and if they want to, - thinking about RDO/openstack cloud sig- they'd probably rebuild epel pkgs in their tags, like they are doing for some years now ...)
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
On Fri, May 7, 2021 at 8:07 AM Fabian Arrotin arrfab@centos.org wrote:
On 05/05/2021 17:21, Davide Cavalca via CentOS-devel wrote:
On Wed, 2021-05-05 at 13:59 +0200, Fabian Arrotin wrote:
I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs.
*but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}).
Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key
I can think of one downside of this: it would result in packages with the same ENVR, but different signatures and checksums. I know this would be a problem for FB (due to how some of our internal tooling works), but I'm not sure what other side effects it could bring. If we go down this path, would it be possible to *not* resign the packages, and just leave them signed with the EPEL key?
Well, pulling/rsync EPEL signed pkgs and import in cbs is "easy" but yeah, the current signing pipeline would just (as it was designed for that particular case) sign pkgs in a tags with the SIG gpg key, and not have "exceptions" So if that's considered an issue to have epel pkgs signed again with SIG gpg key in *their* repositories, we should revisit the original RFE.
The other solution is then : use EPEL as external repo in koji so that pkgs depending on (Build)Requires: at build time would find pkgs and so build .. but that would mean :
- such SIG would probably have a dep on epel-release if other EPEL pkgs
are needed at runtime (probably the case if it was needed also at buildtime)
- no way for SIG to stick to a particular ENVR (and if they want to, -
thinking about RDO/openstack cloud sig- they'd probably rebuild epel pkgs in their tags, like they are doing for some years now ...)
From RDO/CloudSIG perspective, the workflow of getting EPEL imported in CBS
and tagging the required builds on the SIG tags would work fine if resigning and redistribution is not a problem.
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
-- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | twitter: @arrfab _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Fri, May 7, 2021 at 3:37 AM Alfredo Moralejo Alonso amoralej@redhat.com wrote:
On Fri, May 7, 2021 at 8:07 AM Fabian Arrotin arrfab@centos.org wrote:
On 05/05/2021 17:21, Davide Cavalca via CentOS-devel wrote:
On Wed, 2021-05-05 at 13:59 +0200, Fabian Arrotin wrote:
I started to rsync/pull epel7/8 pkgs for x86_64,aarch64,ppc64le on a temporary place and we can start testing importing pkgs.
*but* it's where it needs probably a little bit of clarification : while initial request was to just have access to EPEL pkgs to satisfy Requires: and/or BuildRequires: I'm wondering about a redistribution policy (if any) for pkgs built on fedora infra and that SIGs would be able to just redistribute if they tag such pkg in their own tag (mostly for -{testing,release}).
Each pkg tag for -release would go out on mirror CDN, but signed with SIG gpg key
I can think of one downside of this: it would result in packages with the same ENVR, but different signatures and checksums. I know this would be a problem for FB (due to how some of our internal tooling works), but I'm not sure what other side effects it could bring. If we go down this path, would it be possible to *not* resign the packages, and just leave them signed with the EPEL key?
Well, pulling/rsync EPEL signed pkgs and import in cbs is "easy" but yeah, the current signing pipeline would just (as it was designed for that particular case) sign pkgs in a tags with the SIG gpg key, and not have "exceptions" So if that's considered an issue to have epel pkgs signed again with SIG gpg key in *their* repositories, we should revisit the original RFE.
The other solution is then : use EPEL as external repo in koji so that pkgs depending on (Build)Requires: at build time would find pkgs and so build .. but that would mean :
- such SIG would probably have a dep on epel-release if other EPEL pkgs
are needed at runtime (probably the case if it was needed also at buildtime)
- no way for SIG to stick to a particular ENVR (and if they want to, -
thinking about RDO/openstack cloud sig- they'd probably rebuild epel pkgs in their tags, like they are doing for some years now ...)
From RDO/CloudSIG perspective, the workflow of getting EPEL imported in CBS and tagging the required builds on the SIG tags would work fine if resigning and redistribution is not a problem.
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
From the hyperscale SIG perspective, if we *must* do it this way, then
we want them blocked from publishing (that is, the content *must not* be shipped to our repositories by default as they are effectively buildroot only packages) since complete compatibility with EPEL is a requirement.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Fri, 2021-05-07 at 08:36 -0400, Neal Gompa wrote:
On Fri, May 7, 2021 at 3:37 AM Alfredo Moralejo Alonso amoralej@redhat.com wrote:
On Fri, May 7, 2021 at 8:07 AM Fabian Arrotin arrfab@centos.org wrote: From RDO/CloudSIG perspective, the workflow of getting EPEL imported in CBS and tagging the required builds on the SIG tags would work fine if resigning and redistribution is not a problem.
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
From the hyperscale SIG perspective, if we *must* do it this way, then we want them blocked from publishing (that is, the content *must not* be shipped to our repositories by default as they are effectively buildroot only packages) since complete compatibility with EPEL is a requirement.
Yeah, different SIGs will have different requirements here, which is fine. For SIGs that are already rebuilding and redistributing content from EPEL, and that don't require compatibility with the EPEL repos, resigning and redistributing is going to be ok.
For SIGs like Hyperscale that want compatibility with the EPEL repos, adding a dependency on epel-release and *not* resigning / redistributing will be the preferred solution. I think as long as publishing is left to the SIGs (e.g. by manually tagging the wanted packages), it will be ok. Hyperscale will tag packages to our buildroot tags, other SIGs will tag them to destination tags to have them redistributed.
Once this is implemented, I recommend documenting the tradeoffs of each approach so every SIG can choose the one that's most appropriate for them and their users.
Cheers davide
On Fri, May 07, 2021 at 08:07:12AM +0200, Fabian Arrotin wrote:
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
I don't *think* that would be a problem. It's too bad RPMs can't have multiple signatures.
But wouldn't cherry-picking ENVR cause problems if a system has EPEL enabled?
On Fri, May 7, 2021 at 9:15 AM Matthew Miller mattdm@mattdm.org wrote:
On Fri, May 07, 2021 at 08:07:12AM +0200, Fabian Arrotin wrote:
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
I don't *think* that would be a problem. It's too bad RPMs can't have multiple signatures.
RPM the file format supports it, the problem is that the signing infrastructure can't handle it.
But wouldn't cherry-picking ENVR cause problems if a system has EPEL enabled?
Yes, it will.
On 5/7/21 8:15 AM, Matthew Miller wrote:
On Fri, May 07, 2021 at 08:07:12AM +0200, Fabian Arrotin wrote:
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
I don't *think* that would be a problem. It's too bad RPMs can't have multiple signatures.
But wouldn't cherry-picking ENVR cause problems if a system has EPEL enabled?
I personally think the best option is just to use the EPEL repos as external repos and to require epel-release in repos where you require epel package to be installed.
That way they remain totally independent and we don't have multiple copies of the same rpms and ENVRs with different signatures / keys.
On Mon, 2021-05-10 at 11:08 -0500, Johnny Hughes wrote:
On 5/7/21 8:15 AM, Matthew Miller wrote:
On Fri, May 07, 2021 at 08:07:12AM +0200, Fabian Arrotin wrote:
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
I don't *think* that would be a problem. It's too bad RPMs can't have multiple signatures.
But wouldn't cherry-picking ENVR cause problems if a system has EPEL enabled?
I personally think the best option is just to use the EPEL repos as external repos and to require epel-release in repos where you require epel package to be installed.
That way they remain totally independent and we don't have multiple copies of the same rpms and ENVRs with different signatures / keys.
This is probably the only time I miss the RPMv3 ability to have multiple signatures on a single RPM.
Pat
On 10/05/2021 18:08, Johnny Hughes wrote:
On 5/7/21 8:15 AM, Matthew Miller wrote:
On Fri, May 07, 2021 at 08:07:12AM +0200, Fabian Arrotin wrote:
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
I don't *think* that would be a problem. It's too bad RPMs can't have multiple signatures.
But wouldn't cherry-picking ENVR cause problems if a system has EPEL enabled?
I personally think the best option is just to use the EPEL repos as external repos and to require epel-release in repos where you require epel package to be installed.
Sure, that would solve part of the problems SIGs asked initially to solve by just importing builds but other problems would then remain : - no way for them to tag a particular ENVR (that they can test and control in their tags) - still a need to rebuild EPEL pkgs (like for infra tags, etc)
FWIW, Aoife said (in SIG-infra meeting today, https://centos.org/minutes/2021/May/centos-meeting.2021-05-10-14.03.html) that she'll reach out to Fesco to see if they agree on the "let's import and redistribute - part/tagged pkgs - Epel pkgs through cbs.centos.org" for SIGs
OTOH, if all that is complicated (to find an agreement/policy), we can just continue like before : letting SIGs rebuild epel pkgs in cbs.centos.org and move on (and close RFE tickets about this on the infra tracker) :)
On 5/10/21 11:35 AM, Fabian Arrotin wrote:
On 10/05/2021 18:08, Johnny Hughes wrote:
On 5/7/21 8:15 AM, Matthew Miller wrote:
On Fri, May 07, 2021 at 08:07:12AM +0200, Fabian Arrotin wrote:
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
I don't *think* that would be a problem. It's too bad RPMs can't have multiple signatures.
But wouldn't cherry-picking ENVR cause problems if a system has EPEL enabled?
I personally think the best option is just to use the EPEL repos as external repos and to require epel-release in repos where you require epel package to be installed.
Sure, that would solve part of the problems SIGs asked initially to solve by just importing builds but other problems would then remain :
- no way for them to tag a particular ENVR (that they can test and
control in their tags)
- still a need to rebuild EPEL pkgs (like for infra tags, etc)
EPEL does not maintain multiple ENVRs, right? So if they need that, they will have to create it themselves. Either we use EPEL or we don't. You guys decide.
BUt IF we ARE going to use EPEL, the we need to use it. Not rebuild it and resign it. If that causes too mant problems, then keep doing what we are doing. There are pros and cons to switching.
FWIW, Aoife said (in SIG-infra meeting today, https://centos.org/minutes/2021/May/centos-meeting.2021-05-10-14.03.html) that she'll reach out to Fesco to see if they agree on the "let's import and redistribute - part/tagged pkgs - Epel pkgs through cbs.centos.org" for SIGs
OTOH, if all that is complicated (to find an agreement/policy), we can just continue like before : letting SIGs rebuild epel pkgs in cbs.centos.org and move on (and close RFE tickets about this on the infra tracker) :)
Agreed.
On Mon, May 10, 2021 at 11:52 PM Johnny Hughes johnny@centos.org wrote:
On 5/10/21 11:35 AM, Fabian Arrotin wrote:
On 10/05/2021 18:08, Johnny Hughes wrote:
On 5/7/21 8:15 AM, Matthew Miller wrote:
On Fri, May 07, 2021 at 08:07:12AM +0200, Fabian Arrotin wrote:
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
I don't *think* that would be a problem. It's too bad RPMs can't have multiple signatures.
But wouldn't cherry-picking ENVR cause problems if a system has EPEL enabled?
I personally think the best option is just to use the EPEL repos as external repos and to require epel-release in repos where you require epel package to be installed.
Sure, that would solve part of the problems SIGs asked initially to solve by just importing builds but other problems would then remain :
- no way for them to tag a particular ENVR (that they can test and
control in their tags)
- still a need to rebuild EPEL pkgs (like for infra tags, etc)
EPEL does not maintain multiple ENVRs, right? So if they need that, they will have to create it themselves. Either we use EPEL or we don't. You guys decide.
BUt IF we ARE going to use EPEL, the we need to use it. Not rebuild it and resign it. If that causes too mant problems, then keep doing what we are doing. There are pros and cons to switching.
I want us to use EPEL as-is. SIGs consuming EPEL content should not have the expectation of frozen-ness, they should be working with EPEL maintainers to make sure their stuff works. The problem with this whole discussion is that SIGs basically don't want to work with EPEL maintainers, which is a totally crap way to do things. In Hyperscale, our policy is that stuff goes into our repo *only if it doesn't make sense for EPEL* because we understand that basically *nobody* runs CentOS without EPEL.
To be blunt, any SIG who thinks that their content isn't already used alongside EPEL is basically fooling themselves. They should consume EPEL as it is and deal with it.
FWIW, Aoife said (in SIG-infra meeting today, https://centos.org/minutes/2021/May/centos-meeting.2021-05-10-14.03.html)
that
she'll reach out to Fesco to see if they agree on the "let's import and redistribute - part/tagged pkgs - Epel pkgs through cbs.centos.org" for
SIGs
Just as an fyi, I have opened this ticket https://pagure.io/fesco/issue/2607 for with Fesco for discussion and to make sure we are doing our (CentOS Infra SIG) due diligence with this request to properly facilitate it. Thanks! :)
OTOH, if all that is complicated (to find an agreement/policy), we can
just continue like before : letting SIGs rebuild epel pkgs in cbs.centos.org and move on (and close RFE tickets about this on the infra tracker) :)
Agreed.
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Mon, May 10, 2021 at 12:09 PM Johnny Hughes johnny@centos.org wrote:
I personally think the best option is just to use the EPEL repos as external repos and to require epel-release in repos where you require epel package to be installed.
+1
That way they remain totally independent and we don't have multiple copies of the same rpms and ENVRs with different signatures / keys.
Agree.
--
Kaleb
On Mon, May 10, 2021 at 12:09 PM Johnny Hughes johnny@centos.org wrote:
On 5/7/21 8:15 AM, Matthew Miller wrote:
On Fri, May 07, 2021 at 08:07:12AM +0200, Fabian Arrotin wrote:
So we have two solutions and the easiest/fastest one is probably just to import pkgs in koji and SIG can just tag-build what they want/need (including cherry-picking ENVR) but with the downside effect of pkg signed with a different gpg key (and so my original question to Fedora : is that allowed ?)
I don't *think* that would be a problem. It's too bad RPMs can't have multiple signatures.
But wouldn't cherry-picking ENVR cause problems if a system has EPEL enabled?
I personally think the best option is just to use the EPEL repos as external repos and to require epel-release in repos where you require epel package to be installed.
That way they remain totally independent and we don't have multiple copies of the same rpms and ENVRs with different signatures / keys.
"But Bullwinkle! That trick never works!"
More seriously, the difficulty is that EPEL discards the older packages from the public repos, especially previous releases of the same package or packages that are then brought into CentOS. This happened with ansible and many python modules, and it became quite awkward to find and revert to those previous RPMs for specific package releases. And if you've never encountered the nightmare that can be python dependency resolution with the less integrated or out of date modules over at pypi.org, expected to work with one published yesterday, then good luck to you, sir.
It's therefore common place to set up an internal repository that only aggregates from EPEL, and never deletes content, until after a major Red Hat based OS is entirely discarded. It's why I publish reposync wrapper tools over at github.
On Mon, May 10, 2021 at 10:09 AM Johnny Hughes johnny@centos.org wrote:
I personally think the best option is just to use the EPEL repos as external repos and to require epel-release in repos where you require epel package to be installed.
That way they remain totally independent and we don't have multiple copies of the same rpms and ENVRs with different signatures / keys.
I agree that this is the simplest solution. It's the easiest for everyone to understand and explain.
- Ken
Le 11/05/2021 à 18:03, Ken Dreyer a écrit :
On Mon, May 10, 2021 at 10:09 AM Johnny Hughes johnny@centos.org wrote:
I personally think the best option is just to use the EPEL repos as external repos and to require epel-release in repos where you require epel package to be installed.
That way they remain totally independent and we don't have multiple copies of the same rpms and ENVRs with different signatures / keys.
I agree that this is the simplest solution. It's the easiest for everyone to understand and explain.
- Ken
From my external point of view, with CentOS revamp as an upstream of RHEL, EPEL should now be consider as another and primary SIG for RHEL Ecosystem.
Jean-Marc
On Wed, May 12, 2021 at 06:24:21PM +0200, Jean-Marc Liger wrote:
From my external point of view, with CentOS revamp as an upstream of RHEL, EPEL should now be consider as another and primary SIG for RHEL Ecosystem.
I definitely see at as a bridge, but there are some differences. Notably, EPEL shares the same dist-git as Fedora Linux packages, which makes it easy to cherry-pick and maintain both. I'd not like that to go backwards.
On 4/29/21 6:09 AM, Fabian Arrotin wrote:
On 29/04/2021 00:07, Davide Cavalca via CentOS-devel wrote:
On Wed, 2021-04-28 at 23:15 +0200, Fabian Arrotin wrote:
Just some links to previous threads about this :
(cut)
So worth reading the three threads and then revisit it maybe ? :)
Thanks Fabian, this is really helpful. I actually think the proposal you had made back then in https://lists.centos.org/pipermail/centos-devel/2019-July/073860.html would work well for this, and addresses most of the concerns I've seen raised throughout those threads.
Great, I'd like to have other SIGs opinions too but that's a good start ! :-)
Specifically:
- it does not require rebuilding EPEL onto CBS (which would defeat the
entire purpose of this exercise)
Yes ! :-)
- it gives SIGs autonomy of decision on whether to consume content from
EPEL, rebuild it or ignore it, on a per-package basis
That was indeed the idea : each SIG can tag-build for -testing/-release the version they want
- because EPEL is continuously imported and versions that are consumed
are tagged, this also solves the "yum downgrade" problem -- even if EPEL upstream moves on, older versions that were consumed in CBS would remain available (unless one untags them explicitly)
Yes, idea (as explained below) would be to create internal tag just for this and people can tag-build from that tag in their SIG tag[s]
- OTOH, if a SIG does want to track the latest versions available in
EPEL, that is also possible (e.g. Hyperscale would like this as it would expose potential breakage early on)
One potential issue I don't have a clear sense of is the matter of conflicting NVRs that was raised in https://lists.centos.org/pipermail/centos-devel/2019-July/073863.html . This is not a concern for Hyperscale as we set a custom disttag, but it could be for other SIGs that don't do this and also rebuild packages present in EPEL with the same NVR.
Reason why we'd have a consensus on doing this and properly communicated. If we decide to just import existing builds into cbs (can be an internal tag that will never be pushed out for obvious reasons), that will work for pkgs with different ENVR, but not for existing ones. So if SIGs applied some patches before rebuilding an epel pkg, they'd be more than encouraged to do this at the EPEL level. For the others, they can just carry on with existing pkgs and next time, just 'tag-build' the import EPEL packages to satisfy Requires: or BuildRequires:
I would LOVE to make the EPEL repos available to build against in the CentOS buildroots. I think it is a solid idea.
The problem obviously is, that is not how RHEL is done now. It could be how it is done for Stream and RHEL in the future. This is a great place to discuss the benefits and problems with this idea.
Certainly for SIGs, I see no reason not to do it for CBS and/or any other SIG building location.
On Fri, Apr 30, 2021 at 09:21:41AM -0500, Johnny Hughes wrote:
I would LOVE to make the EPEL repos available to build against in the CentOS buildroots. I think it is a solid idea.
The problem obviously is, that is not how RHEL is done now. It could be how it is done for Stream and RHEL in the future. This is a great place to discuss the benefits and problems with this idea.
I think the thing to be care of there is not allowing RHEL binaries to leak out. This is why EPEL uses external repos (so they can be locked down).
Certainly for SIGs, I see no reason not to do it for CBS and/or any other SIG building location.
+1
kevin
On Fri, Apr 30, 2021 at 10:21 AM Johnny Hughes johnny@centos.org wrote:
I would LOVE to make the EPEL repos available to build against in the CentOS buildroots. I think it is a solid idea.
...
Certainly for SIGs, I see no reason not to do it for CBS and/or any other SIG building location.
Yes, that's a better fit for how many people want to consume Storage SIG packages.
E.g. Ceph's upstream CI builds using EPEL packages. And a lot of other consumers of Storage SIG packages are using EPEL, and we sometimes run into conflicts when they have EPEL enabled.
For C8 Stream I would have preferred to not have to build the two dozen or so dependencies before I could build Ceph itself.
So +1 from me on being able to use EPEL in CBS for SIG packages.
--
Kaleb
-
Alfredo Moralejo Alonso -
Aoife Moloney -
Davide Cavalca -
Fabian Arrotin -
Jean-Marc Liger -
Johnny Hughes -
Kaleb Keithley -
Ken Dreyer -
Kevin Fenzi -
Matthew Miller -
Neal Gompa -
Nico Kadel-Garcia -
Patrick Riehecky