Good Morning Everyone,
There are currently two lookaside caches in use around the CentOS project: * One used by CentOS-Stream: https://sources.stream.centos.org/sources it's not browsable, but it uses the structure: `baseurl/pkgname/tarball/hashtype/hash/tarball`. Example: https://sources.stream.centos.org/sources/rpms/kernel/linux-5.14.0-62.el9.ta... * One used by CentOS-Linux, CentOS-Stream 8 and the SIGs: https://git.centos.org/sources/ this one is browsable and as you can see uses the structure: `baseurl/pkgname/branch/hash`. Example: https://git.centos.org/sources/kernel/c8s/0c4e10577cfd4b4f8e3d83c0406da8ab05...
The rest of this email focuses on this second one. SIGs upload to it using the route: https://git.centos.org/sources/upload.cgi
In an email last week [1] was proposed an idea for how SIGs could leverage the centos namespace in gitlab for those who wishes.
One of the benefits of using gitlab would be increased flexibility for SIGs and a clear example for this would be the ability to drop the branch structures currently imposed on the git repositories. That structure is imposed because the git repositories are shared between CentOS-Linux, CentOS-Stream and (potentially multiple) SIGs, so that structure ensures groups are not stepping on each other's toes. By moving the SIGs out of these shared repositories, imposing that structure is no longer needed.
However, since the lookaside cache relies on branch name, lifting that structure would break the lookaside cache.
I have already brought this idea to a few folks to see if the idea was sane. The consensus that emerged is: * Introduce a new upload endpoint next to the existing one, something like: https://git.centos.org/sources/sig_upload.cgi * That new endpoint would upload the sources given using the same structure as the one used for CentOS-Stream, but ensuring that the person uploading is member of at least one SIG.
The idea of using `sig_upload.cgi` instead of just replacing `upload.cgi` is the assumption that we want to preserve the current structure used for CentOS-Linux and CentOS-Stream, allowing to find more easily which sources are used where and not impacting the process Red Hat uses to push its releases.
Since the structures used by the two upload scripts are different, they will not conflict. What we will end up seeing is something like:
sources │ ├── pkg1 │ ├── c7 │ │ ├── hash1 │ │ └── hash2 │ ├── c8 │ │ ├── hash3 │ │ └── hash4 │ ├── tarball1 │ │ └── sha name │ │ └── hash5 │ │ └── tarball1 │ └── tarball2 │ └── sha name │ └── hash6 │ └── tarball2 │ └── pkg2 ├── c8 │ ├── hash7 │ └── hash8 ├── c8s │ ├── hash9 │ └── hash10 ├── tarball3 │ └── sha name │ └── hash11 │ └── tarball3 └── tarball4 └── sha name └── hash12 └── tarball4 and so on
On CBS, the script that downloads the sources, will then need to be adjusted to try first the old structure before trying the new one. This may slow down a little bit the builds, but that should be most of the time, at most by a single http request.
In this email I'm calling for feedback, do you like the idea?
I'm happy to work on making it happen if there is consensus on this :)
Looking forward for your thoughts, Pierre
[1] https://lists.centos.org/pipermail/centos-devel/2022-February/120216.html
On Mon, Feb 21, 2022 at 10:51 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
Good Morning Everyone,
There are currently two lookaside caches in use around the CentOS project:
- One used by CentOS-Stream: https://sources.stream.centos.org/sources it's not browsable, but it uses the structure: `baseurl/pkgname/tarball/hashtype/hash/tarball`. Example: https://sources.stream.centos.org/sources/rpms/kernel/linux-5.14.0-62.el9.ta...
- One used by CentOS-Linux, CentOS-Stream 8 and the SIGs: https://git.centos.org/sources/ this one is browsable and as you can see uses the structure: `baseurl/pkgname/branch/hash`. Example: https://git.centos.org/sources/kernel/c8s/0c4e10577cfd4b4f8e3d83c0406da8ab05...
The rest of this email focuses on this second one. SIGs upload to it using the route: https://git.centos.org/sources/upload.cgi
In an email last week [1] was proposed an idea for how SIGs could leverage the centos namespace in gitlab for those who wishes.
One of the benefits of using gitlab would be increased flexibility for SIGs and a clear example for this would be the ability to drop the branch structures currently imposed on the git repositories. That structure is imposed because the git repositories are shared between CentOS-Linux, CentOS-Stream and (potentially multiple) SIGs, so that structure ensures groups are not stepping on each other's toes. By moving the SIGs out of these shared repositories, imposing that structure is no longer needed.
However, since the lookaside cache relies on branch name, lifting that structure would break the lookaside cache.
I have already brought this idea to a few folks to see if the idea was sane. The consensus that emerged is:
- Introduce a new upload endpoint next to the existing one, something like: https://git.centos.org/sources/sig_upload.cgi
- That new endpoint would upload the sources given using the same structure as the one used for CentOS-Stream, but ensuring that the person uploading is member of at least one SIG.
The idea of using `sig_upload.cgi` instead of just replacing `upload.cgi` is the assumption that we want to preserve the current structure used for CentOS-Linux and CentOS-Stream, allowing to find more easily which sources are used where and not impacting the process Red Hat uses to push its releases.
Since the structures used by the two upload scripts are different, they will not conflict. What we will end up seeing is something like:
sources │ ├── pkg1 │ ├── c7 │ │ ├── hash1 │ │ └── hash2 │ ├── c8 │ │ ├── hash3 │ │ └── hash4 │ ├── tarball1 │ │ └── sha name │ │ └── hash5 │ │ └── tarball1 │ └── tarball2 │ └── sha name │ └── hash6 │ └── tarball2 │ └── pkg2 ├── c8 │ ├── hash7 │ └── hash8 ├── c8s │ ├── hash9 │ └── hash10 ├── tarball3 │ └── sha name │ └── hash11 │ └── tarball3 └── tarball4 └── sha name └── hash12 └── tarball4 and so on
On CBS, the script that downloads the sources, will then need to be adjusted to try first the old structure before trying the new one. This may slow down a little bit the builds, but that should be most of the time, at most by a single http request.
In this email I'm calling for feedback, do you like the idea?
I'm happy to work on making it happen if there is consensus on this :)
Looking forward for your thoughts, Pierre
The git server and git structure is orthogonal to the lookaside problem. Fundamentally, the issue was that the same upload endpoint is used for both Red Hat compliance and SIG work. We already have authentication/authorization on branches at the Pagure level, so we just lacked a way to handle this for the lookaside upload. By splitting the endpoint, it should be possible to solve that since you can deny access to the Red Hat endpoint to everyone.
However, I'd make a small suggestion: instead of changing the endpoint URL for SIGs, change the endpoint URL for Red Hat. RCM uses that endpoint through automation (I assume), so changing the endpoint for the one service is considerably simpler than dealing with everyone's own scripts to adjust for SIGs.
As an example, I've written automation to deal with Hyperscale work because doing it by hand is a lot of grunt work. While I can probably tweak my stuff easily enough, I don't know if *everyone* can.
And again, the lookaside thing is completely orthogonal to the git structure. I should be able to use it just fine from git.centos.org in the current branched package structure.
On Mon, Feb 21, 2022 at 10:58:58AM -0500, Neal Gompa wrote:
On Mon, Feb 21, 2022 at 10:51 AM Pierre-Yves Chibon pingou@pingoured.fr wrote: The git server and git structure is orthogonal to the lookaside problem. Fundamentally, the issue was that the same upload endpoint is used for both Red Hat compliance and SIG work. We already have authentication/authorization on branches at the Pagure level, so we just lacked a way to handle this for the lookaside upload. By splitting the endpoint, it should be possible to solve that since you can deny access to the Red Hat endpoint to everyone.
However, I'd make a small suggestion: instead of changing the endpoint URL for SIGs, change the endpoint URL for Red Hat. RCM uses that endpoint through automation (I assume), so changing the endpoint for the one service is considerably simpler than dealing with everyone's own scripts to adjust for SIGs.
That is an interesting point, I'll ask around to know how feasible it would be.
As an example, I've written automation to deal with Hyperscale work because doing it by hand is a lot of grunt work. While I can probably tweak my stuff easily enough, I don't know if *everyone* can.
Your automation doesn't use the `lookaside_upload` script from centos-git-commoin then?
And again, the lookaside thing is completely orthogonal to the git structure. I should be able to use it just fine from git.centos.org in the current branched package structure.
I agree, though dropping the branch structure that git.centos.org imposes exacerbates this. I don't know that we want to change the current structure used for all SIGs (vs making it opt-in), but it is an interesting thought.
Thanks for your thoughts, Pierre
On Mon, Feb 21, 2022 at 11:16 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Mon, Feb 21, 2022 at 10:58:58AM -0500, Neal Gompa wrote:
On Mon, Feb 21, 2022 at 10:51 AM Pierre-Yves Chibon pingou@pingoured.fr wrote: The git server and git structure is orthogonal to the lookaside problem. Fundamentally, the issue was that the same upload endpoint is used for both Red Hat compliance and SIG work. We already have authentication/authorization on branches at the Pagure level, so we just lacked a way to handle this for the lookaside upload. By splitting the endpoint, it should be possible to solve that since you can deny access to the Red Hat endpoint to everyone.
However, I'd make a small suggestion: instead of changing the endpoint URL for SIGs, change the endpoint URL for Red Hat. RCM uses that endpoint through automation (I assume), so changing the endpoint for the one service is considerably simpler than dealing with everyone's own scripts to adjust for SIGs.
That is an interesting point, I'll ask around to know how feasible it would be.
As an example, I've written automation to deal with Hyperscale work because doing it by hand is a lot of grunt work. While I can probably tweak my stuff easily enough, I don't know if *everyone* can.
Your automation doesn't use the `lookaside_upload` script from centos-git-commoin then?
I have a vendored copy of stuff from centos-git-common. But other people may have taken different approaches.
And again, the lookaside thing is completely orthogonal to the git structure. I should be able to use it just fine from git.centos.org in the current branched package structure.
I agree, though dropping the branch structure that git.centos.org imposes exacerbates this. I don't know that we want to change the current structure used for all SIGs (vs making it opt-in), but it is an interesting thought.
There are some advantages to the existing per-branch structure. I've used to easily sync and merge Hyperscale modifications onto c8s packages for Hyperscale 8 package builds. That benefit will be gone with c9s, but it was handy with c8s.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Mon, Feb 21, 2022, at 10:16, Pierre-Yves Chibon wrote:
On Mon, Feb 21, 2022 at 10:58:58AM -0500, Neal Gompa wrote:
On Mon, Feb 21, 2022 at 10:51 AM Pierre-Yves Chibon pingou@pingoured.fr wrote: The git server and git structure is orthogonal to the lookaside problem. Fundamentally, the issue was that the same upload endpoint is used for both Red Hat compliance and SIG work. We already have authentication/authorization on branches at the Pagure level, so we just lacked a way to handle this for the lookaside upload. By splitting the endpoint, it should be possible to solve that since you can deny access to the Red Hat endpoint to everyone.
However, I'd make a small suggestion: instead of changing the endpoint URL for SIGs, change the endpoint URL for Red Hat. RCM uses that endpoint through automation (I assume), so changing the endpoint for the one service is considerably simpler than dealing with everyone's own scripts to adjust for SIGs.
That is an interesting point, I'll ask around to know how feasible it would be.
Changing the endpoint for Red Hat's downstream pushes is not an option for us at this time.
As an example, I've written automation to deal with Hyperscale work because doing it by hand is a lot of grunt work. While I can probably tweak my stuff easily enough, I don't know if *everyone* can.
Your automation doesn't use the `lookaside_upload` script from centos-git-commoin then?
And again, the lookaside thing is completely orthogonal to the git structure. I should be able to use it just fine from git.centos.org in the current branched package structure.
I agree, though dropping the branch structure that git.centos.org imposes exacerbates this. I don't know that we want to change the current structure used for all SIGs (vs making it opt-in), but it is an interesting thought.
Thanks for your thoughts, Pierre _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Mon, Feb 21, 2022 at 10:58:58AM -0500, Neal Gompa wrote:
On Mon, Feb 21, 2022 at 10:51 AM Pierre-Yves Chibon pingou@pingoured.fr wrote: The git server and git structure is orthogonal to the lookaside problem. Fundamentally, the issue was that the same upload endpoint is used for both Red Hat compliance and SIG work. We already have authentication/authorization on branches at the Pagure level, so we just lacked a way to handle this for the lookaside upload. By splitting the endpoint, it should be possible to solve that since you can deny access to the Red Hat endpoint to everyone.
However, I'd make a small suggestion: instead of changing the endpoint URL for SIGs, change the endpoint URL for Red Hat. RCM uses that endpoint through automation (I assume), so changing the endpoint for the one service is considerably simpler than dealing with everyone's own scripts to adjust for SIGs.
I have just realized something, I don't think we're planning on restricting the access to the old lookaside upload script (the idea you're talking about in the first paragraph), so if we are not, we are not breaking backward compatibility, just introducing/offering another way to do things: - either you keep your current workflow and upload as you do today with the current lookaside upload endpoint - or you adjust your workflow and upload to the lookaside cache using the new endpoint.
Both approach would work equally well.
And this is where it does ties the git server and git structure to the lookaside, if a SIG keeps using git.centos.org, they will have to keep using the current branch structure and thus changing their workflow to use the new lookaside cache endpoint does not bring them anything, they can just keep doing their work the way they have always been. If a SIG chooses at some point to start using gitlab, they will have to adjust their workflow (if only to change the hostname of the git server) and they will have the choice: - either keep using the branch structure that git.centos.org imposes (but that time it won't be imposed, it'll be by choice) and then they pick which lookaside upload endpoint they use - or they choose a different git branch structure (so another change in their workflow) and there, they won't have the choice and will have to use the new lookaside upload endpoint.
So it seems simpler to me to keep with the original idea of having two lookaside upload endpoint, the current one keeping the current structure and a new one using the structure used by CentOS-Stream and Fedora.
Pierre
On Mon, Feb 28, 2022 at 9:48 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Mon, Feb 21, 2022 at 10:58:58AM -0500, Neal Gompa wrote:
On Mon, Feb 21, 2022 at 10:51 AM Pierre-Yves Chibon pingou@pingoured.fr wrote: The git server and git structure is orthogonal to the lookaside problem. Fundamentally, the issue was that the same upload endpoint is used for both Red Hat compliance and SIG work. We already have authentication/authorization on branches at the Pagure level, so we just lacked a way to handle this for the lookaside upload. By splitting the endpoint, it should be possible to solve that since you can deny access to the Red Hat endpoint to everyone.
However, I'd make a small suggestion: instead of changing the endpoint URL for SIGs, change the endpoint URL for Red Hat. RCM uses that endpoint through automation (I assume), so changing the endpoint for the one service is considerably simpler than dealing with everyone's own scripts to adjust for SIGs.
I have just realized something, I don't think we're planning on restricting the access to the old lookaside upload script (the idea you're talking about in the first paragraph), so if we are not, we are not breaking backward compatibility, just introducing/offering another way to do things:
- either you keep your current workflow and upload as you do today with the current lookaside upload endpoint
- or you adjust your workflow and upload to the lookaside cache using the new endpoint.
Both approach would work equally well.
And this is where it does ties the git server and git structure to the lookaside, if a SIG keeps using git.centos.org, they will have to keep using the current branch structure and thus changing their workflow to use the new lookaside cache endpoint does not bring them anything, they can just keep doing their work the way they have always been. If a SIG chooses at some point to start using gitlab, they will have to adjust their workflow (if only to change the hostname of the git server) and they will have the choice:
- either keep using the branch structure that git.centos.org imposes (but that time it won't be imposed, it'll be by choice) and then they pick which lookaside upload endpoint they use
- or they choose a different git branch structure (so another change in their workflow) and there, they won't have the choice and will have to use the new lookaside upload endpoint.
So it seems simpler to me to keep with the original idea of having two lookaside upload endpoint, the current one keeping the current structure and a new one using the structure used by CentOS-Stream and Fedora.
So what you're saying is that I could just use the new endpoint? If that's the case, then I can *still* use this with git.centos.org if I want. I would just get the flexibility to use the flat layout and I can cherry-pick commits from Fedora or CentOS Stream as needed.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Mon, Feb 28, 2022 at 09:51:28AM -0500, Neal Gompa wrote:
On Mon, Feb 28, 2022 at 9:48 AM Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Mon, Feb 21, 2022 at 10:58:58AM -0500, Neal Gompa wrote:
On Mon, Feb 21, 2022 at 10:51 AM Pierre-Yves Chibon pingou@pingoured.fr wrote: The git server and git structure is orthogonal to the lookaside problem. Fundamentally, the issue was that the same upload endpoint is used for both Red Hat compliance and SIG work. We already have authentication/authorization on branches at the Pagure level, so we just lacked a way to handle this for the lookaside upload. By splitting the endpoint, it should be possible to solve that since you can deny access to the Red Hat endpoint to everyone.
However, I'd make a small suggestion: instead of changing the endpoint URL for SIGs, change the endpoint URL for Red Hat. RCM uses that endpoint through automation (I assume), so changing the endpoint for the one service is considerably simpler than dealing with everyone's own scripts to adjust for SIGs.
I have just realized something, I don't think we're planning on restricting the access to the old lookaside upload script (the idea you're talking about in the first paragraph), so if we are not, we are not breaking backward compatibility, just introducing/offering another way to do things:
- either you keep your current workflow and upload as you do today with the current lookaside upload endpoint
- or you adjust your workflow and upload to the lookaside cache using the new endpoint.
Both approach would work equally well.
And this is where it does ties the git server and git structure to the lookaside, if a SIG keeps using git.centos.org, they will have to keep using the current branch structure and thus changing their workflow to use the new lookaside cache endpoint does not bring them anything, they can just keep doing their work the way they have always been. If a SIG chooses at some point to start using gitlab, they will have to adjust their workflow (if only to change the hostname of the git server) and they will have the choice:
- either keep using the branch structure that git.centos.org imposes (but that time it won't be imposed, it'll be by choice) and then they pick which lookaside upload endpoint they use
- or they choose a different git branch structure (so another change in their workflow) and there, they won't have the choice and will have to use the new lookaside upload endpoint.
So it seems simpler to me to keep with the original idea of having two lookaside upload endpoint, the current one keeping the current structure and a new one using the structure used by CentOS-Stream and Fedora.
So what you're saying is that I could just use the new endpoint? If that's the case, then I can *still* use this with git.centos.org if I want. I would just get the flexibility to use the flat layout and I can cherry-pick commits from Fedora or CentOS Stream as needed.
That is the idea yes. You'll have to adjust your script to point to the new endpoint, but yes you'll be able to use it regardless of where the git repo is hosted.
Pierre
On 21/02/2022 16.51, Pierre-Yves Chibon wrote:
Good Morning Everyone,
There are currently two lookaside caches in use around the CentOS project:
- One used by CentOS-Stream: https://sources.stream.centos.org/sources it's not browsable, but it uses the structure: `baseurl/pkgname/tarball/hashtype/hash/tarball`. Example: https://sources.stream.centos.org/sources/rpms/kernel/linux-5.14.0-62.el9.ta...
- One used by CentOS-Linux, CentOS-Stream 8 and the SIGs: https://git.centos.org/sources/ this one is browsable and as you can see uses the structure: `baseurl/pkgname/branch/hash`. Example: https://git.centos.org/sources/kernel/c8s/0c4e10577cfd4b4f8e3d83c0406da8ab05...
The rest of this email focuses on this second one. SIGs upload to it using the route: https://git.centos.org/sources/upload.cgi
In an email last week [1] was proposed an idea for how SIGs could leverage the centos namespace in gitlab for those who wishes.
One of the benefits of using gitlab would be increased flexibility for SIGs and a clear example for this would be the ability to drop the branch structures currently imposed on the git repositories. That structure is imposed because the git repositories are shared between CentOS-Linux, CentOS-Stream and (potentially multiple) SIGs, so that structure ensures groups are not stepping on each other's toes. By moving the SIGs out of these shared repositories, imposing that structure is no longer needed.
However, since the lookaside cache relies on branch name, lifting that structure would break the lookaside cache.
I have already brought this idea to a few folks to see if the idea was sane. The consensus that emerged is:
- Introduce a new upload endpoint next to the existing one, something like: https://git.centos.org/sources/sig_upload.cgi
- That new endpoint would upload the sources given using the same structure as the one used for CentOS-Stream, but ensuring that the person uploading is member of at least one SIG.
The idea of using `sig_upload.cgi` instead of just replacing `upload.cgi` is the assumption that we want to preserve the current structure used for CentOS-Linux and CentOS-Stream, allowing to find more easily which sources are used where and not impacting the process Red Hat uses to push its releases.
Since the structures used by the two upload scripts are different, they will not conflict. What we will end up seeing is something like:
sources │ ├── pkg1 │ ├── c7 │ │ ├── hash1 │ │ └── hash2 │ ├── c8 │ │ ├── hash3 │ │ └── hash4 │ ├── tarball1 │ │ └── sha name │ │ └── hash5 │ │ └── tarball1 │ └── tarball2 │ └── sha name │ └── hash6 │ └── tarball2 │ └── pkg2 ├── c8 │ ├── hash7 │ └── hash8 ├── c8s │ ├── hash9 │ └── hash10 ├── tarball3 │ └── sha name │ └── hash11 │ └── tarball3 └── tarball4 └── sha name └── hash12 └── tarball4 and so on
As this already requires some changes, does it make sense to have SIG sources in a different directory than RHEL sources? The proposed structure does not allow sharing sources between RHEL and SIGs so what is the benefit of having both in the same directory? It might even lead to confusion due to the two different structures used.
I.e. put SIG sources in git.centos.org/SIGs/sources or whatever is possible/preferred and have a upload.cgi there. I personally prefer a clean separation between content provided by RH and SIGs.
On CBS, the script that downloads the sources, will then need to be adjusted to try first the old structure before trying the new one. This may slow down a little bit the builds, but that should be most of the time, at most by a single http request.
In this email I'm calling for feedback, do you like the idea?
I'm happy to work on making it happen if there is consensus on this :)
Looking forward for your thoughts, Pierre
[1] https://lists.centos.org/pipermail/centos-devel/2022-February/120216.html _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Tue, Feb 22, 2022 at 9:17 AM Peter Georg peter.georg@physik.uni-regensburg.de wrote:
On 21/02/2022 16.51, Pierre-Yves Chibon wrote:
Good Morning Everyone,
There are currently two lookaside caches in use around the CentOS project:
- One used by CentOS-Stream: https://sources.stream.centos.org/sources it's not browsable, but it uses the structure: `baseurl/pkgname/tarball/hashtype/hash/tarball`. Example: https://sources.stream.centos.org/sources/rpms/kernel/linux-5.14.0-62.el9.ta...
- One used by CentOS-Linux, CentOS-Stream 8 and the SIGs: https://git.centos.org/sources/ this one is browsable and as you can see uses the structure: `baseurl/pkgname/branch/hash`. Example: https://git.centos.org/sources/kernel/c8s/0c4e10577cfd4b4f8e3d83c0406da8ab05...
The rest of this email focuses on this second one. SIGs upload to it using the route: https://git.centos.org/sources/upload.cgi
In an email last week [1] was proposed an idea for how SIGs could leverage the centos namespace in gitlab for those who wishes.
One of the benefits of using gitlab would be increased flexibility for SIGs and a clear example for this would be the ability to drop the branch structures currently imposed on the git repositories. That structure is imposed because the git repositories are shared between CentOS-Linux, CentOS-Stream and (potentially multiple) SIGs, so that structure ensures groups are not stepping on each other's toes. By moving the SIGs out of these shared repositories, imposing that structure is no longer needed.
However, since the lookaside cache relies on branch name, lifting that structure would break the lookaside cache.
I have already brought this idea to a few folks to see if the idea was sane. The consensus that emerged is:
- Introduce a new upload endpoint next to the existing one, something like: https://git.centos.org/sources/sig_upload.cgi
- That new endpoint would upload the sources given using the same structure as the one used for CentOS-Stream, but ensuring that the person uploading is member of at least one SIG.
The idea of using `sig_upload.cgi` instead of just replacing `upload.cgi` is the assumption that we want to preserve the current structure used for CentOS-Linux and CentOS-Stream, allowing to find more easily which sources are used where and not impacting the process Red Hat uses to push its releases.
Since the structures used by the two upload scripts are different, they will not conflict. What we will end up seeing is something like:
sources │ ├── pkg1 │ ├── c7 │ │ ├── hash1 │ │ └── hash2 │ ├── c8 │ │ ├── hash3 │ │ └── hash4 │ ├── tarball1 │ │ └── sha name │ │ └── hash5 │ │ └── tarball1 │ └── tarball2 │ └── sha name │ └── hash6 │ └── tarball2 │ └── pkg2 ├── c8 │ ├── hash7 │ └── hash8 ├── c8s │ ├── hash9 │ └── hash10 ├── tarball3 │ └── sha name │ └── hash11 │ └── tarball3 └── tarball4 └── sha name └── hash12 └── tarball4 and so on
As this already requires some changes, does it make sense to have SIG sources in a different directory than RHEL sources? The proposed structure does not allow sharing sources between RHEL and SIGs so what is the benefit of having both in the same directory? It might even lead to confusion due to the two different structures used.
I.e. put SIG sources in git.centos.org/SIGs/sources or whatever is possible/preferred and have a upload.cgi there. I personally prefer a clean separation between content provided by RH and SIGs.
Going forward, only SIGs will use git.centos.org by default. CentOS Stream 9 isn't done there, so you get a much stronger separation now.
On 22/02/2022 15.18, Neal Gompa wrote:
On Tue, Feb 22, 2022 at 9:17 AM Peter Georg peter.georg@physik.uni-regensburg.de wrote:
On 21/02/2022 16.51, Pierre-Yves Chibon wrote:
Good Morning Everyone,
There are currently two lookaside caches in use around the CentOS project:
- One used by CentOS-Stream: https://sources.stream.centos.org/sources it's not browsable, but it uses the structure: `baseurl/pkgname/tarball/hashtype/hash/tarball`. Example: https://sources.stream.centos.org/sources/rpms/kernel/linux-5.14.0-62.el9.ta...
- One used by CentOS-Linux, CentOS-Stream 8 and the SIGs: https://git.centos.org/sources/ this one is browsable and as you can see uses the structure: `baseurl/pkgname/branch/hash`. Example: https://git.centos.org/sources/kernel/c8s/0c4e10577cfd4b4f8e3d83c0406da8ab05...
The rest of this email focuses on this second one. SIGs upload to it using the route: https://git.centos.org/sources/upload.cgi
In an email last week [1] was proposed an idea for how SIGs could leverage the centos namespace in gitlab for those who wishes.
One of the benefits of using gitlab would be increased flexibility for SIGs and a clear example for this would be the ability to drop the branch structures currently imposed on the git repositories. That structure is imposed because the git repositories are shared between CentOS-Linux, CentOS-Stream and (potentially multiple) SIGs, so that structure ensures groups are not stepping on each other's toes. By moving the SIGs out of these shared repositories, imposing that structure is no longer needed.
However, since the lookaside cache relies on branch name, lifting that structure would break the lookaside cache.
I have already brought this idea to a few folks to see if the idea was sane. The consensus that emerged is:
- Introduce a new upload endpoint next to the existing one, something like: https://git.centos.org/sources/sig_upload.cgi
- That new endpoint would upload the sources given using the same structure as the one used for CentOS-Stream, but ensuring that the person uploading is member of at least one SIG.
The idea of using `sig_upload.cgi` instead of just replacing `upload.cgi` is the assumption that we want to preserve the current structure used for CentOS-Linux and CentOS-Stream, allowing to find more easily which sources are used where and not impacting the process Red Hat uses to push its releases.
Since the structures used by the two upload scripts are different, they will not conflict. What we will end up seeing is something like:
sources │ ├── pkg1 │ ├── c7 │ │ ├── hash1 │ │ └── hash2 │ ├── c8 │ │ ├── hash3 │ │ └── hash4 │ ├── tarball1 │ │ └── sha name │ │ └── hash5 │ │ └── tarball1 │ └── tarball2 │ └── sha name │ └── hash6 │ └── tarball2 │ └── pkg2 ├── c8 │ ├── hash7 │ └── hash8 ├── c8s │ ├── hash9 │ └── hash10 ├── tarball3 │ └── sha name │ └── hash11 │ └── tarball3 └── tarball4 └── sha name └── hash12 └── tarball4 and so on
As this already requires some changes, does it make sense to have SIG sources in a different directory than RHEL sources? The proposed structure does not allow sharing sources between RHEL and SIGs so what is the benefit of having both in the same directory? It might even lead to confusion due to the two different structures used.
I.e. put SIG sources in git.centos.org/SIGs/sources or whatever is possible/preferred and have a upload.cgi there. I personally prefer a clean separation between content provided by RH and SIGs.
Going forward, only SIGs will use git.centos.org by default. CentOS Stream 9 isn't done there, so you get a much stronger separation now.
Afaik RHEL9 sources will still be pushed to git.centos.org. At least the RHEL 9 beta sources have already been pushed there.
On 22/02/2022 15:35, Peter Georg wrote:
<snip>
Going forward, only SIGs will use git.centos.org by default. CentOS Stream 9 isn't done there, so you get a much stronger separation now.
Afaik RHEL9 sources will still be pushed to git.centos.org. At least the RHEL 9 beta sources have already been pushed there.
Yes, and that's not going to change (afaik, and as announced by Red Hat when the stream plan was launched and still listed on https://centos.org/distro-faq/#q3-will-the-source-code-for-red-hat-enterpris...)
-
Brian Stinson -
Fabian Arrotin -
Neal Gompa -
Peter Georg -
Pierre-Yves Chibon