As part of opening up the possibilities for SIGs and Variants within the CentOS ecosystem, we need to give some thought to how repositories should be structured so that different can cooperate with each other.
What we have discussed in the past, and what I'd like to propose to the -devel community, is that we adopt a page similar to the KDE framework. The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update or replace anything in the core distribution. (Example: EPEL)
- Tier 2 repositories: These repositories may provide provide packages that update or otherwise enhance packages in the base distribution, but otherwise maintain compatibility. (Example IUS
- Tier 3 repositories: These repositories contain packages that conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
Each tier may choose to rely on dependencies from repositories in the tier above it, but not the other way around. For example, the tier 3 Xen4 repository could choose to rely on packages from a tier1 like EPEL, or a tier2 like IUS. However a tier1 or tier2 could not rely on a tier3. A good real-world example of this would be rpmfusion's dependence on the EPEL repository.
The other aspect of this that needs to be discussed is package duplication. For example, several of the proposed SIGs have some overlap in their packages. Would the preference here be that each SIG/Variant maintain their own version of the package, or should there be a common package set established? Both methods fit into the model described above, and both have pros/cons associated.
What does the community think about this?
On 01/13/2014 08:05 PM, Jim Perrin wrote:
As part of opening up the possibilities for SIGs and Variants within the CentOS ecosystem, we need to give some thought to how repositories should be structured so that different can cooperate with each other.
What we have discussed in the past, and what I'd like to propose to the -devel community, is that we adopt a page similar to the KDE framework. The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
- Tier 2 repositories: These repositories may provide provide packages
that update or otherwise enhance packages in the base distribution, but otherwise maintain compatibility. (Example IUS
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
Each tier may choose to rely on dependencies from repositories in the tier above it, but not the other way around. For example, the tier 3 Xen4 repository could choose to rely on packages from a tier1 like EPEL, or a tier2 like IUS. However a tier1 or tier2 could not rely on a tier3. A good real-world example of this would be rpmfusion's dependence on the EPEL repository.
What about a tier 1 relying upon a tier 2? It seems like there could be some ABI issues in allowing that unless I'm missing some detail.
The other aspect of this that needs to be discussed is package duplication. For example, several of the proposed SIGs have some overlap in their packages. Would the preference here be that each SIG/Variant maintain their own version of the package, or should there be a common package set established? Both methods fit into the model described above, and both have pros/cons associated.
IMO the SIG's should both use the same package and both groups could help maintain it. Obviously there will likely be situations where one SIG needs a certain version and the other needs a different one; seems like a situation where SIG's should work upstream to unify versions. As for whether that's a viable approach (re: upstream coordination) will probably only be told with time.
What does the community think about this?
On 01/13/2014 01:14 PM, Sam Kottler wrote:
On 01/13/2014 08:05 PM, Jim Perrin wrote:
As part of opening up the possibilities for SIGs and Variants within the CentOS ecosystem, we need to give some thought to how repositories should be structured so that different can cooperate with each other.
What we have discussed in the past, and what I'd like to propose to the -devel community, is that we adopt a page similar to the KDE framework. The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
- Tier 2 repositories: These repositories may provide provide packages
that update or otherwise enhance packages in the base distribution, but otherwise maintain compatibility. (Example IUS
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
Each tier may choose to rely on dependencies from repositories in the tier above it, but not the other way around. For example, the tier 3 Xen4 repository could choose to rely on packages from a tier1 like EPEL, or a tier2 like IUS. However a tier1 or tier2 could not rely on a tier3. A good real-world example of this would be rpmfusion's dependence on the EPEL repository.
What about a tier 1 relying upon a tier 2? It seems like there could be some ABI issues in allowing that unless I'm missing some detail.
This would not be permitted. A tier 2 could rely on a tier 1, but a tier 1 would not be allowed to rely on a tier 2. Apologies if I wasn't clear about that. I probably could have worded it better.
The other aspect of this that needs to be discussed is package duplication. For example, several of the proposed SIGs have some overlap in their packages. Would the preference here be that each SIG/Variant maintain their own version of the package, or should there be a common package set established? Both methods fit into the model described above, and both have pros/cons associated.
IMO the SIG's should both use the same package and both groups could help maintain it. Obviously there will likely be situations where one SIG needs a certain version and the other needs a different one; seems like a situation where SIG's should work upstream to unify versions. As for whether that's a viable approach (re: upstream coordination) will probably only be told with time.
I like this approach, but how would you propose resolving two SIGs with conflicting patch requirements?
On 01/13/2014 08:24 PM, Jim Perrin wrote:
On 01/13/2014 01:14 PM, Sam Kottler wrote:
On 01/13/2014 08:05 PM, Jim Perrin wrote:
As part of opening up the possibilities for SIGs and Variants within the CentOS ecosystem, we need to give some thought to how repositories should be structured so that different can cooperate with each other.
What we have discussed in the past, and what I'd like to propose to the -devel community, is that we adopt a page similar to the KDE framework. The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
- Tier 2 repositories: These repositories may provide provide packages
that update or otherwise enhance packages in the base distribution, but otherwise maintain compatibility. (Example IUS
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
Each tier may choose to rely on dependencies from repositories in the tier above it, but not the other way around. For example, the tier 3 Xen4 repository could choose to rely on packages from a tier1 like EPEL, or a tier2 like IUS. However a tier1 or tier2 could not rely on a tier3. A good real-world example of this would be rpmfusion's dependence on the EPEL repository.
What about a tier 1 relying upon a tier 2? It seems like there could be some ABI issues in allowing that unless I'm missing some detail.
This would not be permitted. A tier 2 could rely on a tier 1, but a tier 1 would not be allowed to rely on a tier 2. Apologies if I wasn't clear about that. I probably could have worded it better.
I see, so packages can only rely on packages that fall below their tier.
The other aspect of this that needs to be discussed is package duplication. For example, several of the proposed SIGs have some overlap in their packages. Would the preference here be that each SIG/Variant maintain their own version of the package, or should there be a common package set established? Both methods fit into the model described above, and both have pros/cons associated.
IMO the SIG's should both use the same package and both groups could help maintain it. Obviously there will likely be situations where one SIG needs a certain version and the other needs a different one; seems like a situation where SIG's should work upstream to unify versions. As for whether that's a viable approach (re: upstream coordination) will probably only be told with time.
I like this approach, but how would you propose resolving two SIGs with conflicting patch requirements?
Generally it seems like the 'work upstream to fix it' approach would work in this case, too. As for patches that are mutually exclusive I don't have any real insight; I'm sure others have approaches that work well to solve the problem, though. That being said I think it's actually fairly rare for two patches to be mutually exclusive in tier 1 or 2. Maybe tier 3 should have a different policy around duplication across SIG's than the lower two?
This seems natural anyhow since the reason tier 3 packages would generally exist is that there's something about them that make particularly different from the mainstream version.
On Mon, Jan 13, 2014 at 1:32 PM, Sam Kottler s@shk.io wrote:
This would not be permitted. A tier 2 could rely on a tier 1, but a tier 1 would not be allowed to rely on a tier 2. Apologies if I wasn't clear about that. I probably could have worded it better.
I see, so packages can only rely on packages that fall below their tier.
I'd expect lots of things to depend on EPEL. And at the same time offering packages that might eventually be added to EPEL.
I like this approach, but how would you propose resolving two SIGs with conflicting patch requirements?
Generally it seems like the 'work upstream to fix it' approach would work in this case, too. As for patches that are mutually exclusive I don't have any real insight; I'm sure others have approaches that work well to solve the problem, though. That being said I think it's actually fairly rare for two patches to be mutually exclusive in tier 1 or 2.
I think you are being optimistic, if you consider that you aren't starting from scratch. I'd guess that there would be a bunch of conflicts in terms of same-named packages with differences or different packages with same-named files across ClearOS/SME/Nethserver. And maybe even contents that don't conflict during install but have subtle differences that will break if you have the wrong one. Maybe these things can be worked out before putting them in a layout where mismatching components is easy, though.
Maybe tier 3 should have a different policy around duplication across SIG's than the lower two?
Assuming that all of these repos will be forced to have the same policies about allowed content as EPEL, the nicest thing to do would be to push as much as possible into EPEL with the specialized versions just managing configuration options. And that leaves us with the additional problem of needing uncoordinated repos to contain packages that EPEL won't accept.
On 01/13/2014 01:47 PM, Les Mikesell wrote:
On Mon, Jan 13, 2014 at 1:32 PM, Sam Kottler s@shk.io wrote:
This would not be permitted. A tier 2 could rely on a tier 1, but a tier 1 would not be allowed to rely on a tier 2. Apologies if I wasn't clear about that. I probably could have worded it better.
I see, so packages can only rely on packages that fall below their tier.
I'd expect lots of things to depend on EPEL. And at the same time offering packages that might eventually be added to EPEL.
I like this approach, but how would you propose resolving two SIGs with conflicting patch requirements?
Generally it seems like the 'work upstream to fix it' approach would work in this case, too. As for patches that are mutually exclusive I don't have any real insight; I'm sure others have approaches that work well to solve the problem, though. That being said I think it's actually fairly rare for two patches to be mutually exclusive in tier 1 or 2.
I think you are being optimistic, if you consider that you aren't starting from scratch. I'd guess that there would be a bunch of conflicts in terms of same-named packages with differences or different packages with same-named files across ClearOS/SME/Nethserver. And maybe even contents that don't conflict during install but have subtle differences that will break if you have the wrong one. Maybe these things can be worked out before putting them in a layout where mismatching components is easy, though.
Maybe tier 3 should have a different policy around duplication across SIG's than the lower two?
Assuming that all of these repos will be forced to have the same policies about allowed content as EPEL, the nicest thing to do would be to push as much as possible into EPEL with the specialized versions just managing configuration options. And that leaves us with the additional problem of needing uncoordinated repos to contain packages that EPEL won't accept.
Of course, we need to TRY to make it so that things from (2) "tier 3" can be used together on the same machines. How we might be able to that is one question. Some we might not be able to work together, but in general it should be a goal htat they do.
Hi,
On 01/13/2014 08:32 PM, Sam Kottler wrote:
On 01/13/2014 08:24 PM, Jim Perrin wrote:
What about a tier 1 relying upon a tier 2? It seems like there could be some ABI issues in allowing that unless I'm missing some detail.
This would not be permitted. A tier 2 could rely on a tier 1, but a tier 1 would not be allowed to rely on a tier 2. Apologies if I wasn't clear about that. I probably could have worded it better.
I see, so packages can only rely on packages that fall below their tier.
Makes sense - if I enable EPEL, I don't want to have packages there which require me to enable a different repo. But if I enable the RPO repo-, I'm totally OK with it enabling EPEL as a dependency.
Cheers, Dave.
On Mon, Jan 13, 2014 at 1:05 PM, Jim Perrin jperrin@centos.org wrote:
The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
What about the scenario where you want packages from more than one tier 1 repo? Will there be some coordination to at least try to avoid conflicts across them or document the workarounds if they happen? It will probably be common to require EPEL in addition to at least one other tier 1 (unless there is a horrible amount of duplication) - so what will prevent the addition of packages into EPEL that conflict with other tier 1's even if they start out without conflicts?
On 01/13/2014 08:18 PM, Les Mikesell wrote:
On Mon, Jan 13, 2014 at 1:05 PM, Jim Perrin jperrin@centos.org wrote:
The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
What about the scenario where you want packages from more than one tier 1 repo? Will there be some coordination to at least try to avoid conflicts across them or document the workarounds if they happen? It will probably be common to require EPEL in addition to at least one other tier 1 (unless there is a horrible amount of duplication) - so what will prevent the addition of packages into EPEL that conflict with other tier 1's even if they start out without conflicts?
I proposed (in thread "Repositories in new ecosystem and Desktop version") that yum priorities levels/numbers (and Manuel Wolfshant proposed exclude/include) are being agreed on for CentOS maintained and 3rd party repositories. The whole proposal is in that thread, and I was hoping to get some responses anyway.
On 01/13/2014 01:18 PM, Les Mikesell wrote:
On Mon, Jan 13, 2014 at 1:05 PM, Jim Perrin jperrin@centos.org wrote:
The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
What about the scenario where you want packages from more than one tier 1 repo? Will there be some coordination to at least try to avoid conflicts across them or document the workarounds if they happen? It will probably be common to require EPEL in addition to at least one other tier 1 (unless there is a horrible amount of duplication) - so what will prevent the addition of packages into EPEL that conflict with other tier 1's even if they start out without conflicts?
Yes. To some extent the scenario you've described is our starting point, as there are already a number of reasonably well known repositories that conflict with each other. This is part of what we want to address. By opening up git, and by allowing these organizations to work with us on the inside I'm hoping that we can create and foster this sort of cooperation.
On 13/01/14 19:05, Jim Perrin wrote:
As part of opening up the possibilities for SIGs and Variants within the CentOS ecosystem, we need to give some thought to how repositories should be structured so that different can cooperate with each other.
What we have discussed in the past, and what I'd like to propose to the -devel community, is that we adopt a page similar to the KDE framework. The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
- Tier 2 repositories: These repositories may provide provide packages
that update or otherwise enhance packages in the base distribution, but otherwise maintain compatibility. (Example IUS
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
Each tier may choose to rely on dependencies from repositories in the tier above it, but not the other way around. For example, the tier 3 Xen4 repository could choose to rely on packages from a tier1 like EPEL, or a tier2 like IUS. However a tier1 or tier2 could not rely on a tier3. A good real-world example of this would be rpmfusion's dependence on the EPEL repository.
The other aspect of this that needs to be discussed is package duplication. For example, several of the proposed SIGs have some overlap in their packages. Would the preference here be that each SIG/Variant maintain their own version of the package, or should there be a common package set established? Both methods fit into the model described above, and both have pros/cons associated.
What does the community think about this?
I'm in favour of any process that makes the use of 3rd party repositories more compatible.
Some things to think about:
How would one handle repositories that fall into multiple categories above. For example, Repoforge main repo channel would be tier 1 (.rf packages) whereas packages from their extras channel (.rfx) would be tier 2. Likewise, elrepo has a main tier 1 channel, a tier 2 extras channel and a kernel channel which might be tier 3.
How would one handle conflicts and incompatibilities between multiple tier 1 repositories (e.g, Repoforge and EPEL)?
Of course these aren't new issues and one can easily write a very long list.
As much as I hate the idea of killing off independent 3rd party competition, I'm minded to think the only sane way to handle this is to have one centralised Enterprise Linux repository, quite possibly segregated into 3 tiers as you outline above. As exemplified by the current situation, it's never going to be completely practical having multiple builds of Package A built against different versions of Packages X, Y and Z all floating around in different 3rd party community repositories.
Whether anyone (or any group) is able to demonstrate the leadership necessary to bring everyone together under one umbrella and resolve all the issues is another question.
Just my personal opinion(s).
On 01/15/2014 01:28 AM, Ned Slider wrote:
On 13/01/14 19:05, Jim Perrin wrote:
As part of opening up the possibilities for SIGs and Variants within the CentOS ecosystem, we need to give some thought to how repositories should be structured so that different can cooperate with each other.
What we have discussed in the past, and what I'd like to propose to the -devel community, is that we adopt a page similar to the KDE framework. The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
- Tier 2 repositories: These repositories may provide provide packages
that update or otherwise enhance packages in the base distribution, but otherwise maintain compatibility. (Example IUS
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
Each tier may choose to rely on dependencies from repositories in the tier above it, but not the other way around. For example, the tier 3 Xen4 repository could choose to rely on packages from a tier1 like EPEL, or a tier2 like IUS. However a tier1 or tier2 could not rely on a tier3. A good real-world example of this would be rpmfusion's dependence on the EPEL repository.
The other aspect of this that needs to be discussed is package duplication. For example, several of the proposed SIGs have some overlap in their packages. Would the preference here be that each SIG/Variant maintain their own version of the package, or should there be a common package set established? Both methods fit into the model described above, and both have pros/cons associated.
What does the community think about this?
I'm in favour of any process that makes the use of 3rd party repositories more compatible.
Some things to think about:
How would one handle repositories that fall into multiple categories above. For example, Repoforge main repo channel would be tier 1 (.rf packages) whereas packages from their extras channel (.rfx) would be tier 2. Likewise, elrepo has a main tier 1 channel, a tier 2 extras channel and a kernel channel which might be tier 3.
How would one handle conflicts and incompatibilities between multiple tier 1 repositories (e.g, Repoforge and EPEL)?
Of course these aren't new issues and one can easily write a very long list.
As much as I hate the idea of killing off independent 3rd party competition, I'm minded to think the only sane way to handle this is to have one centralised Enterprise Linux repository, quite possibly segregated into 3 tiers as you outline above. As exemplified by the current situation, it's never going to be completely practical having multiple builds of Package A built against different versions of Packages X, Y and Z all floating around in different 3rd party community repositories.
Whether anyone (or any group) is able to demonstrate the leadership necessary to bring everyone together under one umbrella and resolve all the issues is another question.
Just my personal opinion(s).
We are only talking about CentOS Special Interest Groups and CentOS Variants, not all 3rd Party repositories. And we can't host anything that is encumbered (IP restrictions like MP3, not a license that allows for distribution, etc).
On Wed, Jan 15, 2014 at 5:01 AM, Johnny Hughes johnny@centos.org wrote:
We are only talking about CentOS Special Interest Groups and CentOS Variants, not all 3rd Party repositories. And we can't host anything that is encumbered (IP restrictions like MP3, not a license that allows for distribution, etc).
I'd give a lot of credit for the popularity of Ubuntu to their more sane handling of 3rd party packages. Even the ones they don't host directly are generally available from a repository that is easily enabled (without hunting for it, or guessing which ones will break your system), and generally coordinated to avoid conflicts. Maybe some only-slightly associated site could host the repo-release rpms for one or more repositories that are able to host media players, etc. and that will try to avoid conflicts with each other.
On 01/15/2014 06:09 PM, Les Mikesell wrote:
I'd give a lot of credit for the popularity of Ubuntu to their more sane handling of 3rd party packages. Even the ones they don't host directly are generally available from a repository that is easily enabled (without hunting for it, or guessing which ones will break your system), and generally coordinated to avoid conflicts.
if someone can :
- define what 'breaks your system is' - write code that can test for 'breaks' - is happy to maintain that set of code
we can plumb that into the CI / nightly / pre-release testing to make sure that we can atleast notify the right people in time; ideally building upto proper coordination.
if you cant automate this, then its an education process. Feel free to start write docs and policies, then educating people around it - Fedora's knowldge base is a good place to start from
- KB
On Wed, Jan 15, 2014 at 12:16 PM, Karanbir Singh mail-lists@karan.org wrote:
I'd give a lot of credit for the popularity of Ubuntu to their more
sane handling of 3rd party packages. Even the ones they don't host directly are generally available from a repository that is easily enabled (without hunting for it, or guessing which ones will break your system), and generally coordinated to avoid conflicts.
if someone can :
- define what 'breaks your system is'
I don't think that is possible in the broad sense - it could be as subtle as a package being configured to listen on the same port as an existing service uses. The obvious things are having same-named packages with different contents or same-named files owned by different packages.
- write code that can test for 'breaks'
How do you test the package contents of unknown 3rd party repos against your own? And in an update you expect the contents to change anyway. I don't know if it is still true, but at least at some time in the past EPEL and RPMFORGE both had 'viewvc' packages with incompatible configurations. Whenever one would bump its version number it would update and the one from the alternate repo would be misconfigured. I think I got into that state because EPEL didn't have the package initially but added it after I had installed from RPMFORGE. Some of those problems could be avoided if yum only considered updates from the repo that provided it in the first place unless you do something to tell it to switch.
we can plumb that into the CI / nightly / pre-release testing to make sure that we can atleast notify the right people in time; ideally building upto proper coordination.
I think you should make the assumption that everyone will have EPEL enabled. So every package added to EPEL creates potential new conflicts with every other 3rd party. And since EPEL has restrictions on what they will accept, many people will need to use other 3rd party repos.
if you cant automate this, then its an education process. Feel free to start write docs and policies, then educating people around it - Fedora's knowldge base is a good place to start from
I don't think it is possible/practical for you to test against the contents of all 3rd party repos. Is there a way to reverse the concept and make it easier for the people maintaining the repos of content that you - and EPEL - won't accept to test for conflicts without being strictly coordinated? And what would you suggest as the 'right' way to handle a conflict created when EPEL adds a package that breaks one that had previously been available elsewhere, like the viewvc example above? This isn't strictly a CentOS problem, but it is a problem that CentOS users have to deal with (and RHEL users, although there probably aren't many people paying for RHEL support to build a home media player...)
On 01/15/2014 08:28 AM, Ned Slider wrote:
On 13/01/14 19:05, Jim Perrin wrote:
As part of opening up the possibilities for SIGs and Variants within the CentOS ecosystem, we need to give some thought to how repositories should be structured so that different can cooperate with each other.
What we have discussed in the past, and what I'd like to propose to the -devel community, is that we adopt a page similar to the KDE framework. The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
- Tier 2 repositories: These repositories may provide provide packages
that update or otherwise enhance packages in the base distribution, but otherwise maintain compatibility. (Example IUS
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
Each tier may choose to rely on dependencies from repositories in the tier above it, but not the other way around. For example, the tier 3 Xen4 repository could choose to rely on packages from a tier1 like EPEL, or a tier2 like IUS. However a tier1 or tier2 could not rely on a tier3. A good real-world example of this would be rpmfusion's dependence on the EPEL repository.
The other aspect of this that needs to be discussed is package duplication. For example, several of the proposed SIGs have some overlap in their packages. Would the preference here be that each SIG/Variant maintain their own version of the package, or should there be a common package set established? Both methods fit into the model described above, and both have pros/cons associated.
What does the community think about this?
I'm in favour of any process that makes the use of 3rd party repositories more compatible.
Some things to think about:
How would one handle repositories that fall into multiple categories above. For example, Repoforge main repo channel would be tier 1 (.rf packages) whereas packages from their extras channel (.rfx) would be tier 2. Likewise, elrepo has a main tier 1 channel, a tier 2 extras channel and a kernel channel which might be tier 3.
How would one handle conflicts and incompatibilities between multiple tier 1 repositories (e.g, Repoforge and EPEL)?
Of course these aren't new issues and one can easily write a very long list.
As much as I hate the idea of killing off independent 3rd party competition, I'm minded to think the only sane way to handle this is to have one centralised Enterprise Linux repository, quite possibly segregated into 3 tiers as you outline above. As exemplified by the current situation, it's never going to be completely practical having multiple builds of Package A built against different versions of Packages X, Y and Z all floating around in different 3rd party community repositories.
Whether anyone (or any group) is able to demonstrate the leadership necessary to bring everyone together under one umbrella and resolve all the issues is another question.
First thing to do is to get all interested 3rd party repository maintainers and see if agreement can be made.
Things have changed now that CentOS will have additional Variants/repositories. Many repositories were created by individuals that needed some packages but didn't have access to (unified) building environment. But with git.centos.org many 3rd party repositories maintainers will be happy to move majority of packages "upstream" (CentOS project or EPEL), where they will have the same environment to maintain packages the are interested in. Especially having in mind package version consistency and better building infrastructure + more people working together.
Only problem I see will be non-opensource packages, like addition codecs for gstreamer. Those will have problems keeping up with "upstream" (CentOS/RHEL) repositories, unless source packages carry all of it and non-opensource parts are just disabled during build, so that others can use same src.rpm to build missing packages.
On 01/15/2014 01:28 AM, Ned Slider wrote:
On 13/01/14 19:05, Jim Perrin wrote:
As part of opening up the possibilities for SIGs and Variants within the CentOS ecosystem, we need to give some thought to how repositories should be structured so that different can cooperate with each other.
What we have discussed in the past, and what I'd like to propose to the -devel community, is that we adopt a page similar to the KDE framework. The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
- Tier 2 repositories: These repositories may provide provide packages
that update or otherwise enhance packages in the base distribution, but otherwise maintain compatibility. (Example IUS
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
Each tier may choose to rely on dependencies from repositories in the tier above it, but not the other way around. For example, the tier 3 Xen4 repository could choose to rely on packages from a tier1 like EPEL, or a tier2 like IUS. However a tier1 or tier2 could not rely on a tier3. A good real-world example of this would be rpmfusion's dependence on the EPEL repository.
The other aspect of this that needs to be discussed is package duplication. For example, several of the proposed SIGs have some overlap in their packages. Would the preference here be that each SIG/Variant maintain their own version of the package, or should there be a common package set established? Both methods fit into the model described above, and both have pros/cons associated.
What does the community think about this?
I'm in favour of any process that makes the use of 3rd party repositories more compatible.
Some things to think about:
How would one handle repositories that fall into multiple categories above. For example, Repoforge main repo channel would be tier 1 (.rf packages) whereas packages from their extras channel (.rfx) would be tier 2. Likewise, elrepo has a main tier 1 channel, a tier 2 extras channel and a kernel channel which might be tier 3.
This is primarily for SIG and Variant repos, since we can't control what 3rd parties do. It would be rather nice to get everyone working in the same direction. As you said, repoforge and elrepo already mostly line up with this.
How would one handle conflicts and incompatibilities between multiple tier 1 repositories (e.g, Repoforge and EPEL)?
Ljubomir and Manuel were discussing the required use of include/exclude statements for repository structures, but I'm not sold on that idea yet.
Since this would be for sig/variant use, and in git.centos.org we should be able to reduce the level of duplication and have groups working together.
Apart from that, I think it would best be left to the SIGs to document their requirements and educate their users about any potential for overlap.
Of course these aren't new issues and one can easily write a very long list.
As much as I hate the idea of killing off independent 3rd party competition, I'm minded to think the only sane way to handle this is to have one centralised Enterprise Linux repository, quite possibly segregated into 3 tiers as you outline above. As exemplified by the current situation, it's never going to be completely practical having multiple builds of Package A built against different versions of Packages X, Y and Z all floating around in different 3rd party community repositories.
Agreed. This is a systemic failing that's been around for years. My goal here is simply to provide some basic structure and resources (git, build system, mirror network, and easier deployment within the distro) to the individuals or groups who want to participate. The structure we're outlining is mostly common sense, and is mostly in use already. That should hopefully help smooth the transition for people wishing to participate.
Whether anyone (or any group) is able to demonstrate the leadership necessary to bring everyone together under one umbrella and resolve all the issues is another question.
We'll never be able to make everyone happy or resolve all the issues. If we're able to provide something that works well for the majority of end users and makes project life easier for the builders/developers, I'll consider that a win.
On 01/15/2014 02:47 PM, Jim Perrin wrote:
We'll never be able to make everyone happy or resolve all the issues. If we're able to provide something that works well for the majority of end users and makes project life easier for the builders/developers, I'll consider that a win.
There is no need to over complicate this. Majority will be happy to have "Enabled=" and "Priority=x" in CentOS-Base.repo (and other repo's), and to define priority values for Tier1, Tier2, Tier3 with enough space in-between for 3rd party repositories to nest them selves where they want.
Additional consensus between 3rd party repository maintainers to agree on priority values and which repos should be Enabled by default would be nice, but it is not necessary because the fact that newbies do not have to mess with CentOS-Base.repo and 3rd party repos for default instalation of CentOS + 3rd party will dramatically boost usability to newbies who want stable Linux, even if CentOS Project does not provide Installation media with EPEL and/or ElRepo kmods.
So far, every newbie that installs CentOS on Laptop (and Desktop/Workstation) and needs additional drivers has to go through hell of manual downloading and instalation of release packages, then manual installation of kmods for their wireless/ethernet to work. And so does person who they contact for help. Not to mention if package they need is in tier 2 or 3.
But with priorities already set, task is 90% done, no messing around with .repo files, just fire and forget.
I know that I am pushing hard for this, but that is because I have been waiting this opportunity for several years, and if priority reform is not done, then it will be epic fail in my book for wider adoption of CentOS in non-server environments.
On 01/15/2014 11:15 AM, Ljubomir Ljubojevic wrote:
On 01/15/2014 02:47 PM, Jim Perrin wrote:
We'll never be able to make everyone happy or resolve all the issues. If we're able to provide something that works well for the majority of end users and makes project life easier for the builders/developers, I'll consider that a win.
There is no need to over complicate this. Majority will be happy to have "Enabled=" and "Priority=x" in CentOS-Base.repo (and other repo's), and to define priority values for Tier1, Tier2, Tier3 with enough space in-between for 3rd party repositories to nest them selves where they want.
This could also provide added complication. Let me explain via a hypothetical situation:
We establish a priority structure for repos moving forward, but for the idea to work, the expected structure must be inverted. If the cloud sig provides a newer libvirt, which is covered by the base repository's lower priority value, then you run into problems trying to install the cloud sig's packages.
Additional consensus between 3rd party repository maintainers to agree on priority values and which repos should be Enabled by default would be nice, but it is not necessary because the fact that newbies do not have to mess with CentOS-Base.repo and 3rd party repos for default instalation of CentOS + 3rd party will dramatically boost usability to newbies who want stable Linux, even if CentOS Project does not provide Installation media with EPEL and/or ElRepo kmods.
With 3rd party repositories in the mix, this becomes more of an issue. Even if we establish standard ranges, and get ALL the 3rd parties to agree to both our structure and the established ranges (unlikely), you could potentially still end up with repoforge and epel (still hypothetical, still example, no one freak out) having a dependency conflict during an update, because one had a package that didn't exist in the other but dragged in a conflicting dep chain.
We cannot dictate what 3rd parties do, and we'll very likely have to go through a period of proving to them that becoming a SIG and/or working together is a good idea. There's quite a bit of tension about this from many past experiences.
So far, every newbie that installs CentOS on Laptop (and Desktop/Workstation) and needs additional drivers has to go through hell of manual downloading and instalation of release packages, then manual installation of kmods for their wireless/ethernet to work. And so does person who they contact for help. Not to mention if package they need is in tier 2 or 3.
I agree, but this is where the variants can come into play. Once we have the infra in place to do what we're discussing, there's absolutely nothing stopping someone from making a 'CentOS for laptops' or whatever that has these bits baked in. All it takes is git, and some interested parties to lead and support the SIG.
But with priorities already set, task is 90% done, no messing around with .repo files, just fire and forget.
I disagree. I don't see priorities as a solution to the problem, but more of a half-measure. The priorities and protectbase plugins are walls to keep conflicting packages out. We're trying to keep conflicting packages from existing. It's a different view of the problem.
I know that I am pushing hard for this, but that is because I have been waiting this opportunity for several years, and if priority reform is not done, then it will be epic fail in my book for wider adoption of CentOS in non-server environments.
Good! I want folks caring about CentOS and its guidance. The more folks we have involved, the more things we're able to do.
On Thu, 2014-01-16 at 06:45 -0600, Jim Perrin wrote:
We cannot dictate what 3rd parties do, and we'll very likely have to go through a period of proving to them that becoming a SIG and/or working together is a good idea. There's quite a bit of tension about this from many past experiences.
Working together is a *great* idea, but specifically in the context of third-party repositories, I'd personally see it more as a manpower issue (and mostly on their/our side).
Regarding the past tensions, I'm really hopeful that in view of the recent exciting news, the most prominent sources thereof will become something of the past.
On 01/16/2014 02:31 PM, Yury V. Zaytsev wrote:
On Thu, 2014-01-16 at 06:45 -0600, Jim Perrin wrote:
We cannot dictate what 3rd parties do, and we'll very likely have to go through a period of proving to them that becoming a SIG and/or working together is a good idea. There's quite a bit of tension about this from many past experiences.
Working together is a *great* idea, but specifically in the context of third-party repositories, I'd personally see it more as a manpower issue (and mostly on their/our side).
Regarding the past tensions, I'm really hopeful that in view of the recent exciting news, the most prominent sources thereof will become something of the past.
Yury, you speak as a maintainer still involved with RepoForge/RPMForge, right?
On Thu, 2014-01-16 at 16:43 +0100, Ljubomir Ljubojevic wrote:
Yury, you speak as a maintainer still involved with RepoForge/RPMForge, right?
To clarify this point, I spoke in both my personal capacity and as a maintainer that was previously involved with RepoForge/RPMForge, but has been dormant for the last couple of years (irrespectively of which, the project is currently not doing very well).
I'm not sure what implications this is going to have on your interpretation of my words, though :)
On 01/16/2014 04:58 PM, Yury V. Zaytsev wrote:
On Thu, 2014-01-16 at 16:43 +0100, Ljubomir Ljubojevic wrote:
Yury, you speak as a maintainer still involved with RepoForge/RPMForge, right?
To clarify this point, I spoke in both my personal capacity and as a maintainer that was previously involved with RepoForge/RPMForge, but has been dormant for the last couple of years (irrespectively of which, the project is currently not doing very well).
I'm not sure what implications this is going to have on your interpretation of my words, though :)
I only wanted to point out that you are one of the few involved in RepoForge/RPMForge, more related to Jim Perrin using EPEL - RPMForge conflicts against using Priority= more effectively. So you saying: "Regarding the past tensions, I'm really hopeful that in view of the recent exciting news, the most prominent sources thereof will become something of the past." has little more weight then someone else saying it, and I really want to convince him and others that conflict ARE thing of the past.
I hope you do not mind.
On 01/16/2014 01:45 PM, Jim Perrin wrote:
On 01/15/2014 11:15 AM, Ljubomir Ljubojevic wrote:
On 01/15/2014 02:47 PM, Jim Perrin wrote:
We'll never be able to make everyone happy or resolve all the issues. If we're able to provide something that works well for the majority of end users and makes project life easier for the builders/developers, I'll consider that a win.
There is no need to over complicate this. Majority will be happy to have "Enabled=" and "Priority=x" in CentOS-Base.repo (and other repo's), and to define priority values for Tier1, Tier2, Tier3 with enough space in-between for 3rd party repositories to nest them selves where they want.
This could also provide added complication. Let me explain via a hypothetical situation:
We establish a priority structure for repos moving forward, but for the idea to work, the expected structure must be inverted. If the cloud sig provides a newer libvirt, which is covered by the base repository's lower priority value, then you run into problems trying to install the cloud sig's packages.
What I am striving for, in first round, to have "Priority" and "Enabled" lines into official .repo files. Once that is in place, everything else are next phases and will be much easier to deal with.
Those next phases will depend on how things progress, but they will basically come down to analyzing what repositories and packages are created within CentOS project and EPEL repositories.
Even if some of 3rd party repos decide not to play along (only thing they can do is to set undesirable priority value), they will be either marginalized or users would just be instructed/advised to change Priority value to what we think should be.
Notice that professionals already know how to deal with package conflicts. It's the newbies that are my focus in this quest.
One of the steps I am hoping for is (primarily GUI) package manager that will have profiles, both predetermined and installable via separate packages. That way we as a project, or someone outside of the project could create repository profile that will be used for generic or specific use case (CentOS-Core + EPEL + RPMFusion + repoX) already preconfigured, and user only needs to install package with profile we tell him to.
I already experimented with this, and my best idea is to have separate (~backup) directory for every profile installed (/etc/yum.repos.d/core, /etc/yum.repos.d/ovirt, /etc/yum.repos.d/desktop, etc.) and to have simple command for switching them to new profile, and to either:
1. copy .repo files from selected directory/profile to main directory (backuping and deleting everything prior to switch). My example package you can see here: http://rpms.plnet.rs/plnet-centos5-x86_64/RPMS.plnet-releases/plnet-dev-rele...
2. create symlinks from selected directory/profile to main /etc/yum.repos.d/
3. use a method that will call "yum -c <profile-yum.conf>" or "--set-opt=" to just point to desired configuration.
4. More complicated way would be to add alias for "yum" so it points to latest installed profile config, and "yum-default" for using CentOS-Core/default profile/config.
Since each profile would be in separate package, updating yum config can be easily made, and maybe using virtual package to pull-in all of the main profiles one Variant can use, could make sure switching to other profiles can be done even without internet connection.
Additional consensus between 3rd party repository maintainers to agree on priority values and which repos should be Enabled by default would be nice, but it is not necessary because the fact that newbies do not have to mess with CentOS-Base.repo and 3rd party repos for default instalation of CentOS + 3rd party will dramatically boost usability to newbies who want stable Linux, even if CentOS Project does not provide Installation media with EPEL and/or ElRepo kmods.
With 3rd party repositories in the mix, this becomes more of an issue. Even if we establish standard ranges, and get ALL the 3rd parties to agree to both our structure and the established ranges (unlikely), you could potentially still end up with repoforge and epel (still hypothetical, still example, no one freak out) having a dependency conflict during an update, because one had a package that didn't exist in the other but dragged in a conflicting dep chain.
We cannot dictate what 3rd parties do, and we'll very likely have to go through a period of proving to them that becoming a SIG and/or working together is a good idea. There's quite a bit of tension about this from many past experiences.
So far, every newbie that installs CentOS on Laptop (and Desktop/Workstation) and needs additional drivers has to go through hell of manual downloading and instalation of release packages, then manual installation of kmods for their wireless/ethernet to work. And so does person who they contact for help. Not to mention if package they need is in tier 2 or 3.
I agree, but this is where the variants can come into play. Once we have the infra in place to do what we're discussing, there's absolutely nothing stopping someone from making a 'CentOS for laptops' or whatever that has these bits baked in. All it takes is git, and some interested parties to lead and support the SIG.
But with priorities already set, task is 90% done, no messing around with .repo files, just fire and forget.
I disagree. I don't see priorities as a solution to the problem, but more of a half-measure. The priorities and protectbase plugins are walls to keep conflicting packages out. We're trying to keep conflicting packages from existing. It's a different view of the problem.
The problem is that that "base" is what RHEL caries. So protectbase plugin only protect CentOS-Core, and will not cover variant repositories?
If that is/becomes truth, then priorities/profiles will be necessary tool for accomplishing easy manipulation of various repositories and package configurations different Variants will need. And that is fine with me. CentOS-Core should stay as is, with only difference being provision of "hooks" other Variants can easily attach to so they can deliver packages they need that differ from CentOS-Core.
I know that I am pushing hard for this, but that is because I have been waiting this opportunity for several years, and if priority reform is not done, then it will be epic fail in my book for wider adoption of CentOS in non-server environments.
Good! I want folks caring about CentOS and its guidance. The more folks we have involved, the more things we're able to do.
Thanks.
On Thu, Jan 16, 2014 at 10:24 AM, Ljubomir Ljubojevic centos@plnet.rs wrote:
If that is/becomes truth, then priorities/profiles will be necessary tool for accomplishing easy manipulation of various repositories and package configurations different Variants will need. And that is fine with me. CentOS-Core should stay as is, with only difference being provision of "hooks" other Variants can easily attach to so they can deliver packages they need that differ from CentOS-Core.
How do you see this working for a package that is not in EPEL when you install it from some lower-priority 3rd party repo, but is subsequently added to EPEL with a higher version number and an incompatible configuration?
On 01/16/2014 06:04 PM, Les Mikesell wrote:
On Thu, Jan 16, 2014 at 10:24 AM, Ljubomir Ljubojevic centos@plnet.rs wrote:
If that is/becomes truth, then priorities/profiles will be necessary tool for accomplishing easy manipulation of various repositories and package configurations different Variants will need. And that is fine with me. CentOS-Core should stay as is, with only difference being provision of "hooks" other Variants can easily attach to so they can deliver packages they need that differ from CentOS-Core.
How do you see this working for a package that is not in EPEL when you install it from some lower-priority 3rd party repo, but is subsequently added to EPEL with a higher version number and an incompatible configuration?
1. Same as now. 2. User changing EPEL repo file adding "exclude=<package>". 3. Creating new repo profile that will change EPEL repo file adding "exclude=<package>".
Ultimately, user is responsible for using third party repositories that use different configuration but same package name, and he should be careful about updating it.
I do not have all the answers, the whole problem is very complex, but I think my solution would greatly reduce common problems.
On Thu, Jan 16, 2014 at 12:15 PM, Ljubomir Ljubojevic centos@plnet.rs wrote:
How do you see this working for a package that is not in EPEL when you install it from some lower-priority 3rd party repo, but is subsequently added to EPEL with a higher version number and an incompatible configuration?
- Same as now.
- User changing EPEL repo file adding "exclude=<package>".
- Creating new repo profile that will change EPEL repo file adding
"exclude=<package>".
Would you expect users to do that for every package installed from non-EPEL repos even if they don't currently exist in EPEL or to be alert enough during updates to notice when one has been added into EPEL?. I just wish there were some way that yum could tell you before it updates from a repo different than the one used for the initial install of a package.
On 01/16/2014 09:52 PM, Les Mikesell wrote:
On Thu, Jan 16, 2014 at 12:15 PM, Ljubomir Ljubojevic centos@plnet.rs wrote:
How do you see this working for a package that is not in EPEL when you install it from some lower-priority 3rd party repo, but is subsequently added to EPEL with a higher version number and an incompatible configuration?
- Same as now.
- User changing EPEL repo file adding "exclude=<package>".
- Creating new repo profile that will change EPEL repo file adding
"exclude=<package>".
Would you expect users to do that for every package installed from non-EPEL repos even if they don't currently exist in EPEL or to be alert enough during updates to notice when one has been added into EPEL?. I just wish there were some way that yum could tell you before it updates from a repo different than the one used for the initial install of a package.
That should be request for Yum project, not CentOS, but if we define what we would like, we could even create a yum plugin for that and make a default for any CentOS instalation.
On 01/15/2014 06:15 PM, Ljubomir Ljubojevic wrote:
On 01/15/2014 02:47 PM, Jim Perrin wrote:
We'll never be able to make everyone happy or resolve all the issues. If we're able to provide something that works well for the majority of end users and makes project life easier for the builders/developers, I'll consider that a win.
There is no need to over complicate this. Majority will be happy to have "Enabled=" and "Priority=x" in CentOS-Base.repo (and other repo's), and to define priority values for Tier1, Tier2, Tier3 with enough space in-between for 3rd party repositories to nest them selves where they want.
Additional consensus between 3rd party repository maintainers to agree on priority values and which repos should be Enabled by default would be nice, but it is not necessary because the fact that newbies do not have to mess with CentOS-Base.repo and 3rd party repos for default instalation of CentOS + 3rd party will dramatically boost usability to newbies who want stable Linux, even if CentOS Project does not provide Installation media with EPEL and/or ElRepo kmods.
So far, every newbie that installs CentOS on Laptop (and Desktop/Workstation) and needs additional drivers has to go through hell of manual downloading and instalation of release packages, then manual installation of kmods for their wireless/ethernet to work. And so does person who they contact for help. Not to mention if package they need is in tier 2 or 3.
But with priorities already set, task is 90% done, no messing around with .repo files, just fire and forget.
I know that I am pushing hard for this, but that is because I have been waiting this opportunity for several years, and if priority reform is not done, then it will be epic fail in my book for wider adoption of CentOS in non-server environments.
I forgot to add that yum-plugin-priorities should be made mandatory for any CentOS installation.
On 01/16/2014 06:12 PM, Ljubomir Ljubojevic wrote:
I forgot to add that yum-plugin-priorities should be made mandatory for any CentOS installation.
Adding some default lines to the repository configuration I'm open to, and we can discuss that as a change for an upcoming release.
Mandating a package installation like this I'm pretty strongly against. That decision should be for the sysadmin (or a variant spin) to make.
On 17/01/14 12:25, Jim Perrin wrote:
On 01/16/2014 06:12 PM, Ljubomir Ljubojevic wrote:
I forgot to add that yum-plugin-priorities should be made mandatory for any CentOS installation.
Adding some default lines to the repository configuration I'm open to, and we can discuss that as a change for an upcoming release.
Mandating a package installation like this I'm pretty strongly against. That decision should be for the sysadmin (or a variant spin) to make.
I think it might be good to add priority=1 to both base and updates repos in CentOS-Base.repo. If yum-plugin-priorities is installed then those two repos probably should be the highest priority they can be and if it isn't installed then they are no-ops so don't matter.
T
On 01/17/2014 02:29 PM, Trevor Hemsley wrote:
On 17/01/14 12:25, Jim Perrin wrote:
On 01/16/2014 06:12 PM, Ljubomir Ljubojevic wrote:
I forgot to add that yum-plugin-priorities should be made mandatory for any CentOS installation.
Adding some default lines to the repository configuration I'm open to, and we can discuss that as a change for an upcoming release.
Mandating a package installation like this I'm pretty strongly against. That decision should be for the sysadmin (or a variant spin) to make.
I think it might be good to add priority=1 to both base and updates
make it 5 or 10, please. one might want to supersede base packages with ones from private/local repos. hplip for instance was such a candidate for a long time, my printers were not supported by the stock version of hplip from EL5 and I had to compile a newer one from fedora. and at the time I was not yet relying on include/excludepkgs
repos in CentOS-Base.repo. If yum-plugin-priorities is installed then those two repos probably should be the highest priority they can be and if it isn't installed then they are no-ops so don't matter.
T
On 01/17/2014 07:08 AM, Manuel Wolfshant wrote:
make it 5 or 10, please. one might want to supersede base packages with ones from private/local repos. hplip for instance was such a candidate for a long time, my printers were not supported by the stock version of hplip from EL5 and I had to compile a newer one from fedora. and at the time I was not yet relying on include/excludepkgs
This touches exactly on the point for why I don't want to include priorities by default. To make it useful for repos that provide newer software than what exists in base/updates (php, httpd, libvirt, whatever), you're automatically working against priorities. It doesn't matter if it's for a local repository or for a SIG.
On 01/17/2014 02:35 PM, Jim Perrin wrote:
On 01/17/2014 07:08 AM, Manuel Wolfshant wrote:
make it 5 or 10, please. one might want to supersede base packages with ones from private/local repos. hplip for instance was such a candidate for a long time, my printers were not supported by the stock version of hplip from EL5 and I had to compile a newer one from fedora. and at the time I was not yet relying on include/excludepkgs
This touches exactly on the point for why I don't want to include priorities by default. To make it useful for repos that provide newer software than what exists in base/updates (php, httpd, libvirt, whatever), you're automatically working against priorities. It doesn't matter if it's for a local repository or for a SIG.
So how do you intend to solve this without priorities? Especially if there is a need for some packages to be compiled with different flags?
One example is for those using Virtualmin. Virtualmin provides http, postfix and few other packages compiled little differently (different flags) then RH does, in their own repository. So if both repositories are of same priority, when RH publishes newer version it is automatically an update to modified version. And that can brake things until Virtualmin releases their own version. If we "hide" RH versions with higher priority to Virtualmin packages then nothing can be broken.
Same rule will apply to any package any Variant will have to compile with different flags, including CentOSPlus kernels. If someone uses CentOSPlus kernel because regular RH kernel does not have drivers for their hardware, and happens that CentOSPlus kernel is not released the same time as regular kernel (even 1-5h can make a difference), problem may arise where update to regular kernel fails and we have a unhappy user. With newbies around (I hope we attract in greater numbers), that can be troublesome.
----------------------------------- But, as I said, for now I will be happy with "Priorities=" and "Enabled=" for every repo's config. It will make many things easier, like creating script to change those numbers per users request (you do not have to host it). Adding them to every new config (after centos-release is updated) is not desired. ----------------------------------
With new opportunities creating CentOS Variants, I already thought of few things I would ask of yum project to incorporate, to further provide out-of-the-box experience for Variants users.
With all of my suggestions so far, I think something like "Skip=" for every repo would be useful for those who use their internal mirrors. If you set "Enabled=0" for some repo, using "--enablerepo=*" will enable it. Another use is to have installed but disabled one or more 3rd party repos, so you can quickly locate and install some needed package. No need for search around internet, double checking.
So if you want to use internal mirror, you have two options:
1. "Enabled=0" for CentOS-Base.repo repos, where you loose flexibility of simple "--enablerepo=*" when you want to find something.
2. Removing CentOS-Base.repo to backup folder or commenting out entire file. Both options needing root's manual intervention (sudo yum authorization does not extend to editing files).
When internal mirror is not accessible, and you made changes to CentOS-Base.repo or moved it, then you need to revert the changes in order to use official mirrors. Again, root's manual intervention is needed.
So in order to have 1. but without "--enablerepo=*" problems, introduction of "Skip=1" when "Enabled=0" would allow "--enablerepo=*" use with many disabled and/or (currently) not accessible repositories. Or maybe "--enableactiverepo=*" or something like that.
Working around current "restrictions" in CentOS concerning 3rd party repository manipulation is possible, but I think CentOS-Core should be as close to Variants as possible in terms of repository syntax and use, so we do not have to build (in-house or 3rd party) additional layer over centos-release package or replace it all together (for Variants).
I will go over to yum mailing list and post my ideas there and see what they say, and maybe you can talk about this with them on OfficeHours?
On 01/17/2014 10:06 AM, Ljubomir Ljubojevic wrote:
On 01/17/2014 02:35 PM, Jim Perrin wrote:
On 01/17/2014 07:08 AM, Manuel Wolfshant wrote:
make it 5 or 10, please. one might want to supersede base packages with ones from private/local repos. hplip for instance was such a candidate for a long time, my printers were not supported by the stock version of hplip from EL5 and I had to compile a newer one from fedora. and at the time I was not yet relying on include/excludepkgs
This touches exactly on the point for why I don't want to include priorities by default. To make it useful for repos that provide newer software than what exists in base/updates (php, httpd, libvirt, whatever), you're automatically working against priorities. It doesn't matter if it's for a local repository or for a SIG.
So how do you intend to solve this without priorities? Especially if there is a need for some packages to be compiled with different flags?
Ideally by eliminating package duplication for the SIG/repos who work with us, but that's an ivory tower goal that's likely unattainable.
It will likely be a combination of working with SIGs to coordinate and cooperate, changing to SCL builds where possible and appropriate, and documentation to better educate the users.
SCL builds provide a nice name-spaced way to have multiple versions of the same packages installed. For example, python27, python33, multiple php and httpd versions, etc. All of which are mostly impractical with the older rpm packaging style (see php and httpd for example). This is by no means a silver bullet, but it does solve a fair chunk of the issue.
One example is for those using Virtualmin. Virtualmin provides http, postfix and few other packages compiled little differently (different flags) then RH does, in their own repository. So if both repositories are of same priority, when RH publishes newer version it is automatically an update to modified version. And that can brake things until Virtualmin releases their own version. If we "hide" RH versions with higher priority to Virtualmin packages then nothing can be broken.
These could easily be scl enabled packages. Keep in mind we have a scenario very much like this with the Clear and Neth groups, who seem to be very interested in working with us.
Same rule will apply to any package any Variant will have to compile with different flags, including CentOSPlus kernels. If someone uses CentOSPlus kernel because regular RH kernel does not have drivers for their hardware, and happens that CentOSPlus kernel is not released the same time as regular kernel (even 1-5h can make a difference), problem may arise where update to regular kernel fails and we have a unhappy user. With newbies around (I hope we attract in greater numbers), that can be troublesome.
I'm not sure this is the best example to address your point, but I do understand your meaning. This problem is not unique to CentOS, and I don't consider it to be a core distribution or board problem. The SIGs and Variants are up to their respective communities to maintain and keep current. If they falter, we'll certainly try to help but we're not going to take over the maintenance of a failing SIG.
But, as I said, for now I will be happy with "Priorities=" and "Enabled=" for every repo's config. It will make many things easier, like creating script to change those numbers per users request (you do not have to host it). Adding them to every new config (after centos-release is updated) is not desired.
Enabled, we can likely accommodate, and I'll discuss this with the others. Priorities, I'm not sold on.
With new opportunities creating CentOS Variants, I already thought of few things I would ask of yum project to incorporate, to further provide out-of-the-box experience for Variants users.
You seem to be quite passionate for the laptop/desktop users. Why not participate or create a SIG with that target audience? You could tailor this exactly as you wish with the packages (assuming legally distributable) you wish.
With all of my suggestions so far, I think something like "Skip=" for every repo would be useful for those who use their internal mirrors. If you set "Enabled=0" for some repo, using "--enablerepo=*" will enable it. Another use is to have installed but disabled one or more 3rd party repos, so you can quickly locate and install some needed package. No need for search around internet, double checking.
There is a skip_if_unavailable option, but I'm not sure that's what you're getting at. I believe fedora was also working on some sort of package library that would allow you to search directly. Perhaps this is something we could see about appropriating from them for our own uses.
Working around current "restrictions" in CentOS concerning 3rd party repository manipulation is possible, but I think CentOS-Core should be as close to Variants as possible in terms of repository syntax and use, so we do not have to build (in-house or 3rd party) additional layer over centos-release package or replace it all together (for Variants).
The formats and guidelines we're establishing will apply as equally as possible to all groups. However, I would much prefer overlay release files, similar to how there's an epel-release, elrepo-release, gluster-release, etc. This way one can 'at-a-glance' see what's in play with a simple 'yum list installed *-release'
I will go over to yum mailing list and post my ideas there and see what they say, and maybe you can talk about this with them on OfficeHours?
Certainly.
On Fri, Jan 17, 2014 at 11:06 AM, Jim Perrin jperrin@centos.org wrote:
But, as I said, for now I will be happy with "Priorities=" and "Enabled=" for every repo's config. It will make many things easier, like creating script to change those numbers per users request (you do not have to host it). Adding them to every new config (after centos-release is updated) is not desired.
Enabled, we can likely accommodate, and I'll discuss this with the others. Priorities, I'm not sold on.
I'm not convinced any existing tools can get all of the scenarios right. Priorities might work if you want a mostly-vanilla system with one or two exceptions from 3rd parties, but probably not in the context of a variant SIG plus an assortment of exceptions. There are reasons that the conflicting (or worse, incompatible) packages exist. Sometimes you'll want some of them but there won't be a general rule about which ones and there may be new relationships to enforce. For example, a clearos package is likely to require a stock package and add configuration/management tools. Then, in the (rare, but it happens) event that a stock package update changes configuration options in an incompatible way, you'll want to hold that update back until the matching changes have been added to the clearos additional package. Whenever I've posed questions about holding updates to specific revisions, the answer has always been to 'mirror the repository' in the state you want. That never seemed desirable even in the context of locally tested states, and it seems even less desirable for a proliferation of SIG repos that need known/tested states of the base layer that is managed independently.
SCLs look moderately safe, but aren't they sort of self-contained? What if you really want the alternative program or shared library to be the preferred choice?
Les Mikesell lesmikesell@gmail.com wrote:
On Fri, Jan 17, 2014 at 11:06 AM, Jim Perrin jperrin@centos.org wrote:
But, as I said, for now I will be happy with "Priorities=" and "Enabled=" for every repo's config. It will make many things easier, like creating script to change those numbers per users request (you
do
not have to host it). Adding them to every new config (after centos-release is updated) is not desired.
Enabled, we can likely accommodate, and I'll discuss this with the others. Priorities, I'm not sold on.
I'm not convinced any existing tools can get all of the scenarios right. Priorities might work if you want a mostly-vanilla system with one or two exceptions from 3rd parties, but probably not in the context of a variant SIG plus an assortment of exceptions. There are reasons that the conflicting (or worse, incompatible) packages exist. Sometimes you'll want some of them but there won't be a general rule about which ones and there may be new relationships to enforce. For example, a clearos package is likely to require a stock package and add configuration/management tools. Then, in the (rare, but it happens) event that a stock package update changes configuration options in an incompatible way, you'll want to hold that update back until the matching changes have been added to the clearos additional package. Whenever I've posed questions about holding updates to specific revisions, the answer has always been to 'mirror the repository' in the state you want.
The versionlock plugin exist for this purpose. But you are correct with the rest of your thoughts.
On 01/17/2014 09:10 PM, Manuel Wolfshant wrote:
Les Mikesell lesmikesell@gmail.com wrote:
On Fri, Jan 17, 2014 at 11:06 AM, Jim Perrin jperrin@centos.org wrote:
But, as I said, for now I will be happy with "Priorities=" and "Enabled=" for every repo's config. It will make many things easier, like creating script to change those numbers per users request (you
do
not have to host it). Adding them to every new config (after centos-release is updated) is not desired.
Enabled, we can likely accommodate, and I'll discuss this with the others. Priorities, I'm not sold on.
I'm not convinced any existing tools can get all of the scenarios right. Priorities might work if you want a mostly-vanilla system with one or two exceptions from 3rd parties, but probably not in the context of a variant SIG plus an assortment of exceptions. There are reasons that the conflicting (or worse, incompatible) packages exist. Sometimes you'll want some of them but there won't be a general rule about which ones and there may be new relationships to enforce. For example, a clearos package is likely to require a stock package and add configuration/management tools. Then, in the (rare, but it happens) event that a stock package update changes configuration options in an incompatible way, you'll want to hold that update back until the matching changes have been added to the clearos additional package. Whenever I've posed questions about holding updates to specific revisions, the answer has always been to 'mirror the repository' in the state you want.
The versionlock plugin exist for this purpose. But you are correct with the rest of your thoughts.
Problem with versionlock is that it is static, manual, per-user solution, right? What happens if in 2 months or 2 years list of those packages changes? Who will change it? user? on every single system?
Less, are you speaking about repositories with equal value or with different values? Let me explain:
"clearos" repo has Priority=1, "base" repo has Priority=2. both have package named "httpd" "yum list httpd" will only show package from "clearos" repo, package from "base" is hidden, discarded. Even if it has larger version. Only way to show it is using "--showduplicates" switch.
On Fri, Jan 17, 2014 at 2:10 PM, Manuel Wolfshant wolfy@nobugconsulting.ro wrote:
Whenever I've posed questions about holding updates to specific revisions, the answer has always been to 'mirror the repository' in the state you want.
The versionlock plugin exist for this purpose. But you are correct with the rest of your thoughts.
Versionlock only works on the local system. What I have always wanted is a way to test my local apps against 'current' system updates on one machine, then proceed to update a farm of machines to that same tested state even though newer versions may have subsequently been added to the update repositories - that is, the ability to reproduce an update.
With an assortment of application groups, OS revs, and a procedural requirement to not change all systems at once, needing a mirror of all update repositories in the state they were in when you tested each thing seems insanely inefficient. The SIG packages are likely to work like another layer with similar dependency. That is you won't really care about any specific versions, you just don't want the base-layer packages being pulled in an update until they have been tested against the SIG upper layer package that manages it. Either every SIG will have to have its own mirror of all updates that lags behind the real base/epel repos, or you need a way to tell yum which ones are OK. I don't think it is reasonable to expect everyone to work at the same speed - that is, you can't hold back base/EPEL packages until every SIG test passes.
On 01/17/2014 06:06 PM, Jim Perrin wrote:
On 01/17/2014 10:06 AM, Ljubomir Ljubojevic wrote:
On 01/17/2014 02:35 PM, Jim Perrin wrote:
On 01/17/2014 07:08 AM, Manuel Wolfshant wrote:
make it 5 or 10, please. one might want to supersede base packages with ones from private/local repos. hplip for instance was such a candidate for a long time, my printers were not supported by the stock version of hplip from EL5 and I had to compile a newer one from fedora. and at the time I was not yet relying on include/excludepkgs
This touches exactly on the point for why I don't want to include priorities by default. To make it useful for repos that provide newer software than what exists in base/updates (php, httpd, libvirt, whatever), you're automatically working against priorities. It doesn't matter if it's for a local repository or for a SIG.
So how do you intend to solve this without priorities? Especially if there is a need for some packages to be compiled with different flags?
Ideally by eliminating package duplication for the SIG/repos who work with us, but that's an ivory tower goal that's likely unattainable.
It will likely be a combination of working with SIGs to coordinate and cooperate, changing to SCL builds where possible and appropriate, and documentation to better educate the users.
SCL builds provide a nice name-spaced way to have multiple versions of the same packages installed. For example, python27, python33, multiple php and httpd versions, etc. All of which are mostly impractical with the older rpm packaging style (see php and httpd for example). This is by no means a silver bullet, but it does solve a fair chunk of the issue.
One example is for those using Virtualmin. Virtualmin provides http, postfix and few other packages compiled little differently (different flags) then RH does, in their own repository. So if both repositories are of same priority, when RH publishes newer version it is automatically an update to modified version. And that can brake things until Virtualmin releases their own version. If we "hide" RH versions with higher priority to Virtualmin packages then nothing can be broken.
These could easily be scl enabled packages. Keep in mind we have a scenario very much like this with the Clear and Neth groups, who seem to be very interested in working with us.
Same rule will apply to any package any Variant will have to compile with different flags, including CentOSPlus kernels. If someone uses CentOSPlus kernel because regular RH kernel does not have drivers for their hardware, and happens that CentOSPlus kernel is not released the same time as regular kernel (even 1-5h can make a difference), problem may arise where update to regular kernel fails and we have a unhappy user. With newbies around (I hope we attract in greater numbers), that can be troublesome.
I'm not sure this is the best example to address your point, but I do understand your meaning. This problem is not unique to CentOS, and I don't consider it to be a core distribution or board problem. The SIGs and Variants are up to their respective communities to maintain and keep current. If they falter, we'll certainly try to help but we're not going to take over the maintenance of a failing SIG.
Being proactive and being prepared are two different things. Taking over SiG maintenance would be proactive approach. Enabling Variants and 3rd party repositories to have safety net would be being accommodating while Variants and 3rd party repositories (Va3pr? :) ) would be prepared.
But, as I said, for now I will be happy with "Priorities=" and "Enabled=" for every repo's config. It will make many things easier, like creating script to change those numbers per users request (you do not have to host it). Adding them to every new config (after centos-release is updated) is not desired.
Enabled, we can likely accommodate, and I'll discuss this with the others. Priorities, I'm not sold on.
With new opportunities creating CentOS Variants, I already thought of few things I would ask of yum project to incorporate, to further provide out-of-the-box experience for Variants users.
You seem to be quite passionate for the laptop/desktop users. Why not participate or create a SIG with that target audience? You could tailor this exactly as you wish with the packages (assuming legally distributable) you wish.
I built Desktop quasi-Variant of CentOS back in 2008 I think. I created my own repository and mirrored all the mayor repositories. I have them mirrored even now. For example, I repacked static Skype 2.x for CentOS 5 to rpm, I was the only one to have it in a repository. I also recompiled 32-bit kvm L. Farkas built and used it for several years. And I rebuilt any Fedora package I needed, somewhere 50-100 rpm's overall. My focus was solely on desktop usage. I use exclusively CentOS for Desktops and Laptops for those 6 years. And I battle with 3rd party repositories including my own all those 6 years. I tried numerous solutions and variations, but every one failed due to CentOS restrictions and compilations. In those 6 years, only way to provide out-of-the box experience, but not changing centos-release/CentOS-Base.repo involved guiding newbies to login as root and start editing CentOS-Base.repo after they install my plnet-release package. Tiresome and discouraging job.
In January 2012 I created "DentOS" brand with full set of .repo files for all popular 3rd party repos and a script that will be able to switch .repo files among different sets (in subdirectories), just like I described in one of earlier mails. I would move CentOS-Base.repo to backup subdirectory and replace it with my own .repo files with incorporated "Priority=" and "Enabled=", with priorities set in such a way so Tier3 repositories including Firefox, Thunderbird, etc. packages have greater priority then base, updates, etc. That way there are no surprises on the Desktop systems I maintain for other people.
If you ask why they do not have same priority, reason is for example gstreamer version. gstreamer packages in CentOS have one version, and all gstreamer-* packages from 3rd party repositories (with legal issues) have to keep up with main gstreamer version, or they will not work. So "yum update" will fail when gstreamer package changes version since plugins must be compiled for exactly that version. And if newbie can not update it's system without problems, GUY IN SUPPORT has a problem. Responce from CentOS will be/was take it with 3rd party repo, user will get irritated and will talk about it.
From 2008 to 2010 I was on LinuxQuestions.org forums, only in RHEL/CentOS sections providing help to users (DrLove73). At the same time I was on every Linux forum in my language supporting and promoting CentOS, even as a Desktop solution, explaining 3rd party repositories. Roughly from 2010 to 2012 I was on CentOS forums providing help (DrLove73). In 2012 took on official CentOS Facebook group as main/regular admin. Before I started in CentOS group, I was one of the admins in biggest Linux group on Facebook, promoting CentOS and arguing pro/against with pro-Debian members there. Today CentOS group passed 6100 members, vast majority are newbies, some barely know what Linux is.
So I have WAST experience with newbies, their expectations and what they are looking for. And it is NOT current CentOS with problematic setting up 3rd party repositories and solving conflicts. If you ask people on CentOS forums (and other where someone supports CentOS), you will find out that newbies biggest problems revolve around 3rd party repositories providing "forbidden" packages.
And so we came to the main problem regarding Desktop SiG.
If CentOS project want's bigger adoption amongst newbies, it has to have Desktop Variant that "just work", providing easy add-on model for 3rd party repositories carrying "forbidden" packages. Otherwise newbies will use Mint/Ubuntu/Debian for Desktop and Mint/Ubuntu/Debian for server. They will not bother with learning both Debian and CentOS way. And keep in mind that Windows XP is expiring, Windows 7 is off the shelf's, and people hate Windows 8. So many newbies are already shopping for Desktop/Laptop Linux distro, and many will follow. Once first ones decide to take a Debian route, they will want Debian on servers also.
So if this crucial problem is not solved having in mind easy add-on solution without manual interventions, there is not reason what so ever in even creating Desktop SiG. CentOS will remain server distro for geeks, just like 90% of free-Linux-distro users think today.
If I were smart enough to chose Debian 9 years ago for my server in stead of ClarkConnect, I could have made easy living installing and maintaining Ubuntu/Mint systems in my area.
With all of my suggestions so far, I think something like "Skip=" for every repo would be useful for those who use their internal mirrors. If you set "Enabled=0" for some repo, using "--enablerepo=*" will enable it. Another use is to have installed but disabled one or more 3rd party repos, so you can quickly locate and install some needed package. No need for search around internet, double checking.
There is a skip_if_unavailable option, but I'm not sure that's what you're getting at. I believe fedora was also working on some sort of package library that would allow you to search directly. Perhaps this is something we could see about appropriating from them for our own uses.
Working around current "restrictions" in CentOS concerning 3rd party repository manipulation is possible, but I think CentOS-Core should be as close to Variants as possible in terms of repository syntax and use, so we do not have to build (in-house or 3rd party) additional layer over centos-release package or replace it all together (for Variants).
The formats and guidelines we're establishing will apply as equally as possible to all groups. However, I would much prefer overlay release files, similar to how there's an epel-release, elrepo-release, gluster-release, etc. This way one can 'at-a-glance' see what's in play with a simple 'yum list installed *-release'
That is my sentiment exactly. That is why I would like changes on the level of CentOS-Base.repo file, so it can be compatible with .repo files 3rd party and Variants .repo files will carry, so CentOS-Base.repo file is changed as little as possible.
I will go over to yum mailing list and post my ideas there and see what they say, and maybe you can talk about this with them on OfficeHours?
Certainly.
On 01/17/2014 06:06 PM, Ljubomir Ljubojevic wrote:
On 01/17/2014 02:35 PM, Jim Perrin wrote:
On 01/17/2014 07:08 AM, Manuel Wolfshant wrote:
make it 5 or 10, please. one might want to supersede base packages with ones from private/local repos. hplip for instance was such a candidate for a long time, my printers were not supported by the stock version of hplip from EL5 and I had to compile a newer one from fedora. and at the time I was not yet relying on include/excludepkgs
This touches exactly on the point for why I don't want to include priorities by default. To make it useful for repos that provide newer software than what exists in base/updates (php, httpd, libvirt, whatever), you're automatically working against priorities. It doesn't matter if it's for a local repository or for a SIG.
So how do you intend to solve this without priorities? Especially if there is a need for some packages to be compiled with different flags?
One example is for those using Virtualmin. Virtualmin provides http, postfix and few other packages compiled little differently (different flags) then RH does, in their own repository. So if both repositories are of same priority, when RH publishes newer version it is automatically an update to modified version. And that can brake things until Virtualmin releases their own version. If we "hide" RH versions with higher priority to Virtualmin packages then nothing can be broken.
guess what ? you can use exclude=httpd in centos-base.repo and includepkgs=httpd in the other one and you have no need for priorities
But, as I said, for now I will be happy with "Priorities=" and "Enabled=" for every repo's config. It will make many things easier, like creating script to change those numbers per users request (you do not have to host it). Adding them to every new config (after centos-release is updated) is not desired.
do you mean something like yum-config-manager ?
With new opportunities creating CentOS Variants, I already thought of few things I would ask of yum project to incorporate, to further provide out-of-the-box experience for Variants users.
With all of my suggestions so far, I think something like "Skip=" for every repo would be useful for those who use their internal mirrors. If you set "Enabled=0" for some repo, using "--enablerepo=*" will enable it. Another use is to have installed but disabled one or more 3rd party repos, so you can quickly locate and install some needed package. No need for search around internet, double checking.
So if you want to use internal mirror, you have two options:
- "Enabled=0" for CentOS-Base.repo repos, where you loose flexibility
of simple "--enablerepo=*" when you want to find something.
--enablerepo=* is anyway a bad idea because it will also enable at least all vault repos and the media repo. leaving aside that delay for accessing the vault, attempting to access the media repository will most probably lead to a nice fail
- Removing CentOS-Base.repo to backup folder or commenting out entire
file. Both options needing root's manual intervention (sudo yum authorization does not extend to editing files).
but sudo yum-config-manager does
On 13/01/14 19:05, Jim Perrin wrote:
As part of opening up the possibilities for SIGs and Variants within the CentOS ecosystem, we need to give some thought to how repositories should be structured so that different can cooperate with each other.
What we have discussed in the past, and what I'd like to propose to the -devel community, is that we adopt a page similar to the KDE framework. The way I envision this working is as follows:
- Tier 1 repositories: Adds packages only. These repos may not update
or replace anything in the core distribution. (Example: EPEL)
- Tier 2 repositories: These repositories may provide provide packages
that update or otherwise enhance packages in the base distribution, but otherwise maintain compatibility. (Example IUS
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
Each tier may choose to rely on dependencies from repositories in the tier above it, but not the other way around. For example, the tier 3 Xen4 repository could choose to rely on packages from a tier1 like EPEL, or a tier2 like IUS. However a tier1 or tier2 could not rely on a tier3. A good real-world example of this would be rpmfusion's dependence on the EPEL repository.
The other aspect of this that needs to be discussed is package duplication. For example, several of the proposed SIGs have some overlap in their packages. Would the preference here be that each SIG/Variant maintain their own version of the package, or should there be a common package set established? Both methods fit into the model described above, and both have pros/cons associated.
What does the community think about this?
One other thing to think about in advance perhaps: what happens when one of the packages involved in a SIG variant gets picked up by RHEL and becomes Tech Preview or Supported in the distro? Does that SIG repo go from tier 1 to tier 2 in the timespan it takes to read the RH Release Notes? :-)
Trevor
On 01/15/2014 07:24 PM, Trevor Hemsley wrote:
One other thing to think about in advance perhaps: what happens when one of the packages involved in a SIG variant gets picked up by RHEL and becomes Tech Preview or Supported in the distro? Does that SIG repo go from tier 1 to tier 2 in the timespan it takes to read the RH Release Notes? :-)
This is a decent point to think about. Do you have any suggestions or recommendations about this?
On 17/01/14 12:19, Jim Perrin wrote:
On 01/15/2014 07:24 PM, Trevor Hemsley wrote:
One other thing to think about in advance perhaps: what happens when one of the packages involved in a SIG variant gets picked up by RHEL and becomes Tech Preview or Supported in the distro? Does that SIG repo go from tier 1 to tier 2 in the timespan it takes to read the RH Release Notes? :-)
This is a decent point to think about. Do you have any suggestions or recommendations about this?
Is there anyone from EPEL listening here? I think they've already discussed this sort of thing to death in the last few years so we could probably learn from them. How do they handle this sort of thing? If we don't have anyone from EPEL around... why not?
T
Hi,
On 01/13/2014 07:05 PM, Jim Perrin wrote:
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
at fosdem we've tried to flesh out how this might work with some real world examples, and unfortunately this model will not work.
The actual structure and layers needs to be much simpler with dependancy allowed as you mentioned ( ie. tier 1 can only rely on itself and OS, tier2 can rely on itseld + one or more tier1's + os, tier3 can rely on itself + one or more tier2's, one or more tier1's and os ).
Look at it this way, putting glusterfs, ceph ( bcause they need a modified build for qemu ) will mean that the only way openstack is able to consume it will be if openstack is in the same repo ( and they are all collected at a teir3 ), then cascade down and the entire dep tree needs to move back upto a teir3, at which point its all just a flat OS tree and tier3 repos.
So with that in mind, i think repo's should be allowed to exist at any tier, with a repo being able to decide for itself if it wants to include content that replaces functionality + adds, or just adds. ie. adopt the Extras/ and Plus/ seperation ( with Plus/ automatically -required- to include Extras/ ) at each tier.
nutshell: libvirt, qemu, xen4centos, ceph and friends are all going to be -required- to be tier1's and be allowed to replace ( namespace overload ?) components, so that every other component above in other teri1's or 2's or 3's are able to consume it.
Also, I think we should explore the idea ( or atleast have the option ) of shipping an app lxc container instead of a SCL; Overall, we just need to find a better way to (a) build + deliver them (b) build relationships and (c) consume common metadata from the instance / container host. We might go down the route of using the container delivery mechanism within the VOIP SIG ( proposal for that coming soon... )
Anyone fancy writing a yum plugin to ship and manage conainers ? Only partially joking...
- KB
Hi,
On 01/13/2014 07:05 PM, Jim Perrin wrote:
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
at fosdem we've tried to flesh out how this might work with some real world examples, and unfortunately this model might hit a few issues.
The actual structure and layers needs to be much simpler with dependancy allowed as you mentioned ( ie. tier 1 can only rely on itself and OS, tier2 can rely on itseld + one or more tier1's + os, tier3 can rely on itself + one or more tier2's, one or more tier1's and os ).
Look at it this way, putting glusterfs, ceph in tier3 ( bcause they need a modified build for qemu ) will mean that the only way openstack is able to consume it will be if openstack is in the same repo ( and they are all collected at a teir3 ), then cascade down and the entire dep tree needs to move back upto a teir3, at which point its all just a flat OS tree and tier3 repos.
So with that in mind, i think repo's should be allowed to exist at any tier, with a repo being able to decide for itself if it wants to include content that replaces functionality + adds, or just adds. ie. adopt the Extras/ and Plus/ seperation ( with Plus/ automatically -required- to include Extras/ ) at each tier.
nutshell: libvirt, qemu, xen4centos, ceph and friends are all going to be -required- to be tier1's and be allowed to replace ( namespace overload ?) components, so that every other component above in other teri1's or 2's or 3's are able to consume it.
- KB
On 02/02/2014 05:06 PM, Karanbir Singh wrote:
Hi,
On 01/13/2014 07:05 PM, Jim Perrin wrote:
- Tier 3 repositories: These repositories contain packages that
conflict with software inside the core distribution. This would be Xen4CentOS for example, which replaces the kernel, and adds in xen functionality.
at fosdem we've tried to flesh out how this might work with some real world examples, and unfortunately this model might hit a few issues.
The actual structure and layers needs to be much simpler with dependancy allowed as you mentioned ( ie. tier 1 can only rely on itself and OS, tier2 can rely on itseld + one or more tier1's + os, tier3 can rely on itself + one or more tier2's, one or more tier1's and os ).
Look at it this way, putting glusterfs, ceph in tier3 ( bcause they need a modified build for qemu ) will mean that the only way openstack is able to consume it will be if openstack is in the same repo ( and they are all collected at a teir3 ), then cascade down and the entire dep tree needs to move back upto a teir3, at which point its all just a flat OS tree and tier3 repos.
But openshift would be able to exist as a tier 1 or tier2, since it's simply adding packages without updating core distro rpms. So it depends a bit on what the user wants to do.
So with that in mind, i think repo's should be allowed to exist at any tier, with a repo being able to decide for itself if it wants to include content that replaces functionality + adds, or just adds. ie. adopt the Extras/ and Plus/ seperation ( with Plus/ automatically -required- to include Extras/ ) at each tier.
So the issue seems to be more with the dependency tiering requirement rather than the categorization of the tiers themselves.
nutshell: libvirt, qemu, xen4centos, ceph and friends are all going to be -required- to be tier1's and be allowed to replace ( namespace overload ?) components, so that every other component above in other teri1's or 2's or 3's are able to consume it.
They can't be tier1's if they're updating or conflicting with existing packages, but we can certainly relax the dependency requirements to resolve this. That should resolve this issue, right?
On 02/04/2014 04:02 PM, Jim Perrin wrote:
Look at it this way, putting glusterfs, ceph in tier3 ( bcause they need a modified build for qemu ) will mean that the only way openstack is able to consume it will be if openstack is in the same repo ( and they are all collected at a teir3 ), then cascade down and the entire dep tree needs to move back upto a teir3, at which point its all just a flat OS tree and tier3 repos.
But openshift would be able to exist as a tier 1 or tier2, since it's simply adding packages without updating core distro rpms. So it depends a bit on what the user wants to do.
maybe it needs puppet, which needs ruby which comes from a repo that replaces the base os ruby. So while it can exist as a tier1 in this model, its dep tree is going to cascade upto tier3, so might as well make it a tier3 repo.
So the issue seems to be more with the dependency tiering requirement rather than the categorization of the tiers themselves.
yes, we need stuff to go into the lower tiers that have the ability to change packages, so that other repos can then rely on them ( or require them optionally, eg: opennebula-node-xen )
nutshell: libvirt, qemu, xen4centos, ceph and friends are all going to be -required- to be tier1's and be allowed to replace ( namespace overload ?) components, so that every other component above in other teri1's or 2's or 3's are able to consume it.
They can't be tier1's if they're updating or conflicting with existing packages, but we can certainly relax the dependency requirements to resolve this. That should resolve this issue, right?
or we find a way to have repos contain content that overwrites others, and use namespace overload or something other mechanism to allow it at any tier.
On Tue, Feb 4, 2014 at 11:00 AM, Karanbir Singh mail-lists@karan.org wrote:
They can't be tier1's if they're updating or conflicting with existing packages, but we can certainly relax the dependency requirements to resolve this. That should resolve this issue, right?
or we find a way to have repos contain content that overwrites others, and use namespace overload or something other mechanism to allow it at any tier.
Sigh... did the RPM developers forget that there was always a good reason that PATH and LD_LIBRARY_PATH (a) hold multiple values _and_ (b) are set per-process not as globals for the machine?
On 02/04/2014 06:00 PM, Karanbir Singh wrote:
On 02/04/2014 04:02 PM, Jim Perrin wrote:
Look at it this way, putting glusterfs, ceph in tier3 ( bcause they need a modified build for qemu ) will mean that the only way openstack is able to consume it will be if openstack is in the same repo ( and they are all collected at a teir3 ), then cascade down and the entire dep tree needs to move back upto a teir3, at which point its all just a flat OS tree and tier3 repos.
But openshift would be able to exist as a tier 1 or tier2, since it's simply adding packages without updating core distro rpms. So it depends a bit on what the user wants to do.
maybe it needs puppet, which needs ruby which comes from a repo that replaces the base os ruby. So while it can exist as a tier1 in this model, its dep tree is going to cascade upto tier3, so might as well make it a tier3 repo.
So the issue seems to be more with the dependency tiering requirement rather than the categorization of the tiers themselves.
yes, we need stuff to go into the lower tiers that have the ability to change packages, so that other repos can then rely on them ( or require them optionally, eg: opennebula-node-xen )
nutshell: libvirt, qemu, xen4centos, ceph and friends are all going to be -required- to be tier1's and be allowed to replace ( namespace overload ?) components, so that every other component above in other teri1's or 2's or 3's are able to consume it.
They can't be tier1's if they're updating or conflicting with existing packages, but we can certainly relax the dependency requirements to resolve this. That should resolve this issue, right?
or we find a way to have repos contain content that overwrites others, and use namespace overload or something other mechanism to allow it at any tier.
Yum priority= fits so nicely here. Just saying.
On 02/05/2014 12:00 AM, Ljubomir Ljubojevic wrote:
or we find a way to have repos contain content that overwrites others, and use namespace overload or something other mechanism to allow it at any tier.
Yum priority= fits so nicely here. Just saying.
no it does'nt. priority has the same problem, its static in layers.
btw, have you looked at 'cost' instead ? its a far more elegant and better implemented mechanism that covers most ( but maybe not all ) of the same problem domain as yum priorities.
On 02/05/2014 03:28 AM, Karanbir Singh wrote:
On 02/05/2014 12:00 AM, Ljubomir Ljubojevic wrote:
or we find a way to have repos contain content that overwrites others, and use namespace overload or something other mechanism to allow it at any tier.
Yum priority= fits so nicely here. Just saying.
no it does'nt. priority has the same problem, its static in layers.
btw, have you looked at 'cost' instead ? its a far more elegant and better implemented mechanism that covers most ( but maybe not all ) of the same problem domain as yum priorities.
No, no one mention it so far, but I will check it out as soon as I can.
-
Dave Neary -
Jim Perrin -
Johnny Hughes -
Karanbir Singh -
Les Mikesell -
Ljubomir Ljubojevic -
Manuel Wolfshant -
Ned Slider -
Sam Kottler -
Trevor Hemsley -
Yury V. Zaytsev