Re: [CentOS-devel] A Big Idea for a New Decade [was: Minutes for CentOS Board of Directors 2019-12-18 Meeting]
On Mon, Jan 6, 2020 at 4:13 PM Matthew Miller mattdm@mattdm.org wrote:
On Fri, Jan 03, 2020 at 03:13:53PM -0500, Karsten Wade wrote:
As a centralizing effort, the Board agreed to revisiting the goals document created five years ago, and to undergo an effort to refresh those goals in the light of the project’s evolution. The Board will be inviting various stakeholders into these discussions as we undergo a public revision of the goals at the start of 2020.
Wow, five years goes fast! I'd like to raise an idea for discussion and thought, and possible further discussion between the Fedora Council and the CentOS board. [I'm posting this in both groups.]
Around the same time the CentOS goals were being created, Fedora was working on plans for "Fedora.next", which included among other things Fedora Server. I won't rehash all that here, but in short: while that succeeded at some of its goals, it never really had the user adoption we'd hoped for. Of course, some of that is simply a matter of lifecycle: while a fast-moving server OS is useful in some very real cases, it's decidedly niche.
When Fedora launched that plan, CentOS wasn't really in the "family", and after, we just kind of existed in parallel, with some very loose collaboration in areas like EPEL.
Of course, things are different now. Therefore, I propose that in this next decade, we make that collaboration stronger. Together, we have a very nice answer to the fundamental problem of unified operating systems. That is, some parts move too fast and other parts move too slowly. Between Fedora and CentOS, we have answers to both! And, with the in-progress dist-git merge, we even have both _in the same place_.
Specifically, I'd like to suggest three things.
First, let's replace Fedora Server as a user-focused artifact with CentOS Stream. There's still a need for a Fedora Server as an upstream, but most users of Fedora as a server are making custom builds (or using the basic Cloud image), not using Fedora Server per se. So, I think we should stop marketing that to users, and steer people with server use cases outside of the fast-moving niche to CentOS Stream. The current story of CentOS and Fedora is pretty confusing to users, and Stream only makes it more so. Let's simplify things!
Second, where there's overlap, let's look at merging Fedora and CentOS SIGs into joint bodies. We have groups like the cloud image SIGs which are basically doing the exact same thing in two different ways. Same thing with containers. We also have huge overlaps in documentation and community user support and all sorts of non-technical project areas. I'm not saying these all need to be smooshed together, but where it makes sense let's work together. This, too, brings simplification to the "what's with CentOS vs. Fedora" story: rather than making contributors pick, we can welcome all with open arms.
And finally, let's really take advantage of a merged dist-git repository, and the capabilities that Modularity gives us. SIGs building things for CentOS should be able to directly reference Fedora branches where faster-moving software is useful — and people building things in Fedora should be able to pull in branches from Stream where a slower-moving version would make sense. And we could even make additional shared branches where necessary, using modularity to make optional streams which could be used in CentOS SIGs, EPEL, or a Fedora OS.
What do you think?
First, sorry that this email probably doesn't thread will with either one or the other target mailing lists... I had to pick one to thread with, and other slightly loses. The downside of the initial email not being sent to both MLs at once. :(
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature. There are several issues with doing this as it currently stands:
1. CentOS Koji is pretty closed to a couple of folks in CentOS and Red Hat. This also includes blocking the download of artifacts from Koji. For reasons I don't understand, CentOS is still gated by the RHEL QE folks, and thus RPMs built in Koji cannot be downloaded until they are finally published. This ruins any testing/development before releasing to users. The CBS is a worthless substitute since it's purely for addons. I've been around long enough to remember when Fedora was like this, and back then, it wasn't really considered "okay" for Fedora to be like that (the intent was always there to fix it, and we eventually did!). Today, however, it is considered "okay", and that's a problem.
2. The CentOS community is simply not mature enough to take on the role Fedora does in any meaningful capacity. We already know that this is a problem even with Fedora EPEL, and I strongly believe the issue would be massively magnified if those transitioned to the CentOS Project. The CentOS community is largely a user community that takes and does not give back much, if at all. I've experienced variations of this issue when people (even from Red Hat!) have done finger-pointing to *me* with my packages in EPEL without any acknowledgement that they could have helped. It's depressing that even some Red Hatters consider EPEL not worth contributing to, but still worth being upset about.
3. Today, the CentOS Project is (sadly) a community project in a very superficial sense. Aside from the Virt SIG's Xen subgroup, all SIGs are basically Red Hat product teams, with varying degrees of allowances for community contributions. The community processes within CentOS are mostly nonfunctional because they've had no reason to develop it. There has been very little community development for CentOS. Even with Fermilab and CERN shutting down Scientific Linux, I haven't seen anything from those people coming "into the fold". There's been no outreach or collaboration with other CentOS based derivatives, commercial or otherwise. The private discussions I've had on the matter seem to indicate that there's still ambivalence, if not outright antipathy and hostility toward the broader ecosystem of CentOS derivatives.
What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities for people to iterate on server-oriented changes before they go into rhel-rawhide and push into CentOS Stream. You also essentially kneecap any motivation for other things related to server environments to iterate faster (such as language stacks that are heavily used for web service software) because you've eliminated the major ability for that to ship to users and contributors. It also further accelerates a trend that I think we need to reverse where people consider Fedora unacceptable for server roles. If anything, Fedora is a lot better at being used for servers then it was five years ago. I've personally *stopped* using CentOS for servers because Fedora has gotten so good at it. Upgrades are a breeze and stuff generally works. When it doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
My personal feeling is there have been very high barriers to contributing to CentOS.
1. Almost every bug report and fix needs to go to/come from RHEL and Fedora. Very few bug reports are fixed in CentOS then submitted upstream to RHEL and Fedora. 2. The Contributing Documentation is sparse and very confusing. It needs to be re-written and updated, and the Wiki needs to either be upgraded to something more user friendly, or replaced with a proper knowledgebase application. 3. Meetings need to be held in a more accessible location. Freenode IRC may have worked well in the past, but many companies are locking down their outbound rules and blocking services like IRC. I used to use a SSH tunnel to access Freenode, but even there, the "scheduled" meetings for the SIGs never seemed to happen. Even looking at the minutes from many, you see the host is the only person who joins. Let's migrate to a Slack/Discord server for meetings, and figure out a way to better publish the meetings. 4. We should be encouraging the Fedora members to join the CentOS SIGs and vice versa. If the two teams see what each other are working on, they can eliminate overlap, and I have a feeling they would quickly work on merging their efforts into one SIG. 5. The process to contribute should be made a lot easier. As an example, I have a patch I want to submit to the Jetty project. Their process is clearly defined, and the only holdup I am having right now is getting the internal approval from my employer to contribute (the normal review processes). With CentOS, I'm not sure what I can contribute to, and what I need to contribute to RHEL or Fedora. I also find that bug reports to all three go un-touched for years at a time. 6. For the love of all that is pink and fluffy, we need to update the versions of third party packages we ship. If RHEL won't, CentOS should. For instance, we still ship Jetty 9.2, which is EOL and not receiving security updates. 9.3 is also EOL. 9.4 is quite stable at this point (as they are about to go beta on 10.0), so we should be shipping 9.4. PostgreSQL is another example. We ship 9.2. 12.1 is the current release, and the appliance I work on is shipping 10.11 as it's stable version. 9.2 is so far out of EOL... it has been superseded by 9.3, 9.4, 9.5, 9.6, 10, 11 and 12. If you want an example of what we should be shipping as the stable versions, cPanel is a very good example. They run on CentOS, but sooooooo many packages have to be built by them to make their servers secure.
Gregory Young
-----Original Message----- From: CentOS-devel centos-devel-bounces@centos.org On Behalf Of Neal Gompa Sent: Tuesday, January 7, 2020 7:10 AM To: Discussions with the Fedora Council and community council-discuss@lists.fedoraproject.org Cc: The CentOS developers mailing list. CentOS-devel@centos.org Subject: Re: [CentOS-devel] A Big Idea for a New Decade [was: Minutes for CentOS Board of Directors 2019-12-18 Meeting]
On Mon, Jan 6, 2020 at 4:13 PM Matthew Miller mattdm@mattdm.org wrote:
On Fri, Jan 03, 2020 at 03:13:53PM -0500, Karsten Wade wrote:
As a centralizing effort, the Board agreed to revisiting the goals document created five years ago, and to undergo an effort to refresh those goals in the light of the project’s evolution. The Board will be inviting various stakeholders into these discussions as we undergo a public revision of the goals at the start of 2020.
Wow, five years goes fast! I'd like to raise an idea for discussion and thought, and possible further discussion between the Fedora Council and the CentOS board. [I'm posting this in both groups.]
Around the same time the CentOS goals were being created, Fedora was working on plans for "Fedora.next", which included among other things Fedora Server. I won't rehash all that here, but in short: while that succeeded at some of its goals, it never really had the user adoption we'd hoped for. Of course, some of that is simply a matter of lifecycle: while a fast-moving server OS is useful in some very real cases, it's decidedly niche.
When Fedora launched that plan, CentOS wasn't really in the "family", and after, we just kind of existed in parallel, with some very loose collaboration in areas like EPEL.
Of course, things are different now. Therefore, I propose that in this next decade, we make that collaboration stronger. Together, we have a very nice answer to the fundamental problem of unified operating systems. That is, some parts move too fast and other parts move too slowly. Between Fedora and CentOS, we have answers to both! And, with the in-progress dist-git merge, we even have both _in the same place_.
Specifically, I'd like to suggest three things.
First, let's replace Fedora Server as a user-focused artifact with CentOS Stream. There's still a need for a Fedora Server as an upstream, but most users of Fedora as a server are making custom builds (or using the basic Cloud image), not using Fedora Server per se. So, I think we should stop marketing that to users, and steer people with server use cases outside of the fast-moving niche to CentOS Stream. The current story of CentOS and Fedora is pretty confusing to users, and Stream only makes it more so. Let's simplify things!
Second, where there's overlap, let's look at merging Fedora and CentOS SIGs into joint bodies. We have groups like the cloud image SIGs which are basically doing the exact same thing in two different ways. Same thing with containers. We also have huge overlaps in documentation and community user support and all sorts of non-technical project areas. I'm not saying these all need to be smooshed together, but where it makes sense let's work together. This, too, brings simplification to the "what's with CentOS vs. Fedora" story: rather than making contributors pick, we can welcome all with open arms.
And finally, let's really take advantage of a merged dist-git repository, and the capabilities that Modularity gives us. SIGs building things for CentOS should be able to directly reference Fedora branches where faster-moving software is useful — and people building things in Fedora should be able to pull in branches from Stream where a slower-moving version would make sense. And we could even make additional shared branches where necessary, using modularity to make optional streams which could be used in CentOS SIGs, EPEL, or a Fedora OS.
What do you think?
First, sorry that this email probably doesn't thread will with either one or the other target mailing lists... I had to pick one to thread with, and other slightly loses. The downside of the initial email not being sent to both MLs at once. :(
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature. There are several issues with doing this as it currently stands:
1. CentOS Koji is pretty closed to a couple of folks in CentOS and Red Hat. This also includes blocking the download of artifacts from Koji. For reasons I don't understand, CentOS is still gated by the RHEL QE folks, and thus RPMs built in Koji cannot be downloaded until they are finally published. This ruins any testing/development before releasing to users. The CBS is a worthless substitute since it's purely for addons. I've been around long enough to remember when Fedora was like this, and back then, it wasn't really considered "okay" for Fedora to be like that (the intent was always there to fix it, and we eventually did!). Today, however, it is considered "okay", and that's a problem.
2. The CentOS community is simply not mature enough to take on the role Fedora does in any meaningful capacity. We already know that this is a problem even with Fedora EPEL, and I strongly believe the issue would be massively magnified if those transitioned to the CentOS Project. The CentOS community is largely a user community that takes and does not give back much, if at all. I've experienced variations of this issue when people (even from Red Hat!) have done finger-pointing to *me* with my packages in EPEL without any acknowledgement that they could have helped. It's depressing that even some Red Hatters consider EPEL not worth contributing to, but still worth being upset about.
3. Today, the CentOS Project is (sadly) a community project in a very superficial sense. Aside from the Virt SIG's Xen subgroup, all SIGs are basically Red Hat product teams, with varying degrees of allowances for community contributions. The community processes within CentOS are mostly nonfunctional because they've had no reason to develop it. There has been very little community development for CentOS. Even with Fermilab and CERN shutting down Scientific Linux, I haven't seen anything from those people coming "into the fold". There's been no outreach or collaboration with other CentOS based derivatives, commercial or otherwise. The private discussions I've had on the matter seem to indicate that there's still ambivalence, if not outright antipathy and hostility toward the broader ecosystem of CentOS derivatives.
What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities for people to iterate on server-oriented changes before they go into rhel-rawhide and push into CentOS Stream. You also essentially kneecap any motivation for other things related to server environments to iterate faster (such as language stacks that are heavily used for web service software) because you've eliminated the major ability for that to ship to users and contributors. It also further accelerates a trend that I think we need to reverse where people consider Fedora unacceptable for server roles. If anything, Fedora is a lot better at being used for servers then it was five years ago. I've personally *stopped* using CentOS for servers because Fedora has gotten so good at it. Upgrades are a breeze and stuff generally works. When it doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 1/7/20 10:48 AM, Young, Gregory wrote:
My personal feeling is there have been very high barriers to contributing to CentOS.
- Almost every bug report and fix needs to go to/come from RHEL and Fedora. Very few bug reports are fixed in CentOS then submitted upstream to RHEL and Fedora.
- The Contributing Documentation is sparse and very confusing. It needs to be re-written and updated, and the Wiki needs to either be upgraded to something more user friendly, or replaced with a proper knowledgebase application.
We eagerly welcome people to contribute changes of any kind to the contributing documentation - the document itself - https://wiki.centos.org/Contribute - describes how to get edit rights to the Wiki. There is, indeed, a lot of confusing and/or outdated content in the wiki. We encourage folks to join the docs mailing list - https://lists.centos.org/mailman/listinfo/centos-docs - to discuss proposed changes/updates.
If you haven't looked at the wiki in the past month, you may have missed that our awesome infra folks have, in fact, updated it. It's both more attractive, and *much* faster, than it was last year.
- Meetings need to be held in a more accessible location. Freenode IRC may have worked well in the past, but many companies are locking down their outbound rules and blocking services like IRC. I used to use a SSH tunnel to access Freenode, but even there, the "scheduled" meetings for the SIGs never seemed to happen. Even looking at the minutes from many, you see the host is the only person who joins. Let's migrate to a Slack/Discord server for meetings, and figure out a way to better publish the meetings.
Even ignoring the innate controversy of the "what chat platform" discussion, there's the question of tooling. We currently have tools to capture meeting minutes for IRC, but not for other platforms. We recently moved meetings from #centos-devel to #centos-meeting to remove some of the noise factor on the main channel.
But, as you note, the biggest concern, to me, is that these meetings don't even always happen, leaving uncertainty of what's going on, and how one can join the effort. This is something that is on my list to work on in 2020, and is part of the board's renewed focus on transparency.
- We should be encouraging the Fedora members to join the CentOS SIGs and vice versa. If the two teams see what each other are working on, they can eliminate overlap, and I have a feeling they would quickly work on merging their efforts into one SIG.
Yes, please. But we also need to be careful not to repeat the mistakes of the past. The Cloud SIG in CentOS siphoned off a number of the active members of the Fedora Cloud SIG, effectively killing any Openstack work in Fedora, and we want to avoid doing anything like that going forward. We want to merge/combine effort, not force people to pick one project or the other.
Matthew mentions else-thread that we need to figure out how to get our SIGs to work together, in ways that don't just mean that everyone has to do twice as much work. I'm looking for some space/time at FOSDEM where we can kick off this discussion and see what needs to happen.
- The process to contribute should be made a lot easier. As an example, I have a patch I want to submit to the Jetty project. Their process is clearly defined, and the only holdup I am having right now is getting the internal approval from my employer to contribute (the normal review processes). With CentOS, I'm not sure what I can contribute to, and what I need to contribute to RHEL or Fedora. I also find that bug reports to all three go un-touched for years at a time.
This is definitely something that we're trying to address with CentOS Stream, and we appreciate everyone's patience as we move towards that goal. The RHEL process is enormous, in terms of people, and steps, and all of the mechanisms and tools built around it, and CentOS's heritage is as a rebuild of that, not as a place for innovation.
On Tue, 7 Jan 2020 at 10:48, Young, Gregory gregory.young@solarwinds.com wrote:
My personal feeling is there have been very high barriers to contributing to CentOS.
- For the love of all that is pink and fluffy, we need to update the versions of third party packages we ship. If RHEL won't, CentOS should. For instance, we still ship Jetty 9.2, which is EOL and not receiving security updates. 9.3 is also EOL. 9.4 is quite stable at this point (as they are about to go beta on 10.0), so we should be shipping 9.4. PostgreSQL is another example. We ship 9.2. 12.1 is the current release, and the appliance I work on is shipping 10.11 as it's stable version. 9.2 is so far out of EOL... it has been superseded by 9.3, 9.4, 9.5, 9.6, 10, 11 and 12. If you want an example of what we should be shipping as the stable versions, cPanel is a very good example. They run on CentOS, but sooooooo many packages have to be built by them to make their servers secure.
The true purpose of an enterprise software is to make sure that a site can run crufty old software which depends on some version of a library no longer supported by upstream beyond simple bug fixes. [I can say from experience that updating jetty will break all kinds of commercial payroll apps which expect X version]. In the end, enterprise software upgrades at the rate of whatever the majority of accounting systems need to make payroll and tax filings happen. And then you find out that even if you move to the cloud, you still have to keep the old records active for 10+ years for all kinds of reasons.
Enterprise Operating Systems are the semi trucks of distributions. They are slow, lumbering, noisy but carry a lot of stuff from point A to point B. Other operating systems are like smaller trucks or cars or even gulf-carts.. they each have their place and reasons for moving at the speeds they are for. The issue is to not try and expect a Bugatti to haul a 40 ton trailer or a Peterbilt to outlap a Lamborghini [you can do both with a lot of work, but there is little practicality of said work.] And I would not want either one of them on a golf course unless I was expecting to returf the entire area.
-- Stephen J Smoogen.
On Tue, Jan 07, 2020 at 11:22:59AM -0500, Stephen John Smoogen wrote:
- For the love of all that is pink and fluffy, we need to update the
versions of third party packages we ship. If RHEL won't, CentOS should. For instance, we still ship Jetty 9.2, which is EOL and not receiving security updates. 9.3 is also EOL. 9.4 is quite stable at this point (as they are about to go beta on 10.0), so we should be shipping 9.4.
[...]
The true purpose of an enterprise software is to make sure that a site can run crufty old software which depends on some version of a library no longer supported by upstream beyond simple bug fixes. [I can say from experience that updating jetty will break all kinds of commercial payroll apps which expect X version]. In the end, enterprise software
What about providing an updated Jetty as an optional module in EPEL? I see we have 9.4.24 in Fedora. This seems like a pretty good example of what I'm saying about fast and slow streams -- we actually _have_ this in our ecosystem already, just not in a consumable way. If it were in EPEL, RHEL or CentOS users who want to strap a nitro-burning sidecar on their semi truck for their use case could do so.
On Tue, 7 Jan 2020 at 11:46, Matthew Miller mattdm@mattdm.org wrote:
On Tue, Jan 07, 2020 at 11:22:59AM -0500, Stephen John Smoogen wrote:
- For the love of all that is pink and fluffy, we need to update the
versions of third party packages we ship. If RHEL won't, CentOS should. For instance, we still ship Jetty 9.2, which is EOL and not receiving security updates. 9.3 is also EOL. 9.4 is quite stable at this point (as they are about to go beta on 10.0), so we should be shipping 9.4.
[...]
The true purpose of an enterprise software is to make sure that a site can run crufty old software which depends on some version of a library no longer supported by upstream beyond simple bug fixes. [I can say from experience that updating jetty will break all kinds of commercial payroll apps which expect X version]. In the end, enterprise software
What about providing an updated Jetty as an optional module in EPEL? I see we have 9.4.24 in Fedora. This seems like a pretty good example of what I'm saying about fast and slow streams -- we actually _have_ this in our ecosystem already, just not in a consumable way. If it were in EPEL, RHEL or CentOS users who want to strap a nitro-burning sidecar on their semi truck for their use case could do so.
From my experience with jetty in the past on enterprise systems you
find out that you need to run 2-3 versions on the same system at the same time. All of them want to be called jetty and they all want a bunch of environment variables set up that are 'unique' to that version. This is where SCL's are a better fit.
For Gregory's problem, modularity does look like a better fix for 8 BUT the person writing the module needs to know a bunch of things: 1. How to make a module. [This seems to be about ~20 people.] 2. What software depends on jetty which will need to be modularized also. This isn't software dep but API deps. [That will be the packagers of java applications and such.] 3. Know how to avoid setting up a libgit2 and similar problems that Fedora has run into. We do not want someone to install EPEL-8 modularity and find out that we smoked their production system on the first yum update or some similar thing because a module replaced something it needed but was required for some other thing (OR vice versa). From the conversations in FESCO about this.. htis is still reliant on human knowledge and detection versus a test framework.
However it doesn't fix it for 7 which is probably where the lion share of people using Gregory's software that needs a newer jetty are.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Tue, Jan 07, 2020 at 12:41:14PM -0500, Stephen John Smoogen wrote:
For Gregory's problem, modularity does look like a better fix for 8 BUT the person writing the module needs to know a bunch of things:
- How to make a module. [This seems to be about ~20 people.]
Probably true, but the docs are actually quite good. https://docs.fedoraproject.org/en-US/modularity/making-modules/adding-new-mo...
- What software depends on jetty which will need to be modularized
also. This isn't software dep but API deps. [That will be the packagers of java applications and such.]
Yes. But that can start small: first the jetty module, then consumers of it.
- Know how to avoid setting up a libgit2 and similar problems that
Fedora has run into. We do not want someone to install EPEL-8 modularity and find out that we smoked their production system on the first yum update or some similar thing because a module replaced something it needed but was required for some other thing (OR vice versa). From the conversations in FESCO about this.. htis is still reliant on human knowledge and detection versus a test framework.
As I understand it, this is entirely a problem with making something a *default* module, which wouldn't be the case here.
What about providing an updated Jetty as an optional module in EPEL? I see we have 9.4.24 in Fedora. This seems like a pretty good example of what I'm saying about fast and slow streams -- we actually _have_ this in our ecosystem already, just not in a consumable way. If it were in EPEL, RHEL or CentOS users who want to strap a nitro-burning sidecar on their semi truck for their use case could do so.
[Gregory Young] Providing them to RHEL/CentOS through EPEL is also a good idea, but I would suggest we include the older versions in EPEL as well and mirror the current stable version from EPEL in the CentOS base. This way, people can enable EPEL if they want the older stream (like Jetty 9.2). To be honest, it would be nice to see many of the Fedora packages in EPEL so they can be consumed by EL7 and 8. Even if the newer, less stable versions were available, but not as the default version. As an example, cPanel is currently building an EL7 version of OpenSSL 1.1.1 so they can enable TLS 1.3 on EL7. It would be nice if EL7 had access to the Fedora build of OpenSSL 1.1.1 through EPEL, even of 1.0.2k is the default at this time. I'm all for the defaults being stable versions, but they also need to be secure versions that are at the very least in the security fix supported stage of sunset. Once a third-party package hits EOL and no longer gets security updates, we should be moving to the next or latest stable, supported version as the default, and move the EOL version to EPEL.
As the person who also reviews all security reports that come in for our appliance, I know that way too many vulnerability scanners work simply on version matching, and don't actually test for the vulnerability they are flagging. At this point my Support team has a pre-approved response on any OpenSSH vulnerabilities that come in that are version matching, directing our partners to the Red Hat Backporting policy. The issue is the number of cases Support is seeing is climbing as the world becomes more security aware, and nobody seems to be pushing back against these false positives (the vendor isn't going to want to fix them, it makes their report look like it is really good, look at all these vulnerabilities we found!!!). Now I know EL7 isn't going to move to the latest and greatest OpenSSH version, we need to be on a stable version, but at some point, there needs to be the discussion about bumping to the next stable version during a point release. With the lifecycle of the OS, and how fast the security field is changing, I think it warrants jumping to a newer stable release at least once during the lifetime of the OS, and in some cases, even more frequently.
On Tue, Jan 07, 2020 at 06:54:36PM +0000, Young, Gregory wrote:
Providing them to RHEL/CentOS through EPEL is also a good idea, but I would suggest we include the older versions in EPEL as well and mirror the current stable version from EPEL in the CentOS base. This way, people
The problem is: modularity gives us a shared technology between Fedora and RHEL 8 (and CentOS 8) where we can provide alternate versions in a clean way. In EPEL 7 and before, we didn't have that, so the policy forbids updating RHEL packages.
You could do it with a Copr repo, though
On Tue, Jan 07, 2020 at 06:54:36PM +0000, Young, Gregory wrote:
Providing them to RHEL/CentOS through EPEL is also a good idea, but I would suggest we include the older versions in EPEL as well and mirror the current stable version from EPEL in the CentOS base.
Please don't. It'd make CentOS provide something different than RHEL and then people wouldn't be able to use CentOS as a reliable testing/building environment for products built on RHEL.
I'm fine if you want to include them in a SCL or a SIG repo, just don't put it in base unless you also convince RH to do it too.
On Tue, Jan 07, 2020 at 02:00:53PM -0500, Jonathan Billings wrote:
Please don't. It'd make CentOS provide something different than RHEL and then people wouldn't be able to use CentOS as a reliable testing/building environment for products built on RHEL.
I'm fine if you want to include them in a SCL or a SIG repo, just don't put it in base unless you also convince RH to do it too.
Right, to be clear, I'm absolutely not suggesting having different _default_ packages.
On Tue, 7 Jan 2020 at 10:48, Young, Gregory gregory.young@solarwinds.com wrote:
My personal feeling is there have been very high barriers to contributing to CentOS.
- For the love of all that is pink and fluffy, we need to update the versions of third party packages we ship. If RHEL won't, CentOS should. For instance, we still ship Jetty 9.2, which is EOL and not receiving security updates. 9.3 is also EOL. 9.4 is quite stable at this point (as they are about to go beta on 10.0), so we should be shipping 9.4. PostgreSQL is another example. We ship 9.2. 12.1 is the current release, and the appliance I work on is shipping 10.11 as it's stable version. 9.2 is so far out of EOL... it has been superseded by 9.3, 9.4, 9.5, 9.6, 10, 11 and 12. If you want an example of what we should be shipping as the stable versions, cPanel is a very good example. They run on CentOS, but sooooooo many packages have to be built by them to make their servers secure.
The true purpose of an enterprise software is to make sure that a site can run crufty old software which depends on some version of a library no longer supported by upstream beyond simple bug fixes. [I can say from experience that updating jetty will break all kinds of commercial payroll apps which expect X version]. In the end, enterprise software upgrades at the rate of whatever the majority of accounting systems need to make payroll and tax filings happen. And then you find out that even if you move to the cloud, you still have to keep the old records active for 10+ years for all kinds of reasons.
Enterprise Operating Systems are the semi trucks of distributions.
They are slow, lumbering, noisy but carry a lot of stuff from point A to point B. Other operating systems are like smaller trucks or cars or even gulf-carts.. they each have their place and reasons for moving at the speeds they are for. The issue is to not try and expect a Bugatti to haul a 40 ton trailer or a Peterbilt to outlap a Lamborghini [you can do both with a lot of work, but there is little practicality of said work.] And I would not want either one of them on a golf course unless I was expecting to returf the entire area.
I'm not against retaining older streams of the software. Jetty 9.2 can stay for legacy reasons (although in 2020 I would not recommend running it, especially in systems that handle finances/payroll/taxes/PCI compliance tasks). Having different jetty packages where the package name is i.e. "jetty-9.4", "jetty-9.3", etc. and properly configuring the RPM SPEC conflicts can avoid 2 jetty versions being installed on the same system, with the same folder config. If you want to install a parallel, different version of Jetty, that would be something you have to do manually (as you also will need to change the default port bindings, folders, library, web root, etc.). If we do so though, we should set a more current, stable version as the default (aka in the jetty 9.4 SPEC set an alias of "jetty" so if someone types "yum -y install jetty" they get 9.4). A similar thing is currently done with OpenJDK and the various 7, 8, 9 versions. As for the dependencies, the .SPEC file has you covered as well. There is a Requires field and you can set version numbers on the dependencies as well (Requires: java >= 1:1.8.0, somelib < 1.2.1). Yes, we will need to also maintain those older dependencies in the RPM Repos.
As the OS guy for a major appliance, and as a former tech in both the datacenter and MSP fields, I can advise you in a lot of cases people and companies will put off upgrades as long as they can (Heck we still have customers using RHEL/CentOS 5, Windows XP and Server 2003, even though we dropped support for them 4 years ago), until it reaches a point where they are forced to upgrade. Maybe not right now in 8, but in streams we should be moving towards the more modern, stable versions of third-party packages, even if it means we need to also provide separate older packages as well (like jetty 9.3). At some point down the road, either on a point release, or the next major release, we should announce deprecation of the older versions (well in advance so enterprises have time to plan the upgrades) and remove them from production.
A couple points in-line for clarity:
On 1/7/20 4:09 AM, Neal Gompa wrote:
First, sorry that this email probably doesn't thread will with either one or the other target mailing lists... I had to pick one to thread with, and other slightly loses. The downside of the initial email not being sent to both MLs at once. :(
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature. There are several issues with doing this as it currently stands:
- CentOS Koji is pretty closed to a couple of folks in CentOS and Red
Hat. This also includes blocking the download of artifacts from Koji. For reasons I don't understand, CentOS is still gated by the RHEL QE folks, and thus RPMs built in Koji cannot be downloaded until they are finally published. This ruins any testing/development before releasing to users. The CBS is a worthless substitute since it's purely for addons. I've been around long enough to remember when Fedora was like this, and back then, it wasn't really considered "okay" for Fedora to be like that (the intent was always there to fix it, and we eventually did!). Today, however, it is considered "okay", and that's a problem.
There were a variety of reasons this was done initially. In part for bandwidth protection, ensuring debranding, and that packages had some sanity checking (they link appropriately and didn't need to be rebuilt, which happens frequently). Fabian and Brian have kicked around ideas for how we can solve the majority of these issues, so we will be resolving this in the future. We know it's not "okay", but it's also not the biggest problem we need to solve.
- The CentOS community is simply not mature enough to take on the
role Fedora does in any meaningful capacity. We already know that this is a problem even with Fedora EPEL, and I strongly believe the issue would be massively magnified if those transitioned to the CentOS Project. The CentOS community is largely a user community that takes and does not give back much, if at all. I've experienced variations of this issue when people (even from Red Hat!) have done finger-pointing to *me* with my packages in EPEL without any acknowledgement that they could have helped. It's depressing that even some Red Hatters consider EPEL not worth contributing to, but still worth being upset about.
That's because until Stream existed, there wasn't really much of a way to give back that didn't make the project deviate from its core mission. We have absolutely taken contributions from the community in the past, but most of the fixes offered were to fix bugs we didn't "own".
Stream gives us the opportunity to solicit and accept contributions, so this point feels like it's holding the past up as a reason to not address the future.
What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities for people to iterate on server-oriented changes before they go into rhel-rawhide and push into CentOS Stream. You also essentially kneecap any motivation for other things related to server environments to iterate faster (such as language stacks that are heavily used for web service software) because you've eliminated the major ability for that to ship to users and contributors. It also further accelerates a trend that I think we need to reverse where people consider Fedora unacceptable for server roles. If anything, Fedora is a lot better at being used for servers then it was five years ago. I've personally *stopped* using CentOS for servers because Fedora has gotten so good at it. Upgrades are a breeze and stuff generally works. When it doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".
I'd like to see more data driven discussion and to take the feelings and ego out of it. Anecdotal stories lead toward the loudest voice winning instead of the largest mass of users. This does assume that we (both Fedora and CentOS) have discussed which goals and metrics matter and should be used to shape a decision one way or the other.
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I disagree a bit here. I think it's worth discussing, and I think it's worth being clear about what the expectations are both for Stream, and Fedora Server.
I think we should have the conversation (possibly again) about what we want from Fedora Server. Is it serving the purpose originally envisioned. Should that continue to be the purpose for it, etc.
Stream is what RH *intends* to be in future versions of RHEL, and I think that intent matters when it comes to people developing for oVirt, RDO, ansible or something else that would run on top of RHEL. I think that intent matters for feature development when CERN or FERMI have fixes or input for things they want to see, and contribute code to make it happen. It will still be RH that makes the decision to commit to accepting those fixes and agreeing to put them into future RHEL releases. This is what we're actively building out, and what Carl is working on.
On 1/7/20 12:08 PM, Jim Perrin wrote:
On 1/7/20 4:09 AM, Neal Gompa wrote:
- CentOS Koji is pretty closed to a couple of folks in CentOS and Red
Hat.
There were a variety of reasons this was done initially.
...
That's because until Stream existed, there wasn't really much of a way to give back that didn't make the project deviate from its core mission. We have absolutely taken contributions from the community in the past, but most of the fixes offered were to fix bugs we didn't "own".
I don't think it's *entirely* fair to say that the fixes offered weren't relevant to CentOS, when no one outside the core maintainers had access to the build process, where fixes relevant to CentOS could be offered.
On 1/7/20 3:34 PM, Gordon Messmer wrote:
On 1/7/20 12:08 PM, Jim Perrin wrote:
On 1/7/20 4:09 AM, Neal Gompa wrote:
- CentOS Koji is pretty closed to a couple of folks in CentOS and Red
Hat.
There were a variety of reasons this was done initially.
...
That's because until Stream existed, there wasn't really much of a way to give back that didn't make the project deviate from its core mission. We have absolutely taken contributions from the community in the past, but most of the fixes offered were to fix bugs we didn't "own".
I don't think it's *entirely* fair to say that the fixes offered weren't relevant to CentOS, when no one outside the core maintainers had access to the build process, where fixes relevant to CentOS could be offered.
That is because CentOS Linux (other than Stream) is a rebuild of RHEL source code .. so, that is how things get into base CentOS Linux. We don't accept input that deviates base CentOS Linux from RHEL source code. So, the way to get things into CentOS Linux is to get it into RHEL Source Code. The CentOS team does not control that process.
Now that CentOS Stream is being done .. that will be the process to get things into RHEL Source Code from a CentOS Linux perspective.
On 1/8/20 7:43 AM, Johnny Hughes wrote:
I don't think it's*entirely* fair to say that the fixes offered weren't relevant to CentOS, when no one outside the core maintainers had access to the build process, where fixes relevant to CentOS could be offered.
That is because CentOS Linux (other than Stream) is a rebuild of RHEL source code .. so, that is how things get into base CentOS Linux.
Yes, I know.
We don't accept input that deviates base CentOS Linux from RHEL source code.
Of course not. But that's not the point. Neal pointed out that the process of debranding and rebuilding CentOS is not a community process, and that the user community takes and does not give back much. That is true. I agree with him.
I don't, however, know a good reason to believe that the community doesn't want to contribute, or isn't capable of contributing. I have seen people offer to contribute numerous times, especially when new major or minor releases lag far behind upstream. I tend to think those lags are evidence that the project does need more contributors, but I simply don't know any way for capable people to access the work queue and assist in getting it done.
When I look at https://wiki.centos.org/Contribute , I don't see any entry points for capable people to participate early in the process. I see a note that testing packages are announced on the devel list, but that's pretty far along in the process. It's hard to imagine that participating at that point will significantly improve the turnaround time for the process of rebuilding and publishing the system.
I could be wrong, since I see the situation from the perspective of the user community that I'm a part of, but my point is that Jim said "there wasn't really much of a way to give back that didn't make the project deviate from its core mission" and that is literally true. I'm not arguing otherwise. I just want to suggest that the fact that there wasn't a way to give back without deviating from the core mission is the result of decisions that the developers made to keep the process very closed and private, and we should acknowledge that when we're discussing community contributions.
On Wed, Jan 8, 2020, at 16:25, Gordon Messmer wrote:
On 1/8/20 7:43 AM, Johnny Hughes wrote:
I don't think it's*entirely* fair to say that the fixes offered weren't relevant to CentOS, when no one outside the core maintainers had access to the build process, where fixes relevant to CentOS could be offered.
That is because CentOS Linux (other than Stream) is a rebuild of RHEL source code .. so, that is how things get into base CentOS Linux.
Yes, I know.
We don't accept input that deviates base CentOS Linux from RHEL source code.
Of course not. But that's not the point. Neal pointed out that the process of debranding and rebuilding CentOS is not a community process, and that the user community takes and does not give back much. That is true. I agree with him.
I don't, however, know a good reason to believe that the community doesn't want to contribute, or isn't capable of contributing. I have seen people offer to contribute numerous times, especially when new major or minor releases lag far behind upstream. I tend to think those lags are evidence that the project does need more contributors, but I simply don't know any way for capable people to access the work queue and assist in getting it done.
When I look at https://wiki.centos.org/Contribute , I don't see any entry points for capable people to participate early in the process. I see a note that testing packages are announced on the devel list, but that's pretty far along in the process. It's hard to imagine that participating at that point will significantly improve the turnaround time for the process of rebuilding and publishing the system.
I could be wrong, since I see the situation from the perspective of the user community that I'm a part of, but my point is that Jim said "there wasn't really much of a way to give back that didn't make the project deviate from its core mission" and that is literally true. I'm not arguing otherwise. I just want to suggest that the fact that there wasn't a way to give back without deviating from the core mission is the result of decisions that the developers made to keep the process very closed and private, and we should acknowledge that when we're discussing community contributions.
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
Most of the delays for 8.x are related to the way we literally put CentOS together (buildsystem, compose, delivery), and major shifts in the technology that makes up Enterprise Linux (modules). Neither of these things are really ripe for large-scale contribution in CentOS Linux, although some of the decisions that were made at this layer put us in a better place for when CentOS Stream is fully realized.
I understand some of the frustrations mentioned here, taking the example of seeing the builds and not being able to download binaries we can look backwards at previous releases: - in CentOS 6 there were no public logs at all - in CentOS 7 there were public but somewhat hard to discover logs (buildlogs.centos.org) - in CentOS 8 there are public logs with a relatively familiar frontend (koji), with binaries hidden by a configuration option
Are we finished? absolutely not, but I don't want to discount the progress we've made, especially when we measure progress against 15 years worth of build/release processes that we're actively evaluating, and given that configuration options like this can be changed easily.
One goal that drove some of the decisions at the beginning of CentOS 8 was to refocus building the distribution itself around some on familiar tools. Koji, pungi, and dist-git (git.centos.org) are a lot more familiar to Fedora and RHEL workflows than reimzul and SRPMs. In the past buildsystem and tooling has been part of what has made releases so jarring for us, and coming closer to familiar workflows now lets us factor out some of those differences as we grow CentOS Stream and deal with future releases.
Refactoring systems takes time, is messy, and we made some tradeoffs, but the intent was and is to make transparency and contribution easier in a way that helps us think critically about the way we've done things in the past.
-- Brian Stinson
On 1/8/20 4:25 PM, Gordon Messmer wrote:
On 1/8/20 7:43 AM, Johnny Hughes wrote:
I don't think it's*entirely* fair to say that the fixes offered weren't relevant to CentOS, when no one outside the core maintainers had access to the build process, where fixes relevant to CentOS could be offered.
That is because CentOS Linux (other than Stream) is a rebuild of RHEL source code .. so, that is how things get into base CentOS Linux.
Yes, I know.
We don't accept input that deviates base CentOS Linux from RHEL source code.
Of course not. But that's not the point. Neal pointed out that the process of debranding and rebuilding CentOS is not a community process, and that the user community takes and does not give back much. That is true. I agree with him.
Well sure .. but that is also true for any Linux distribution. It's not like Ubuntu or Debian or Linux Mint (or anyone else) let community members build their distributions.
Obviously, what gets built and what gets accepted into their build system, as well as signature to verify the distributions are totally on system they control .. they validate any patches received before it gets into their distros too.
We are never going to allow other people to build things not in our validated systems and then sign and release it with CentOS Keys as CentOS Linux. That would be ridiculously stupid :)
And .. building the packages is NOT what is the hold up in most cases. Testing and validation is the major issue. Brian addressed a bunch of that in his mail.
I don't, however, know a good reason to believe that the community doesn't want to contribute, or isn't capable of contributing. I have seen people offer to contribute numerous times, especially when new major or minor releases lag far behind upstream. I tend to think those lags are evidence that the project does need more contributors, but I simply don't know any way for capable people to access the work queue and assist in getting it done.
Many users see CentOS as a clone (i hate that word, it is not a clone) of RHEL or even worse .. as Free RHEL. It is neither. It is rebuilt source code on a separate closed system and nothing more.
We don't need help building it .. we need help testing it.
If you want RHEL, with all its certifications and software assurance and CVE verification .. then buy RHEL. If RHEL ceases to exist, then so does CentOS Linux. It is just that simple. If you are running a business and if you have to answer security questions to investors, you need certified solutions. Again .. pretty simple. CentOS Linux has no certification of any kind. If you are using it, YOU are providing the certification that it mets YOUR needs.
When I look at https://wiki.centos.org/Contribute , I don't see any entry points for capable people to participate early in the process. I see a note that testing packages are announced on the devel list, but that's pretty far along in the process. It's hard to imagine that participating at that point will significantly improve the turnaround time for the process of rebuilding and publishing the system.
I could be wrong, since I see the situation from the perspective of the user community that I'm a part of, but my point is that Jim said "there wasn't really much of a way to give back that didn't make the project deviate from its core mission" and that is literally true. I'm not arguing otherwise. I just want to suggest that the fact that there wasn't a way to give back without deviating from the core mission is the result of decisions that the developers made to keep the process very closed and private, and we should acknowledge that when we're discussing community contributions.
Sure, as does every other distribution .. it can only be built from git.centos.org .. any changes to that need to be validated and accepted by a very small number of people to make sure it meets the standards.
There are special interst groups that build things NOT base CentOS Linux .. and they all can use more people. There are bugs listed at bugs.centos.org that need validated and if they are RHEL issues, bugs need to be filed at bugzilla.redhat.com to get those fixed. There are questions on the forums that need answers. All of these things can be done by the community .. if the community wants to do them. They all help everyone. Yet, no one does them.
If by community help .. you want community members building and signing the base os packages .. that is probably never going to happen. It also would never happen in Ubuntu or Debian or any other distribution.
We are looking at some mechanism to automate building things that do not need approval at some period of time. But that will be discussed later.
On 2020-01-10 16:19, Johnny Hughes wrote:
Well sure .. but that is also true for any Linux distribution. It's not like Ubuntu or Debian or Linux Mint (or anyone else) let community members build their distributions.
Do you consider Debian developers "community members"? The binary packages for one architecture (mostly amd64) are built, signed and uploaded by each Debian developer on their own machine, with the binaries for the other architectures being built by buildd on Debian infrastructure. It's probably also the reason they are investing so much effort in reproducible builds. That approach is controversial, building everything from source on dedicated machines belonging to the project, like Fedora is doing, would be more secure.
I don't think anyone wants your private keys. There are many ways to contribute to a Linux distro, from documentation, helping with sysadmin work, to contribute patches, packaging and developing bigger parts of software. Patches submitted by random users are not unusual for Debian, and the package maintainers are welcoming them and will adapt and integrate them, very fast (maybe Debian has more technical users than say, Ubuntu or Mint).
In principle, nothing would stop CentOS users from submitting patches, but it's more difficult to do so. The standard answer to bugs is "we only rebuild RHEL, go submit the bug to their Bugzilla and CentOS will only get the fix when and if they fix it". RH won't even consider bugs for CentOS installations, you have to install RHEL and reproduce it first (that requires significant time and effort and even then, good luck if you're not a major customer). Maybe they would accept submitted patches even if you're not a paying customer, not sure - they would still need to invest time to validate your patch. The entire process is much more closed compared to Debian or Fedora, so it's understandable that CentOS users feel kept at a distance and not really welcome (then again, Debian is a community distro, the interactions with Canonical or Novell might be similar to the ones with RH). I think that explains partially why our community is less involved than the Debian one.
We are never going to allow other people to build things not in our validated systems and then sign and release it with CentOS Keys as CentOS Linux. That would be ridiculously stupid :)
Indeed.
In principle, nothing would stop CentOS users from submitting patches, but it's more difficult to do so. The standard answer to bugs is "we only rebuild RHEL, go submit the bug to their Bugzilla and CentOS will only get the fix when and if they fix it". RH won't even consider bugs for CentOS installations, you have to install RHEL and reproduce it first (that requires significant time and effort and even then, good luck if you're not a major customer). Maybe they would accept submitted patches even if you're not a paying customer, not sure - they would still need to invest time to validate your patch. The entire process is much more closed compared to Debian or Fedora, so it's understandable that CentOS users feel kept at a distance and not really welcome (then again, Debian is a community distro, the interactions with Canonical or Novell might be similar to the ones with RH). I think that explains partially why our community is less involved than the Debian one.
I think this is a fair assessment of the status quo. Going forward, CentOS Stream is our opportunity to improve things. We are currently working on the guidelines for contributing to Stream. The contribution process won't be perfect right away, but hopefully we can address the points you brought up and more in an iterative fashion.
On Fri, Jan 10, 2020 at 1:05 PM Laurențiu Păncescu lpancescu@centosproject.org wrote:
On 2020-01-10 16:19, Johnny Hughes wrote:
Well sure .. but that is also true for any Linux distribution. It's not like Ubuntu or Debian or Linux Mint (or anyone else) let community members build their distributions.
Do you consider Debian developers "community members"? The binary packages for one architecture (mostly amd64) are built, signed and uploaded by each Debian developer on their own machine, with the binaries for the other architectures being built by buildd on Debian infrastructure. It's probably also the reason they are investing so much effort in reproducible builds. That approach is controversial, building everything from source on dedicated machines belonging to the project, like Fedora is doing, would be more secure.
I don't think anyone wants your private keys. There are many ways to contribute to a Linux distro, from documentation, helping with sysadmin work, to contribute patches, packaging and developing bigger parts of software. Patches submitted by random users are not unusual for Debian, and the package maintainers are welcoming them and will adapt and integrate them, very fast (maybe Debian has more technical users than say, Ubuntu or Mint).
In principle, nothing would stop CentOS users from submitting patches, but it's more difficult to do so. The standard answer to bugs is "we only rebuild RHEL, go submit the bug to their Bugzilla and CentOS will only get the fix when and if they fix it". RH won't even consider bugs for CentOS installations, you have to install RHEL and reproduce it first (that requires significant time and effort and even then, good luck if you're not a major customer). Maybe they would accept submitted patches even if you're not a paying customer, not sure - they would still need to invest time to validate your patch. The entire process is much more closed compared to Debian or Fedora, so it's understandable that CentOS users feel kept at a distance and not really welcome (then again, Debian is a community distro, the interactions with Canonical or Novell might be similar to the ones with RH). I think that explains partially why our community is less involved than the Debian one.
We are never going to allow other people to build things not in our validated systems and then sign and release it with CentOS Keys as CentOS Linux. That would be ridiculously stupid :)
Indeed. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 1/10/20 7:19 AM, Johnny Hughes wrote:
Well sure .. but that is also true for any Linux distribution. It's not like Ubuntu or Debian or Linux Mint (or anyone else) let community members build their distributions.
Debian and Fedora are exactly the kind of community-driven projects that I think CentOS should be, but isn't. They have documented processes for becoming a maintainer. Their package management is publicly available, so their current status and activity level are clearly visible. Anyone can fork a package, build and test their changes, and send suggested changes back in the form of a pull request in order to contribute to an existing maintainer without requiring access to build and publish their changes to end users.
CentOS does not have an onboarding process that I'm aware of, and does not appear to believe it needs one. There is a "git.centos.org" system, but all of the repos I looked at are empty (It's possible that's because I'm not logged in. Logging in currently results in a fatal error. Either way, it's disappointing). Koji is apparently gated to Red Hat QA only, so there's no opportunity or mechanism to collaborate until very late in the process.
Point being: I agree with the people in this thread who think it would be premature to discontinue Fedora Server, until CentOS becomes a mature community project.
We are never going to allow other people to build things not in our validated systems and then sign and release it with CentOS Keys as CentOS Linux. That would be ridiculously stupid :)
Of course it would. I'm bewildered as to why you're even arguing that point. It's possible that you think we are ridiculously stupid. It's possible that you're arguing in bad faith. It's possible that you're unfamiliar with the collaboration processes common across the entire Free Software community. None of those would be to your credit, so I'd love to hear another explanation.
On Sat, Jan 11, 2020, at 14:24, Gordon Messmer wrote:
On 1/10/20 7:19 AM, Johnny Hughes wrote:
Well sure .. but that is also true for any Linux distribution. It's not like Ubuntu or Debian or Linux Mint (or anyone else) let community members build their distributions.
Debian and Fedora are exactly the kind of community-driven projects that I think CentOS should be, but isn't. They have documented processes for becoming a maintainer. Their package management is publicly available, so their current status and activity level are clearly visible. Anyone can fork a package, build and test their changes, and send suggested changes back in the form of a pull request in order to contribute to an existing maintainer without requiring access to build and publish their changes to end users.
CentOS does not have an onboarding process that I'm aware of, and does not appear to believe it needs one. There is a "git.centos.org" system, but all of the repos I looked at are empty (It's possible that's because I'm not logged in. Logging in currently results in a fatal error. Either way, it's disappointing). Koji is apparently gated to Red Hat QA only, so there's no opportunity or mechanism to collaborate until very late in the process.
To be precise, Red Hat at-large has the exact same access to mbox (the koji that we use to build the distribution) as the rest of the community. It's the CentOS QA group (which includes a couple of members who happen to be Red Hat employees) that works with pre-release content for CentOS Linux. This is not much different from how content was handled in past releases, it's just now publicly obvious that builds are happening.
Point being: I agree with the people in this thread who think it would be premature to discontinue Fedora Server, until CentOS becomes a mature community project.
We are never going to allow other people to build things not in our validated systems and then sign and release it with CentOS Keys as CentOS Linux. That would be ridiculously stupid :)
Of course it would. I'm bewildered as to why you're even arguing that point. It's possible that you think we are ridiculously stupid. It's possible that you're arguing in bad faith. It's possible that you're unfamiliar with the collaboration processes common across the entire Free Software community. None of those would be to your credit, so I'd love to hear another explanation.
I think the pattern is different with CentOS Linux though. There isn't really much 'contribution' to be done because the idea is to match what comes from RHEL sources as closely as possible. We don't have individual package maintainers in CentOS because 99% of the time we can get builds through, the other 1% takes some targeted work from the QA group, or targeted contact with subject-matter experts from Fedora or RHEL.
What you're noting as the desired collaboration model is to have community maintainers accepting patches either for new features, or to follow new upstream releases of whatever the package happens to be. That model is indeed widely practiced in other distributions and FOSS projects, but it is probably a better fit for CentOS Stream and with the SIGs; such a model is an antipattern, though, for CentOS Linux.
--Brian
On 1/11/20 1:12 PM, Brian Stinson wrote:
I think the pattern is different with CentOS Linux though. There isn't really much 'contribution' to be done because the idea is to match what comes from RHEL sources as closely as possible. We don't have individual package maintainers in CentOS because 99% of the time we can get builds through
I think the frustrating thing about CentOS is that when users ask why delays are so long, the maintainers say "producing CentOS is't just a matter of rebuilding Red Hat sources" and when users ask why there isn't a process to contribute the maintainers say "we're just rebuilding Red Hat sources, and we don't need help with that."
On Sat, Jan 11, 2020, at 15:57, Gordon Messmer wrote:
On 1/11/20 1:12 PM, Brian Stinson wrote:
I think the pattern is different with CentOS Linux though. There isn't really much 'contribution' to be done because the idea is to match what comes from RHEL sources as closely as possible. We don't have individual package maintainers in CentOS because 99% of the time we can get builds through
I think the frustrating thing about CentOS is that when users ask why delays are so long, the maintainers say "producing CentOS is't just a matter of rebuilding Red Hat sources" and when users ask why there isn't a process to contribute the maintainers say "we're just rebuilding Red Hat sources, and we don't need help with that."
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
I guess that's sort of the point that I'm trying to make. Dealing with buildorder, dealing with changes between when Red Hat built a package and when they release a package, and trying to match content up 1:1 in the repos are what takes the most time. And those aren't things that fit with the package-maintainer model. Those activities happen precisely because we try to match sources as closely as possible, but it's also why we say that it's not *just* building all of the packages and blindly shipping whatever comes out of the buildsystem.
On Sat, 11 Jan 2020 at 16:57, Gordon Messmer gordon.messmer@gmail.com wrote:
On 1/11/20 1:12 PM, Brian Stinson wrote:
I think the pattern is different with CentOS Linux though. There isn't really much 'contribution' to be done because the idea is to match what comes from RHEL sources as closely as possible. We don't have individual package maintainers in CentOS because 99% of the time we can get builds through
I think the frustrating thing about CentOS is that when users ask why delays are so long, the maintainers say "producing CentOS is't just a matter of rebuilding Red Hat sources" and when users ask why there isn't a process to contribute the maintainers say "we're just rebuilding Red Hat sources, and we don't need help with that."
I think that the problem is that re-building an OS is a very very complicated problem and everyone doesn't expect it to be. The only way I have found to get a real idea of the amount of work is to actually set up your own build system and go through what is needed to rebuild the OS during the X.0 -> X.4 stage. You can get some set of packages built without problems and then you find that they don't have some feature that was expected. Then you have to work out the reorder of the builds to get that feature in the various tools. Then you figure out that you need to actually build the OS in multiple steps:
1. Take X.Y-1 and rebuild some of the packages from X.Y 2. Rebuild some of X-Y.1+new packages with that. 3. Rebuild X.Y packages with the set 4. Find out that a buildroot package (a package which isn't shipped by Red Hat but used to build the OS) isn't working. Figure out that they used a different version than what you have. 5. Go through that package docs and figure out which version/patch you did or did not need. 6. Rebuild X.Y packages 7. Re order some of the builds (but not all of them) and rebuild a couple more 8. Rebuild more of X.Y 9. Make a compose 10. Find out that the compose fails because X.Y used some new features. 11. Go back to 3 with a couple of other items.
After a couple of iterations you get an OS working and the QA people can do installs. There they find that something was wrong in X.Y-1 which causes updates to X.Y to break. Come up with new package fixes for that.
Spend each day getting pinged at least 3*Ndays (since X.Y release) times on IRC, personal email and mailing lists by users wanting to know why they can't update to the latest package yet.
At times you will try giving out some packages before you have the entire compose, and then find out that email is flooded with broken systems which needed some soft dependency you haven't gotten done yet. Or you might think you got all of them and find out that you had to go back to 3 but someone who put X.Y-beta in production on 4000 servers is spamming every list about how your OS is crap... and lots of people add on to it that they found the software was broken too. [You might get some people who point out that it was not ready and then you get slammed with people saying if it wasn't ready why did you even let anyone get it.]
Pretty much everyone I know who has done a rebuild of RHEL or CentOS or the clone of CentOS they found.. ends up finding this is what a release is like.
On Sun, 12 Jan 2020 at 00:58, Stephen John Smoogen smooge@gmail.com wrote:
On Sat, 11 Jan 2020 at 16:57, Gordon Messmer gordon.messmer@gmail.com wrote:
On 1/11/20 1:12 PM, Brian Stinson wrote:
I think the pattern is different with CentOS Linux though. There isn't
really much 'contribution' to be done because the idea is to match what comes from RHEL sources as closely as possible. We don't have individual package maintainers in CentOS because 99% of the time we can get builds through
I think the frustrating thing about CentOS is that when users ask why delays are so long, the maintainers say "producing CentOS is't just a matter of rebuilding Red Hat sources" and when users ask why there isn't a process to contribute the maintainers say "we're just rebuilding Red Hat sources, and we don't need help with that."
I think that the problem is that re-building an OS is a very very complicated problem and everyone doesn't expect it to be. The only way I have found to get a real idea of the amount of work is to actually set up your own build system and go through what is needed to rebuild the OS during the X.0 -> X.4 stage. You can get some set of packages built without problems and then you find that they don't have some feature that was expected. Then you have to work out the reorder of the builds to get that feature in the various tools. Then you figure out that you need to actually build the OS in multiple steps:
- Take X.Y-1 and rebuild some of the packages from X.Y
- Rebuild some of X-Y.1+new packages with that.
- Rebuild X.Y packages with the set
- Find out that a buildroot package (a package which isn't shipped by
Red Hat but used to build the OS) isn't working. Figure out that they used a different version than what you have. 5. Go through that package docs and figure out which version/patch you did or did not need. 6. Rebuild X.Y packages 7. Re order some of the builds (but not all of them) and rebuild a couple more 8. Rebuild more of X.Y 9. Make a compose 10. Find out that the compose fails because X.Y used some new features. 11. Go back to 3 with a couple of other items.
After a couple of iterations you get an OS working and the QA people can do installs. There they find that something was wrong in X.Y-1 which causes updates to X.Y to break. Come up with new package fixes for that.
Great, this sounds like the process of building an OS in multiple steps
that I am familiar with; Is there any way to make this process more transparant? Letting the community know what step in the process you are at, what is failing, what patches have been tried, and when you are sleeping let people build some packages on top of your work, find out what works and contribute that back to you so in the morning when you wake up the entire process is a few steps further along?
I have some experience in building a build system/framework (and community of people contributing patches to this framework in parallel) that methodically keeps track of every package build and what dependencies work and don't work in different stages of the build. We accept input from people who tried builds that failed, figured out the problem and submitted a patch to the build process to us using this framework, in the end we can automatically generate a complete reproducible build with the press of a button, that can be rebuild on the machine of everyone involved.
Does this sound like anything you would want to have in the future? Or are you happy with doing the above process behind closed doors with just a few people in control?
-- Stephen J Smoogen. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Tue, Jan 7, 2020 at 12:08 PM Jim Perrin jperrin@centos.org wrote:
On 1/7/20 4:09 AM, Neal Gompa wrote:
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I disagree a bit here. I think it's worth discussing, and I think it's worth being clear about what the expectations are both for Stream, and Fedora Server.
I think we should have the conversation (possibly again) about what we want from Fedora Server. Is it serving the purpose originally envisioned. Should that continue to be the purpose for it, etc.
Stream is what RH *intends* to be in future versions of RHEL, and I think that intent matters when it comes to people developing for oVirt, RDO, ansible or something else that would run on top of RHEL. I think that intent matters for feature development when CERN or FERMI have fixes or input for things they want to see, and contribute code to make it happen. It will still be RH that makes the decision to commit to accepting those fixes and agreeing to put them into future RHEL releases. This is what we're actively building out, and what Carl is working on.
Between Neal and Jim's excellent points and counterpoints I'd like to relate a little extra RHEL information that may aid discussion. When we launched RHEL 8 we shared 2 goals for future release timings- One got all the attention, the other almost none. The focus was predictable minor release updates every 6 months (So far so good on that front). The less noticed goal was to make a new major every ~3 years. RHEL 9 will, if all goes well, will spend its pre-release development at the intersection of Fedora Server and CentOS Stream. The exact form this takes is yet to be determined, though we have some ideas getting into shape to be shared. This will be the first major release since the advent of CentOS Stream. It's a very good time to talk about future goals for Fedora Server, goals for CentOS SIGs and community members all. My simple observation and suggestion is this: CentOS Stream creates opportunities for us to work together in unprecedented ways, so let's explore some ideas, dare to dream, *then* work on plans to make them practical.
@Matthew
First, let's replace Fedora Server as a user-focused artifact with CentOS Stream. There's still a need for a Fedora Server as an upstream, but most users of Fedora as a server are making custom builds (or using the basic Cloud image), not using Fedora Server per se. So, I think we should stop marketing that to users, and steer people with server use cases outside of the fast-moving niche to CentOS Stream. The current story of CentOS and Fedora is pretty confusing to users, and Stream only makes it more so.
Let's
simplify things!
I rather enjoy using Fedora on servers. I'm concerned that steering users away from this use case sends a bad message. It's a common misconception that it's a bad idea to use Fedora on servers. I'd rather the project itself not implicitly agree with this.
My personal preference would be for Fedora Server and CentOS Stream to be presented as equally attractive options and let users choose the one that suits their needs. Perhaps the marketing for both could cross reference each other. Fedora Server's marketing could say something like "If you're concerned about the fast release cycle of Fedora, CentOS Stream may be more to your liking". CentOS Stream's marketing could say something like "If you need newer versions of core packages, you may consider Fedora Server".
Second, where there's overlap, let's look at merging Fedora and CentOS
SIGs
into joint bodies. We have groups like the cloud image SIGs which are basically doing the exact same thing in two different ways. Same thing
with
containers. We also have huge overlaps in documentation and community user support and all sorts of non-technical project areas. I'm not saying these all need to be smooshed together, but where it makes sense let's work together. This, too, brings simplification to the "what's with CentOS vs. Fedora" story: rather than making contributors pick, we can welcome all
with
open arms.
I *love* this idea. We could rebrand them as "Red Hat Community SIGs" that operate in both the Fedora and CentOS communities. I think CentOS SIGs could learn a lot from how Fedora SIGs operate.
@Neal
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature.
You raised some great points. I want to reiterate this part of Matthew's original email:
"...in this next decade..."
I think it's reasonable to believe we can address the concerns you raised within the next 10 years. Your feedback (and everyone else's) is critical to getting us there, so thank you for sharing it.
@Jim and Neal
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I disagree a bit here. I think it's worth discussing, and I think it's worth being clear about what the expectations are both for Stream, and Fedora Server.
I think we should have the conversation (possibly again) about what we want from Fedora Server. Is it serving the purpose originally envisioned. Should that continue to be the purpose for it, etc.
I think you're both right. I don't want Fedora Server to be eliminated, but I do believe we should have a clear understanding of how they overlap and how they differ in order to present them appropriately to the community.
On Tue, Jan 7, 2020 at 2:09 PM Jim Perrin jperrin@centos.org wrote:
A couple points in-line for clarity:
On 1/7/20 4:09 AM, Neal Gompa wrote:
First, sorry that this email probably doesn't thread will with either one or the other target mailing lists... I had to pick one to thread with, and other slightly loses. The downside of the initial email not being sent to both MLs at once. :(
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature. There are several issues with doing this as it currently stands:
- CentOS Koji is pretty closed to a couple of folks in CentOS and Red
Hat. This also includes blocking the download of artifacts from Koji. For reasons I don't understand, CentOS is still gated by the RHEL QE folks, and thus RPMs built in Koji cannot be downloaded until they are finally published. This ruins any testing/development before releasing to users. The CBS is a worthless substitute since it's purely for addons. I've been around long enough to remember when Fedora was like this, and back then, it wasn't really considered "okay" for Fedora to be like that (the intent was always there to fix it, and we eventually did!). Today, however, it is considered "okay", and that's a problem.
There were a variety of reasons this was done initially. In part for bandwidth protection, ensuring debranding, and that packages had some sanity checking (they link appropriately and didn't need to be rebuilt, which happens frequently). Fabian and Brian have kicked around ideas for how we can solve the majority of these issues, so we will be resolving this in the future. We know it's not "okay", but it's also not the biggest problem we need to solve.
- The CentOS community is simply not mature enough to take on the
role Fedora does in any meaningful capacity. We already know that this is a problem even with Fedora EPEL, and I strongly believe the issue would be massively magnified if those transitioned to the CentOS Project. The CentOS community is largely a user community that takes and does not give back much, if at all. I've experienced variations of this issue when people (even from Red Hat!) have done finger-pointing to *me* with my packages in EPEL without any acknowledgement that they could have helped. It's depressing that even some Red Hatters consider EPEL not worth contributing to, but still worth being upset about.
That's because until Stream existed, there wasn't really much of a way to give back that didn't make the project deviate from its core mission. We have absolutely taken contributions from the community in the past, but most of the fixes offered were to fix bugs we didn't "own".
Stream gives us the opportunity to solicit and accept contributions, so this point feels like it's holding the past up as a reason to not address the future.
What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities for people to iterate on server-oriented changes before they go into rhel-rawhide and push into CentOS Stream. You also essentially kneecap any motivation for other things related to server environments to iterate faster (such as language stacks that are heavily used for web service software) because you've eliminated the major ability for that to ship to users and contributors. It also further accelerates a trend that I think we need to reverse where people consider Fedora unacceptable for server roles. If anything, Fedora is a lot better at being used for servers then it was five years ago. I've personally *stopped* using CentOS for servers because Fedora has gotten so good at it. Upgrades are a breeze and stuff generally works. When it doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".
I'd like to see more data driven discussion and to take the feelings and ego out of it. Anecdotal stories lead toward the loudest voice winning instead of the largest mass of users. This does assume that we (both Fedora and CentOS) have discussed which goals and metrics matter and should be used to shape a decision one way or the other.
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I disagree a bit here. I think it's worth discussing, and I think it's worth being clear about what the expectations are both for Stream, and Fedora Server.
I think we should have the conversation (possibly again) about what we want from Fedora Server. Is it serving the purpose originally envisioned. Should that continue to be the purpose for it, etc.
Stream is what RH *intends* to be in future versions of RHEL, and I think that intent matters when it comes to people developing for oVirt, RDO, ansible or something else that would run on top of RHEL. I think that intent matters for feature development when CERN or FERMI have fixes or input for things they want to see, and contribute code to make it happen. It will still be RH that makes the decision to commit to accepting those fixes and agreeing to put them into future RHEL releases. This is what we're actively building out, and what Carl is working on.
-- Jim Perrin The CentOS Project | http://www.centos.org twitter: @BitIntegrity | GPG Key: FA09AD77 _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
Second, where there's overlap, let's look at merging Fedora and CentOS SIGs into joint bodies. We have groups like the cloud image SIGs which are basically doing the exact same thing in two different ways. Same thing with containers. We also have huge overlaps in documentation and community user support and all sorts of non-technical project areas. I'm not saying these all need to be smooshed together, but where it makes sense let's work together. This, too, brings simplification to the "what's with CentOS vs. Fedora" story: rather than making contributors pick, we can welcome all with open arms.
I *love* this idea. We could rebrand them as "Red Hat Community SIGs" that operate in both the Fedora and CentOS communities. I think CentOS SIGs could learn a lot from how Fedora SIGs operate.
While I agree with consolidation, over time, of the SIGs, I do think the name should be “Enterprise Linux Community SIGs” or similar. I get that Red Hat has a massive influence in all four flavours (Fedora, RHEL, CentOS, Streams), but for someone new to the communities, it could be confusing to refer to them as “Red Hat”, as most people outside the community see “Red Hat” as referring only to “RHEL” and not the wider Enterprise Linux ecosystem.
Greg
Gregory Young
From: CentOS-devel centos-devel-bounces@centos.org On Behalf Of Carl George Sent: Wednesday, January 8, 2020 9:30 AM To: The CentOS developers mailing list. centos-devel@centos.org Subject: Re: [CentOS-devel] A Big Idea for a New Decade [was: Minutes for CentOS Board of Directors 2019-12-18 Meeting]
@Matthew
First, let's replace Fedora Server as a user-focused artifact with CentOS Stream. There's still a need for a Fedora Server as an upstream, but most users of Fedora as a server are making custom builds (or using the basic Cloud image), not using Fedora Server per se. So, I think we should stop marketing that to users, and steer people with server use cases outside of the fast-moving niche to CentOS Stream. The current story of CentOS and Fedora is pretty confusing to users, and Stream only makes it more so. Let's simplify things!
I rather enjoy using Fedora on servers. I'm concerned that steering users away from this use case sends a bad message. It's a common misconception that it's a bad idea to use Fedora on servers. I'd rather the project itself not implicitly agree with this.
My personal preference would be for Fedora Server and CentOS Stream to be presented as equally attractive options and let users choose the one that suits their needs. Perhaps the marketing for both could cross reference each other. Fedora Server's marketing could say something like "If you're concerned about the fast release cycle of Fedora, CentOS Stream may be more to your liking". CentOS Stream's marketing could say something like "If you need newer versions of core packages, you may consider Fedora Server".
Second, where there's overlap, let's look at merging Fedora and CentOS SIGs into joint bodies. We have groups like the cloud image SIGs which are basically doing the exact same thing in two different ways. Same thing with containers. We also have huge overlaps in documentation and community user support and all sorts of non-technical project areas. I'm not saying these all need to be smooshed together, but where it makes sense let's work together. This, too, brings simplification to the "what's with CentOS vs. Fedora" story: rather than making contributors pick, we can welcome all with open arms.
I *love* this idea. We could rebrand them as "Red Hat Community SIGs" that operate in both the Fedora and CentOS communities. I think CentOS SIGs could learn a lot from how Fedora SIGs operate.
@Neal
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature.
You raised some great points. I want to reiterate this part of Matthew's original email:
"...in this next decade..."
I think it's reasonable to believe we can address the concerns you raised within the next 10 years. Your feedback (and everyone else's) is critical to getting us there, so thank you for sharing it.
@Jim and Neal
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I disagree a bit here. I think it's worth discussing, and I think it's worth being clear about what the expectations are both for Stream, and Fedora Server.
I think we should have the conversation (possibly again) about what we want from Fedora Server. Is it serving the purpose originally envisioned. Should that continue to be the purpose for it, etc.
I think you're both right. I don't want Fedora Server to be eliminated, but I do believe we should have a clear understanding of how they overlap and how they differ in order to present them appropriately to the community.
On Tue, Jan 7, 2020 at 2:09 PM Jim Perrin <jperrin@centos.orgmailto:jperrin@centos.org> wrote: A couple points in-line for clarity:
On 1/7/20 4:09 AM, Neal Gompa wrote:
First, sorry that this email probably doesn't thread will with either one or the other target mailing lists... I had to pick one to thread with, and other slightly loses. The downside of the initial email not being sent to both MLs at once. :(
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature. There are several issues with doing this as it currently stands:
- CentOS Koji is pretty closed to a couple of folks in CentOS and Red
Hat. This also includes blocking the download of artifacts from Koji. For reasons I don't understand, CentOS is still gated by the RHEL QE folks, and thus RPMs built in Koji cannot be downloaded until they are finally published. This ruins any testing/development before releasing to users. The CBS is a worthless substitute since it's purely for addons. I've been around long enough to remember when Fedora was like this, and back then, it wasn't really considered "okay" for Fedora to be like that (the intent was always there to fix it, and we eventually did!). Today, however, it is considered "okay", and that's a problem.
There were a variety of reasons this was done initially. In part for bandwidth protection, ensuring debranding, and that packages had some sanity checking (they link appropriately and didn't need to be rebuilt, which happens frequently). Fabian and Brian have kicked around ideas for how we can solve the majority of these issues, so we will be resolving this in the future. We know it's not "okay", but it's also not the biggest problem we need to solve.
- The CentOS community is simply not mature enough to take on the
role Fedora does in any meaningful capacity. We already know that this is a problem even with Fedora EPEL, and I strongly believe the issue would be massively magnified if those transitioned to the CentOS Project. The CentOS community is largely a user community that takes and does not give back much, if at all. I've experienced variations of this issue when people (even from Red Hat!) have done finger-pointing to *me* with my packages in EPEL without any acknowledgement that they could have helped. It's depressing that even some Red Hatters consider EPEL not worth contributing to, but still worth being upset about.
That's because until Stream existed, there wasn't really much of a way to give back that didn't make the project deviate from its core mission. We have absolutely taken contributions from the community in the past, but most of the fixes offered were to fix bugs we didn't "own".
Stream gives us the opportunity to solicit and accept contributions, so this point feels like it's holding the past up as a reason to not address the future.
What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities for people to iterate on server-oriented changes before they go into rhel-rawhide and push into CentOS Stream. You also essentially kneecap any motivation for other things related to server environments to iterate faster (such as language stacks that are heavily used for web service software) because you've eliminated the major ability for that to ship to users and contributors. It also further accelerates a trend that I think we need to reverse where people consider Fedora unacceptable for server roles. If anything, Fedora is a lot better at being used for servers then it was five years ago. I've personally *stopped* using CentOS for servers because Fedora has gotten so good at it. Upgrades are a breeze and stuff generally works. When it doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".
I'd like to see more data driven discussion and to take the feelings and ego out of it. Anecdotal stories lead toward the loudest voice winning instead of the largest mass of users. This does assume that we (both Fedora and CentOS) have discussed which goals and metrics matter and should be used to shape a decision one way or the other.
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I disagree a bit here. I think it's worth discussing, and I think it's worth being clear about what the expectations are both for Stream, and Fedora Server.
I think we should have the conversation (possibly again) about what we want from Fedora Server. Is it serving the purpose originally envisioned. Should that continue to be the purpose for it, etc.
Stream is what RH *intends* to be in future versions of RHEL, and I think that intent matters when it comes to people developing for oVirt, RDO, ansible or something else that would run on top of RHEL. I think that intent matters for feature development when CERN or FERMI have fixes or input for things they want to see, and contribute code to make it happen. It will still be RH that makes the decision to commit to accepting those fixes and agreeing to put them into future RHEL releases. This is what we're actively building out, and what Carl is working on.
-- Jim Perrin The CentOS Project | http://www.centos.org twitter: @BitIntegrity | GPG Key: FA09AD77 _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.orgmailto:CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-- Carl George
While I agree with consolidation, over time, of the SIGs, I do think the
name
should be “Enterprise Linux Community SIGs” or similar. I get that Red
Hat has
a massive influence in all four flavours (Fedora, RHEL, CentOS, Streams),
but
for someone new to the communities, it could be confusing to refer to
them as
“Red Hat”, as most people outside the community see “Red Hat” as
referring only
to “RHEL” and not the wider Enterprise Linux ecosystem.
Fedora is not Enterprise Linux. To me, the name "Enterprise Linux Community" excludes Fedora. I'm not married to the name "Red Hat Community", but it's the best name I can think of that accurately conveys the scope. What term would you use to describe the entire ecosystem of Fedora, RHEL, CentOS, CentOS Stream, and EPEL? I've always called this the "Red Hat ecosystem".
On Wed, Jan 8, 2020 at 9:20 AM Young, Gregory gregory.young@solarwinds.com wrote:
Second, where there's overlap, let's look at merging Fedora and CentOS
SIGs
into joint bodies. We have groups like the cloud image SIGs which are
basically doing the exact same thing in two different ways. Same thing
with
containers. We also have huge overlaps in documentation and community
user
support and all sorts of non-technical project areas. I'm not saying
these
all need to be smooshed together, but where it makes sense let's work
together. This, too, brings simplification to the "what's with CentOS
vs.
Fedora" story: rather than making contributors pick, we can welcome all
with
open arms.
I *love* this idea. We could rebrand them as "Red Hat Community SIGs"
that
operate in both the Fedora and CentOS communities. I think CentOS SIGs
could
learn a lot from how Fedora SIGs operate.
While I agree with consolidation, over time, of the SIGs, I do think the name should be “Enterprise Linux Community SIGs” or similar. I get that Red Hat has a massive influence in all four flavours (Fedora, RHEL, CentOS, Streams), but for someone new to the communities, it could be confusing to refer to them as “Red Hat”, as most people outside the community see “Red Hat” as referring only to “RHEL” and not the wider Enterprise Linux ecosystem.
Greg
*Gregory Young *
*From:* CentOS-devel centos-devel-bounces@centos.org *On Behalf Of *Carl George *Sent:* Wednesday, January 8, 2020 9:30 AM *To:* The CentOS developers mailing list. centos-devel@centos.org *Subject:* Re: [CentOS-devel] A Big Idea for a New Decade [was: Minutes for CentOS Board of Directors 2019-12-18 Meeting]
@Matthew
First, let's replace Fedora Server as a user-focused artifact with CentOS Stream. There's still a need for a Fedora Server as an upstream, but most users of Fedora as a server are making custom builds (or using the basic Cloud image), not using Fedora Server per se. So, I think we should stop marketing that to users, and steer people with server use cases outside
of
the fast-moving niche to CentOS Stream. The current story of CentOS and Fedora is pretty confusing to users, and Stream only makes it more so.
Let's
simplify things!
I rather enjoy using Fedora on servers. I'm concerned that steering users away from this use case sends a bad message. It's a common misconception that it's a bad idea to use Fedora on servers. I'd rather the project itself not implicitly agree with this.
My personal preference would be for Fedora Server and CentOS Stream to be presented as equally attractive options and let users choose the one that suits their needs. Perhaps the marketing for both could cross reference each other. Fedora Server's marketing could say something like "If you're concerned about the fast release cycle of Fedora, CentOS Stream may be more to your liking". CentOS Stream's marketing could say something like "If you need newer versions of core packages, you may consider Fedora Server".
Second, where there's overlap, let's look at merging Fedora and CentOS
SIGs
into joint bodies. We have groups like the cloud image SIGs which are basically doing the exact same thing in two different ways. Same thing
with
containers. We also have huge overlaps in documentation and community
user
support and all sorts of non-technical project areas. I'm not saying
these
all need to be smooshed together, but where it makes sense let's work together. This, too, brings simplification to the "what's with CentOS vs. Fedora" story: rather than making contributors pick, we can welcome all
with
open arms.
I *love* this idea. We could rebrand them as "Red Hat Community SIGs" that operate in both the Fedora and CentOS communities. I think CentOS SIGs could learn a lot from how Fedora SIGs operate.
@Neal
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature.
You raised some great points. I want to reiterate this part of Matthew's
original email:
"...in this next decade..."
I think it's reasonable to believe we can address the concerns you raised within the next 10 years. Your feedback (and everyone else's) is critical to getting us there, so thank you for sharing it.
@Jim and Neal
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I disagree a bit here. I think it's worth discussing, and I think it's worth being clear about what the expectations are both for Stream, and Fedora Server.
I think we should have the conversation (possibly again) about what we want from Fedora Server. Is it serving the purpose originally envisioned. Should that continue to be the purpose for it, etc.
I think you're both right. I don't want Fedora Server to be eliminated, but I do believe we should have a clear understanding of how they overlap and how they differ in order to present them appropriately to the community.
On Tue, Jan 7, 2020 at 2:09 PM Jim Perrin jperrin@centos.org wrote:
A couple points in-line for clarity:
On 1/7/20 4:09 AM, Neal Gompa wrote:
First, sorry that this email probably doesn't thread will with either one or the other target mailing lists... I had to pick one to thread with, and other slightly loses. The downside of the initial email not being sent to both MLs at once. :(
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature. There are several issues with doing this as it currently stands:
- CentOS Koji is pretty closed to a couple of folks in CentOS and Red
Hat. This also includes blocking the download of artifacts from Koji. For reasons I don't understand, CentOS is still gated by the RHEL QE folks, and thus RPMs built in Koji cannot be downloaded until they are finally published. This ruins any testing/development before releasing to users. The CBS is a worthless substitute since it's purely for addons. I've been around long enough to remember when Fedora was like this, and back then, it wasn't really considered "okay" for Fedora to be like that (the intent was always there to fix it, and we eventually did!). Today, however, it is considered "okay", and that's a problem.
There were a variety of reasons this was done initially. In part for bandwidth protection, ensuring debranding, and that packages had some sanity checking (they link appropriately and didn't need to be rebuilt, which happens frequently). Fabian and Brian have kicked around ideas for how we can solve the majority of these issues, so we will be resolving this in the future. We know it's not "okay", but it's also not the biggest problem we need to solve.
- The CentOS community is simply not mature enough to take on the
role Fedora does in any meaningful capacity. We already know that this is a problem even with Fedora EPEL, and I strongly believe the issue would be massively magnified if those transitioned to the CentOS Project. The CentOS community is largely a user community that takes and does not give back much, if at all. I've experienced variations of this issue when people (even from Red Hat!) have done finger-pointing to *me* with my packages in EPEL without any acknowledgement that they could have helped. It's depressing that even some Red Hatters consider EPEL not worth contributing to, but still worth being upset about.
That's because until Stream existed, there wasn't really much of a way to give back that didn't make the project deviate from its core mission. We have absolutely taken contributions from the community in the past, but most of the fixes offered were to fix bugs we didn't "own".
Stream gives us the opportunity to solicit and accept contributions, so this point feels like it's holding the past up as a reason to not address the future.
What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities for people to iterate on server-oriented changes before they go into rhel-rawhide and push into CentOS Stream. You also essentially kneecap any motivation for other things related to server environments to iterate faster (such as language stacks that are heavily used for web service software) because you've eliminated the major ability for that to ship to users and contributors. It also further accelerates a trend that I think we need to reverse where people consider Fedora unacceptable for server roles. If anything, Fedora is a lot better at being used for servers then it was five years ago. I've personally *stopped* using CentOS for servers because Fedora has gotten so good at it. Upgrades are a breeze and stuff generally works. When it doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".
I'd like to see more data driven discussion and to take the feelings and ego out of it. Anecdotal stories lead toward the loudest voice winning instead of the largest mass of users. This does assume that we (both Fedora and CentOS) have discussed which goals and metrics matter and should be used to shape a decision one way or the other.
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I disagree a bit here. I think it's worth discussing, and I think it's worth being clear about what the expectations are both for Stream, and Fedora Server.
I think we should have the conversation (possibly again) about what we want from Fedora Server. Is it serving the purpose originally envisioned. Should that continue to be the purpose for it, etc.
Stream is what RH *intends* to be in future versions of RHEL, and I think that intent matters when it comes to people developing for oVirt, RDO, ansible or something else that would run on top of RHEL. I think that intent matters for feature development when CERN or FERMI have fixes or input for things they want to see, and contribute code to make it happen. It will still be RH that makes the decision to commit to accepting those fixes and agreeing to put them into future RHEL releases. This is what we're actively building out, and what Carl is working on.
-- Jim Perrin The CentOS Project | http://www.centos.org twitter: @BitIntegrity | GPG Key: FA09AD77 _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
--
Carl George _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 1/7/20 1:09 PM, Neal Gompa wrote:
- Today, the CentOS Project is (sadly) a community project in a very
superficial sense. Aside from the Virt SIG's Xen subgroup, all SIGs are basically Red Hat product teams, with varying degrees of allowances for community contributions. The community processes within CentOS are mostly nonfunctional because they've had no reason to develop it.
I think that's true. I'm a non-RH member of the CentOS Cloud Instance SIG and I've been developing the CentOS images for Vagrant since the beginning of 2016. Having to have the generated images copied to cloud.centos.org by a member of the core team and then signed by another one (both RH, both very busy) made the entire process slower and more frustrating than it needs to be. Things have gotten worse since CentOS 8, since we lost the ability to build the images in CBS due to the builders being EL7. I built the first Vagrant images for 8 locally around November 2019 using libvirt and submitted a PR on GitHub. I wanted to ask on the list for testers for VirtualBox, HyperV and VMware; I think I have around 4 unanswered emails to the core team, and I discovered there are centos/8 images available for download, albeit only for libvirt and VirtualBox, without any explanation about what's going on. It doesn't even make sense to generate new images for 6 or 7 on CBS, since I'm not able to upload them myself and my emails go unanswered. When I volunteer work and my free time, having to beg people all the time and be ignored is not my idea of fun.
Out of curiosity, how is the experience of contributing to Fedora if you're not a RH employee, same as here?
Best regards, Laurențiu
-
Brendan Conoboy -
Brian Stinson -
Carl George -
Gordon Messmer -
Jens Timmerman -
Jim Perrin -
Johnny Hughes -
Jonathan Billings -
Laurențiu Păncescu -
Matthew Miller -
Neal Gompa -
Rich Bowen -
Stephen John Smoogen -
Young, Gregory