[CentOS-devel] A Big Idea for a New Decade [was: Minutes for CentOS Board of Directors 2019-12-18 Meeting]

Young, Gregory

gregory.young at solarwinds.com
Tue Jan 7 15:48:38 UTC 2020

My personal feeling is there have been very high barriers to contributing to CentOS.

1. Almost every bug report and fix needs to go to/come from RHEL and Fedora. Very few bug reports are fixed in CentOS then submitted upstream to RHEL and Fedora.
2. The Contributing Documentation is sparse and very confusing. It needs to be re-written and updated, and the Wiki needs to either be upgraded to something more user friendly, or replaced with a proper knowledgebase application.
3. Meetings need to be held in a more accessible location. Freenode IRC may have worked well in the past, but many companies are locking down their outbound rules and blocking services like IRC. I used to use a SSH tunnel to access Freenode, but even there, the "scheduled" meetings for the SIGs never seemed to happen. Even looking at the minutes from many, you see the host is the only person who joins. Let's migrate to a Slack/Discord server for meetings, and figure out a way to better publish the meetings.
4. We should be encouraging the Fedora members to join the CentOS SIGs and vice versa. If the two teams see what each other are working on, they can eliminate overlap, and I have a feeling they would quickly work on merging their efforts into one SIG.
5. The process to contribute should be made a lot easier. As an example, I have a patch I want to submit to the Jetty project. Their process is clearly defined, and the only holdup I am having right now is getting the internal approval from my employer to contribute (the normal review processes). With CentOS, I'm not sure what I can contribute to, and what I need to contribute to RHEL or Fedora. I also find that bug reports to all three go un-touched for years at a time.
6. For the love of all that is pink and fluffy, we need to update the versions of third party packages we ship. If RHEL won't, CentOS should. For instance, we still ship Jetty 9.2, which is EOL and not receiving security updates. 9.3 is also EOL. 9.4 is quite stable at this point (as they are about to go beta on 10.0), so we should be shipping 9.4. PostgreSQL is another example. We ship 9.2. 12.1 is the current release, and the appliance I work on is shipping 10.11 as it's stable version. 9.2 is so far out of EOL... it has been superseded by 9.3, 9.4, 9.5, 9.6, 10, 11 and 12. If you want an example of what we should be shipping as the stable versions, cPanel is a very good example. They run on CentOS, but sooooooo many packages have to be built by them to make their servers secure.

Gregory Young

-----Original Message-----
From: CentOS-devel <centos-devel-bounces at centos.org> On Behalf Of Neal Gompa
Sent: Tuesday, January 7, 2020 7:10 AM
To: Discussions with the Fedora Council and community <council-discuss at lists.fedoraproject.org>
Cc: The CentOS developers mailing list. <CentOS-devel at centos.org>
Subject: Re: [CentOS-devel] A Big Idea for a New Decade [was: Minutes for CentOS Board of Directors 2019-12-18 Meeting]

On Mon, Jan 6, 2020 at 4:13 PM Matthew Miller <mattdm at mattdm.org> wrote:
> On Fri, Jan 03, 2020 at 03:13:53PM -0500, Karsten Wade wrote:
> > As a centralizing effort, the Board agreed to revisiting the goals 
> > document created five years ago, and to undergo an effort to refresh 
> > those goals in the light of the project’s evolution. The Board will 
> > be inviting various stakeholders into these discussions as we 
> > undergo a public revision of the goals at the start of 2020.
> Wow, five years goes fast! I'd like to raise an idea for discussion 
> and thought, and possible further discussion between the Fedora 
> Council and the CentOS board. [I'm posting this in both groups.]
> Around the same time the CentOS goals were being created, Fedora was 
> working on plans for "Fedora.next", which included among other things Fedora Server.
> I won't rehash all that here, but in short: while that succeeded at 
> some of its goals, it never really had the user adoption we'd hoped 
> for. Of course, some of that is simply a matter of lifecycle: while a 
> fast-moving server OS is useful in some very real cases, it's decidedly niche.
> When Fedora launched that plan, CentOS wasn't really in the "family", 
> and after, we just kind of existed in parallel, with some very loose 
> collaboration in areas like EPEL.
> Of course, things are different now. Therefore, I propose that in this 
> next decade, we make that collaboration stronger. Together, we have a 
> very nice answer to the fundamental problem of unified operating 
> systems. That is, some parts move too fast and other parts move too 
> slowly. Between Fedora and CentOS, we have answers to both! And, with 
> the in-progress dist-git merge, we even have both _in the same place_.
> Specifically, I'd like to suggest three things.
> First, let's replace Fedora Server as a user-focused artifact with 
> CentOS Stream. There's still a need for a Fedora Server as an 
> upstream, but most users of Fedora as a server are making custom 
> builds (or using the basic Cloud image), not using Fedora Server per 
> se. So, I think we should stop marketing that to users, and steer 
> people with server use cases outside of the fast-moving niche to 
> CentOS Stream. The current story of CentOS and Fedora is pretty 
> confusing to users, and Stream only makes it more so. Let's simplify things!
> Second, where there's overlap, let's look at merging Fedora and CentOS 
> SIGs into joint bodies. We have groups like the cloud image SIGs which 
> are basically doing the exact same thing in two different ways. Same 
> thing with containers. We also have huge overlaps in documentation and 
> community user support and all sorts of non-technical project areas. 
> I'm not saying these all need to be smooshed together, but where it 
> makes sense let's work together. This, too, brings simplification to the "what's with CentOS vs.
> Fedora" story: rather than making contributors pick, we can welcome 
> all with open arms.
> And finally, let's really take advantage of a merged dist-git 
> repository, and the capabilities that Modularity gives us. SIGs 
> building things for CentOS should be able to directly reference Fedora 
> branches where faster-moving software is useful — and people building 
> things in Fedora should be able to pull in branches from Stream where 
> a slower-moving version would make sense. And we could even make 
> additional shared branches where necessary, using modularity to make 
> optional streams which could be used in CentOS SIGs, EPEL, or a Fedora OS.
> What do you think?

First, sorry that this email probably doesn't thread will with either one or the other target mailing lists... I had to pick one to thread with, and other slightly loses. The downside of the initial email not being sent to both MLs at once. :(

In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature. There are several issues with doing this as it currently

1. CentOS Koji is pretty closed to a couple of folks in CentOS and Red Hat. This also includes blocking the download of artifacts from Koji.
For reasons I don't understand, CentOS is still gated by the RHEL QE folks, and thus RPMs built in Koji cannot be downloaded until they are finally published. This ruins any testing/development before releasing to users. The CBS is a worthless substitute since it's purely for addons. I've been around long enough to remember when Fedora was like this, and back then, it wasn't really considered "okay" for Fedora to be like that (the intent was always there to fix it, and we eventually did!). Today, however, it is considered "okay", and that's a problem.

2. The CentOS community is simply not mature enough to take on the role Fedora does in any meaningful capacity. We already know that this is a problem even with Fedora EPEL, and I strongly believe the issue would be massively magnified if those transitioned to the CentOS Project. The CentOS community is largely a user community that takes and does not give back much, if at all. I've experienced variations of this issue when people (even from Red Hat!) have done finger-pointing to *me* with my packages in EPEL without any acknowledgement that they could have helped. It's depressing that even some Red Hatters consider EPEL not worth contributing to, but still worth being upset about.

3. Today, the CentOS Project is (sadly) a community project in a very superficial sense. Aside from the Virt SIG's Xen subgroup, all SIGs are basically Red Hat product teams, with varying degrees of allowances for community contributions. The community processes within CentOS are mostly nonfunctional because they've had no reason to develop it. There has been very little community development for CentOS. Even with Fermilab and CERN shutting down Scientific Linux, I haven't seen anything from those people coming "into the fold".
There's been no outreach or collaboration with other CentOS based derivatives, commercial or otherwise. The private discussions I've had on the matter seem to indicate that there's still ambivalence, if not outright antipathy and hostility toward the broader ecosystem of CentOS derivatives.

What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities for people to iterate on server-oriented changes before they go into rhel-rawhide and push into CentOS Stream. You also essentially kneecap any motivation for other things related to server environments to iterate faster (such as language stacks that are heavily used for web service software) because you've eliminated the major ability for that to ship to users and contributors. It also further accelerates a trend that I think we need to reverse where people consider Fedora unacceptable for server roles. If anything, Fedora is a lot better at being used for servers then it was five years ago. I've personally
*stopped* using CentOS for servers because Fedora has gotten so good at it. Upgrades are a breeze and stuff generally works. When it doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".

So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?

真実はいつも一つ!/ Always, there's only one truth!
CentOS-devel mailing list
CentOS-devel at centos.org

More information about the CentOS-devel mailing list