On Sat, Jan 2, 2021 at 8:50 AM orkcu via CentOS-devel centos-devel@centos.org wrote:
I don't think CentOS (or any other clone/rebuild) was ever a Bazaar, it can not be if the goal is to be a good clone/rebuild. Since the very beginning the control was in a handful of people, very tight, and I remember one crisis when the founder didn't want (was unable/unavailable) to release control of dns domain, sign keys and few other main components that now you complain RedHat don't want to release
I think you are right. The original intent, and the announcements and documentation for the project suggested something greater, but the reality turned out to be difficulty in delegation and scale. This is a common problem for volunteer or non-profit organizations. It doesn't matter if you have 3000 people willing to work on your project for free, if you can't effectively assign them work to do. It's worse if you can't even scale beyond a small number like "3", as these 3 eventually move on, and there is no succession plan, resulting in the project failing.
For many of us - I think we had a naive trust that this would be figured out, and we would be invited when the time came to invite us. However, it didn't happen, and in 2014, Red Hat offered to take the reins. It's not that Red Hat set out to destroy the community - it is only that Red Hat has a conflict of interest, and it is really difficult for this conflict of interest to not eventually result in favouring certain self-interest outcomes over other non self-interest outcomes, and this is exactly what is happening in 2021. In effect, it was an inevitable conclusion. It was even predicted by several people in 2014.
A similar thing is playing out in Rocky right now, however, there is a great deal of accumulated know-how, access to resources, and sober reflection on what went wrong leading up to 2014 and 2020. 3000+ people willing to work for free, but probably only 70 will be able to directly contribute, and initially the number will be a lot fewer as the initial setup takes place. This is why I say that Rocky is what CentOS should have been in 2014. We all know it wasn't. Now there is a chance to fix this. It could be a very good thing.
I don't like CentOS to evolve to Stream, I agree there is a need of a cheap RHEL's quality distribution, and I think all the arguments given so far at the end (in the very bottom) it is about cost (money), from the people complaining about the end because we are losing a very good distro that was free/gratis, from RedHat it is about moving the money to a direction that might provide better benefits for the company. If RedHat would offer its RHEL with a minimum price (20? 50? 100?) for self support and then charge for a ticket a good amount of money ( like 200? 500?), that might work for a lot of people who are complaining now, I think we all understand that everyone need to have an income to feed its needs, even the small shops that are trying to save in (software) costs as much as possible.
If Red Hat had agreed to this in 2016 and 2017 when I was asking for a reasonable solution to our problem, the company I work for would still be a Red Hat only shop. This was an opportunity missed then, and I believe it will be an opportunity missed when the new plans are announced. I don't believe the new subscription plans will be enough to make them an attractive option, and after the stumble with CentOS, even a good subscription plan that might meet some use cases will be frowned upon as a result of this debacle.
Bottom line, I don't think we ever had a community built OS in CentOS, we had people of the community assisting in testing and QA, but not even that was open to the public. I might be wrong but I think even before 2014 we had builders that were not (much) active in mailing list or IRC, and that is OK, each person have different strengths.
I don't think you are wrong. Healthy communities need to prioritize delegation and scalability. It means investing specific effort in ensuring that people of different skills and capabilities can successfully contribute to the project.
Centos is RedHat's property, full stop. Even in the post 2014 centos bylaws, RedHat can do whatever it damn pleases with centos, including but not limited to cheerfully ignoring the centos steering committee. I would not be surprised if it has the option to replace the entire committee,
Stop thinking of centos as a community project and start seeing it as RedHat's IP and everything -- licensing, the new relationship with RHES -- will make sense.
The Cheese has moved: follow the cheese or find a new one. Each choice has repercussions.
+1
On Sat, Jan 2, 2021, 9:03 PM Mauricio Tavares raubvogel@gmail.com wrote:
Centos is RedHat's property, full stop. Even in the post 2014centos bylaws, RedHat can do whatever it damn pleases with centos, including but not limited to cheerfully ignoring the centos steering committee. I would not be surprised if it has the option to replace the entire committee,
Stop thinking of centos as a community project and start seeing it as RedHat's IP and everything -- licensing, the new relationship with RHES -- will make sense.
The Cheese has moved: follow the cheese or find a new one. Each choice has repercussions. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Sat, Jan 2, 2021 at 9:03 PM Mauricio Tavares raubvogel@gmail.com wrote:
Centos is RedHat's property, full stop. Even in the post 2014centos bylaws, RedHat can do whatever it damn pleases with centos, including but not limited to cheerfully ignoring the centos steering committee. I would not be surprised if it has the option to replace the entire committee,
Yes, they can. Including the option of fracturing the EL community and dis-enfranchising even long-term Red Hat supporters. They can totally do all of this. And, if they admitted to this openly, they might even deserve kudos for being so bold. :-)
It is the charade of it all that is the most frustrating. For example, saying that it is "upstream" so this will facilitate contribution, but then admitting that actually Red Hat is still the upstream, and Red Hat will make all the decisions on what content and direction is allowed. At least Fedora has some ability to choose a different path, like BTRFS as one example. Will CentOS Stream community members be permitted to add BTRFS back into CentOS Stream? Or will Red Hat prevent such a thing?
Stop thinking of centos as a community project and start seeing it as RedHat's IP and everything -- licensing, the new relationship with RHES -- will make sense.
Yes, exactly. And this is why the response should be in this exact context. This includes shining a light on what has just happened, and making sure that the correct business decisions are made as a response to this event. Red Hat is a business, and deserves the exact same scrutiny and public review as the community would give to Oracle or Microsoft.
The Cheese has moved: follow the cheese or find a new one. Each choice has repercussions.
Yes. :-)
On Sat, Jan 02, 2021 at 11:51:35PM -0500, Mark Mielke wrote:
It is the charade of it all that is the most frustrating. For example, saying that it is "upstream" so this will facilitate contribution, but then admitting that actually Red Hat is still the upstream, and Red Hat will make all the decisions on what content and direction is allowed. At least Fedora has some ability to choose a different path, like BTRFS as one example. Will CentOS Stream community members be permitted to add BTRFS back into CentOS Stream? Or will Red Hat prevent such a thing?
It's not a "charade". You don't have to believe it, but everything we're saying is sincere.
"Upstream" doesn't necessarily mean "community controlled upstream". This is common in a lot of open source projects with corporate backing. (I think Fedora is pretty special in this regard.) But it does mean that it's open to the community for feedback in the ways that have been described, in a way which is a clear improvement from the previous internal-only structure.
The original CentOS stream announcement used the term "midstream", which I like, but I also understand that introducing new terminology can create problems of its own. That announcement is also clear about the contribution model for Stream — transparent but ultimately Red Hat decisions. Characterizing further explanations of this as some kind of gotcha "admitting" isn't fair.
As for BTRFS, I'm sure this has already been answered, but I'll say it again: Red Hat is definitely not at this time interested in BTRFS for RHEL, therefore unless Red Hat engineer and business minds are changed, that's not going to happen in CentOS Stream. However, a CentOS SIG which adds BTRFS would be very welcome.
And, honestly, putting my Fedora hat back on, this makes perfect sense. We already _have_ the Fedora Project. We don't need a second, overlapping one. If you want to work directly on operating system design, come join us over in Fedora. If you want to help guide Red Hat's decisions for RHEL minor releases, if you want early access to that development, if you want to build alternate ideas on that base, you're already in the right place.
On Sunday, January 3, 2021 11:36 AM, Matthew Miller mattdm@mattdm.org wrote:
> On Sat, Jan 02, 2021 at 11:51:35PM -0500, Mark Mielke wrote: > > > It is the charade of it all that is the most frustrating. For example, > > saying that it is "upstream" so this will facilitate contribution, but > > then admitting that actually Red Hat is still the upstream, and Red > > Hat will make all the decisions on what content and direction is > > allowed. At least Fedora has some ability to choose a different path, > > like BTRFS as one example. Will CentOS Stream community members be > > permitted to add BTRFS back into CentOS Stream? Or will Red Hat > > prevent such a thing? > > It's not a "charade". You don't have to believe it, but everything we're > saying is sincere. > > "Upstream" doesn't necessarily mean "community controlled upstream". This is > common in a lot of open source projects with corporate backing. (I think > Fedora is pretty special in this regard.) But it does mean that it's open to > the community for feedback in the ways that have been described, in a way > which is a clear improvement from the previous internal-only structure.
Here is a claim from a Karsten Wade that feels like a charade:
> "On the product/customer side, there was an openness gap—RHEL users (and consequently all rebuild users) couldn’t contribute easily to RHEL."
How exactly are we being empowered to "contribute easily to RHEL?"
Will Red Hat's new recognition of an openness gap mean we will finally get access to the kernel patches? Or did Wade just say that because he thought it would sound good? Is there any point in the community suggesting a SIG based around a better delivery system for kpatch?
Even if we aren't fully community controlled, how will the roadmap of Stream/RHEL be communicated? Will Bex *EVER* introduce himself to the mailing list or do we get to guess what the roadmap is based on how packages are changing over time?
Will there be a community controlled Stream Plus packages? Currently the "centosplus" directory but the Stream one continues to sit empty. Will it someday silently just disappear? Is there any point in purposing a BTRFS SIG to start populating Stream Plus?
When Red Hat employees post to the mailing list that we already have enough information to make a decision, does that indicate they have any interest in answering any of the above questions?!
> The original CentOS stream announcement used the term "midstream", which I > like, but I also understand that introducing new terminology can create > problems of its own. That announcement is also clear about the contribution > model for Stream — transparent but ultimately Red Hat decisions. > Characterizing further explanations of this as some kind of gotcha > "admitting" isn't fair. > > As for BTRFS, I'm sure this has already been answered, but I'll say it > again: Red Hat is definitely not at this time interested in BTRFS for RHEL, > therefore unless Red Hat engineer and business minds are changed, that's not > going to happen in CentOS Stream. However, a CentOS SIG which adds BTRFS > would be very welcome.
The issue of BTRFS for RHEL was indirectly answered by the addition of Stratis in RHEL 8. Red Hat put a lot of work into make XFS have the look/feel of BTRFS.
Also, when you say "very welcome" I think you mean Stream SIGs will still need a governance board sponsor to proceed. Which of the governance board members has publicly expressed at any point in 2020 an interest in making Stream SIGs feel very welcome in getting sponsored.
Maybe we can work with Mike McLean, Red Hat senior software engineer, about getting a BTRFS SIG added to the empty Stream Plus directory? He introduced himself and gave a list of SIGs he is interested in sponsoring on the mailing list on ... well, never.
How about Brian Exelbierd? Stream's roadmap is RHEL's roadmap. Clearly interacting with the community should be important to him now? He publicly express interest in sponsoring a SIG... uh, never.
Or should we go to Jim Perrin at Microsoft?? Is he even still on the governance board? Is the list even kept up to date on who to ask to get a sponsor for a SIG?
How about you, Matthew Miller? Can we go to you to get Stream SIG sponsorship? Are you even on the governance board for Stream? Do you have any personal investment in making us "very welcome" to form a BTRFS SIG?
Should I continue down the list?
There is a reason why Stream Plus directory is empty. We might find someone that pretends to express interest and asks us to do the work to make the packages only to then indicate there was no promise they would be the ones to sponsor.
Maybe it is unfair for us to toss around the word "charade" or maybe that is the hard reality?
> And, honestly, putting my Fedora hat back on, this makes perfect sense. We > already have the Fedora Project. We don't need a second, overlapping one. > If you want to work directly on operating system design, come join us over > in Fedora. If you want to help guide Red Hat's decisions for RHEL minor > releases, if you want early access to that development, if you want to build > alternate ideas on that base, you're already in the right place.
Yes, we have a RHEL Insider Fast Ring called the Fedora Project.
What would make perfect sense is to make the RHEL Insider Slow Ring called Stream fall under Fedora Project as well.
If the CentOS community needs to pull their own weight to keep CentOS 8 going beyond 2021, then they should just say so.
Having Karsten Wade say that the vote back on November 11th was about Red Hat wanting to work with the community by forcing the termination of CentOS 8 is insulting. They demostrated a desire to work with the community by waiting nearly a month to announce the change?
What SIG I want to work on is keeping CentOS 8 going without Red Hat employees having to do the work. How "very welcome" will that SIG purposal go?
Instead of giving the community options, Wade makes the false claim that there exist a document which shows Stream can cover 95% of user workloads. Stef Walter's document on "Stream is Continuous Delivery" is extremely well written and makes me excited about Stream. (At this point it may sounds like I am being sarcastic, but I'm not. I really think Stef Walter did a wonderful job. I wish he was on the governance board.) But Walter never says anything about 95% user workloads. Wade is misrepresenting what Walter's blog post is about.
I honestly don't know if the word charade is being thrown around unfairly or not. I do know if someone ask me to defend Red Hat's recent actions, I wouldn't know how to "sell" this. I do feel confident that the 2014 merger assurances have panned out to be a charade. Maybe only time can tell if this is another charade or not.
On Mon, Jan 4, 2021 at 2:54 AM redbaronbrowser via CentOS-devel centos-devel@centos.org wrote:
As for BTRFS, I'm sure this has already been answered, but I'll say it again: Red Hat is definitely not at this time interested in BTRFS for RHEL, therefore unless Red Hat engineer and business minds are changed, that's not going to happen in CentOS Stream. However, a CentOS SIG which adds BTRFS would be very welcome.
I asked about BTRFS earlier and as far as I can tell, it was not answered - at least not in this thread, or in the context of "CentOS Stream" and what this new commitment to openness would do to change the answer.
The reason I asked about BTRFS is because of this:
The issue of BTRFS for RHEL was indirectly answered by the addition of Stratis in RHEL 8. Red Hat put a lot of work into make XFS have the look/feel of BTRFS.
Architecturally, Stratis seems to follow the Docker LVM / dm-thinp storage driver, which was an interesting approach, but ultimately ended up being deprecated as a poor model due to its emphasis on correct file system isolation over practical concerns like shared file system caching and performance. It means that it performs poorly at scale, and I suspect the same thing will happen with Stratis. There are too many compromises being made with Stratis, to re-use XFS for a purpose that it wasn't really meant for. BTRFS will prove to be better, having a much larger community to support it, and Stratis will end up a Red Hat only project, that does not get broadly adopted. Even if Stratis might be better - without the community, it will eventually be too expensive to invest in given the limited returns. But, this isn't a discussion about Stratis... I'm just giving some context of what a reasonable community discussion *might* include...
The question is about how CentOS Stream can ensure this conversation happens, and that the "community" (not just developers, but USERS... you know, the people that the product is being designed to satisfy?) can provide this feedback, and provide true influence on the direction of RHEL. Right now, Red Hat is deciding to go with Stratis, and drop BTRFS, in direct contrast to what the "community" wants to see. I would argue right now, that one of the *best* reasons to use Oracle Linux instead of CentOS or RHEL, is because Oracle supports BTRFS in their distribution, and if you need BTRFS for an Enterprise Linux distribution, you don't really have a choice any more. Red Hat opted out.
Also, when you say "very welcome" I think you mean Stream SIGs will still need a governance board sponsor to proceed. Which of the governance board members has publicly expressed at any point in 2020 an interest in making Stream SIGs feel very welcome in getting sponsored.
I don't understand why SIG requires CentOS Stream, and does not require CentOS. Eventually, we're going to have a paradox to deal with:
1) CentOS Stream is the only supported "free" EL distribution from Red Hat. 2) CentOS SIG will need to support all of: CentOS Stream, RHEL, and at least one other - Rocky?
This paradox means that the SIG now have *more* work to do than before. What about projects like RDO? Will they be publishing for Fedora, CentOS Stream, and EL? Will they be able to do this long-term? Or will they end up:
A) Dropping EL support, and focusing on CentOS Stream. Now, projects like RDO only work on CentOS Stream, and don't work on RHEL? How will this help RHEL? B) Dropping CentOS Stream support, and only work on RHEL / Rocky? How will this leverage CentOS Stream?
This is a mess about to happen. This mess is worse due to the people who claim CentOS Stream is good enough for "95%" of all users. It will lead to SIG specifically targeting CentOS Stream and only CentOS Stream, and no guarantee that their product output will work on RHEL or Rocky, since RHEL and Rocky will be older baselines that cannot be easily or reliably built from a CentOS Stream source. Do the people making the decisions realize this? Because, it leaves me in a state of concern that either they do know this, and they are planning to totally upset the ecosystem (without any consultation with the users who are the most impacted), or they don't know this, and they are soon to be surprised (leaving the users to scramble from one sinking ship to another at their own peril).
Maybe it is unfair for us to toss around the word "charade" or maybe that is the hard reality?
Individual people are sincere. I believe Matthew is sincere.
However, sincerity doesn't really change the results. I am not claiming anybody here is a bad person or is being less than completely sincere. I am claiming that there are varied interests, and the people with these interests are emphasizing the values that they see, and downplaying the consequences that they are aware of, and the result is basically the same thing. Willingly, or unwillingly, it is a charade - "an absurd pretense intended to create a pleasant or respectable appearance". In this case, Red Hat is killing CentOS as we know it (including using their acquisition of the trademark to force it to die), and selling this as a win for the "community" (= which doesn't seem to include users?) everywhere, because instead of CentOS, an OS distro that inherits the hardened nature of RHEL, you can now run CentOS Stream as a preview of RHEL instead, upstream of RHEL, and it will be good enough for "95% of users" (with no clear definition of who these users are). This is the absurd pretense intended to create a please or respectable appearance. Not everybody is making this absurd pretense, but some still are. Or, perhaps, I just need a little "faith"... it is a Cathedral, and I am basically speaking blasphemy. :-)
And, honestly, putting my Fedora hat back on, this makes perfect sense. We already have the Fedora Project. We don't need a second, overlapping one. If you want to work directly on operating system design, come join us over in Fedora. If you want to help guide Red Hat's decisions for RHEL minor releases, if you want early access to that development, if you want to build alternate ideas on that base, you're already in the right place.
Yes, we have a RHEL Insider Fast Ring called the Fedora Project. What would make perfect sense is to make the RHEL Insider Slow Ring called Stream fall under Fedora Project as well.
RHEL minor releases introduce and deprecate functions. RHEL has historically been around 10 years, and that is a long time in Internet years. Docker was in its infancy in RHEL 7.0 time-frame, and thinks like overlayfs were necessary backports to keep RHEL 7 relevant. This is part of the definition of the "slow ring", and ability to contribute to it means ability to influence what decisions get made, despite having stronger requirements for stability from the users. Fedora moves too quickly, and doesn't have a point release. It can be used for production purposes (I've done it before), but it comes at a cost that shouldn't be taken lightly, similarly but to a lesser degree, why CentOS Stream in production use cases would come at a cost that should not be taken lightly. You basically have to be prepared for your systems to break at any time, and have your own testing and stabilization processes in place, to prevent this from happening. Some of us can do this - most of us cannot. And, even for those that can do it - we want to use it sparingly, only where it matters the most. Not everything has to be new all the time, despite what easily excitable developers will push for. We need both an upstream and a downstream to have an effective community. CentOS Stream is not a downstream - it is neither downstream of Fedora, nor downstream of RHEL. CentOS Stream is a branch of RHEL (and RHEL is a branch of Fedora) that runs ahead of RHEL, and acts as a more effective beta program for RHEL. I would love to have a more effective beta program for RHEL, but this is not a replacement for CentOS.
Also on the comment of "come join us over in Fedora", I do. Fedora is a very important project that goes beyond RHEL. I'm one of your many "users" who you co-opt as developers according to the Bazaar vs Cathedral models, by doing a lot of the hard work of diagnosing problems and raising them to the attention of people who can fix them, including patches when I am able to. But, companies don't run on just one thing. I work with and on Fedora, so that 2 to 5 years from now, it will be available in EL. It is a long-term investment. The same would apply to CentOS Stream, and the same applies to CentOS. All of these are necessary components. Without CentOS, I don't see how SIG can confirm their products will work on EL - whether RHEL, or Rocky, or Oracle Linux. Possibly, Red Hat will make RHEL free for this limited use case, and effectively constrain the size of the community to either those who pay nearly full list price for RHEL for all systems (as is required today), or special licenses that are encumbered, that might allow a limited amount of "free" usage, but is almost too hard to obtain in the first place, so most people don't bother.
On 1/4/21 2:53 AM, redbaronbrowser via CentOS-devel wrote:
Also, when you say "very welcome" I think you mean Stream SIGs will still need a governance board sponsor to proceed. Which of the governance board members has publicly expressed at any point in 2020 an interest in making Stream SIGs feel very welcome in getting sponsored.
I would be glad to be your sponsor in this. Note that I don't have any of the technical chops in BTRFS, but that's not the role of the board sponsor.
Maybe we can work with Mike McLean, Red Hat senior software engineer, about getting a BTRFS SIG added to the empty Stream Plus directory? He introduced himself and gave a list of SIGs he is interested in sponsoring on the mailing list on ... well, never.
That's not how this works. A SIG is proposed (Where was that proposal? I certainly didn't see it on the mailing list, and would appreciate a pointer.) and a sponsor steps up. It's documented here: https://wiki.centos.org/SIGGuide Asking directors to provide a list of the SIGs they'd be willing to sponsor is backwards - particularly when you've been complaining through this entire thread that governance is top-down rather than community-drive. You're the community. Feel free to drive.
If you want to make a proposal, I'd be glad to twist Mike's arm to step up, as he'd be a better fit than me, for the reasons you list. Perhaps this should be in a new thread, since this thread is *very* dense, and chances are most folks won't make it this far.
How about Brian Exelbierd? Stream's roadmap is RHEL's roadmap. Clearly interacting with the community should be important to him now? He publicly express interest in sponsoring a SIG... uh, never.
Again ... that's not how this works.
Or should we go to Jim Perrin at Microsoft?? Is he even still on the governance board? Is the list even kept up to date on who to ask to get a sponsor for a SIG?
Yes. The list is up to date. I updated it after Ralph stepped down, as per the recently-published meeting minutes.
I think a lot of this is either already well-addressed (like the participation model in Stream) or will become clear over time. You reference Stef's blog post, so I know you're keeping up on those things and don't need repeats. So, I'm not going to do a line-by-line reply. I do want to specifically address the parts where you ask about directly about me and suggest things for Fedora, though:
How about you, Matthew Miller? Can we go to you to get Stream SIG sponsorship? Are you even on the governance board for Stream? Do you have any personal investment in making us "very welcome" to form a BTRFS SIG?
I am not part of CentOS governance. As Rich says, the list of members is current.
However, I very much care about CentOS success as part of the whole Red Hat distribution family and a Fedora downstream, and because as I mentioned I consider myself part of the CentOS community. So, while I can't sponsor anything, I very much would like to see CentOS SIGs and Fedora teams working more closely together. I'm positive that the people working on BTRFS in Fedora would also like to see it in CentOS Stream. So I'm happy to do what I can to facilitate that.
Yes, we have a RHEL Insider Fast Ring called the Fedora Project. What would make perfect sense is to make the RHEL Insider Slow Ring called Stream fall under Fedora Project as well.
I've thought about this a lot! While I love things that grow Fedora, I think there's also benefit to a clear distinction. Since the Fedora Extras/Core merge, it's an underlying expectation that we have community participation and access across the board. There's no "only Red Hat decides this". With CentOS, on the other hand, the core expectation has _always_ been that Red Hat makes the fundamental engineering decisions for the OS itself while the CentOS community does things _around_ that. It's a different kind of project with a different kind of participation.
CentOS Stream doesn't change that expectation for the CentOS Project; it's just a better version of the model: discussion and pull requests welcome, rather than "well, you could file a bug with Red Hat and hope someone notices".
Centos is RedHat's property, full stop. Even in the post 2014
No, not really. That may be true 100% for the name CentOS, the content however is owned 99% by others.
We should never forget that when comparing CentOS with other software products.
Regards, Simon
centos bylaws, RedHat can do whatever it damn pleases with centos, including but not limited to cheerfully ignoring the centos steering committee. I would not be surprised if it has the option to replace the entire committee,
Stop thinking of centos as a community project and start seeing it as RedHat's IP and everything -- licensing, the new relationship with RHES -- will make sense.
The Cheese has moved: follow the cheese or find a new one. Each choice has repercussions. _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
-
Jeffrey Layton -
Mark Mielke -
Matthew Miller -
Mauricio Tavares -
redbaronbrowser -
Rich Bowen -
Simon Matter