On 12/11/20 9:56 AM, Ljubomir Ljubojevic wrote:
And I will repeat that millions of CentOS users found free clone of RHEL trustworthy enough to use it for production, even without "official endorsement".
Exactly. That's why it's so weird that those people, today, think that CentOS Stream won't be usable, based on their interpretation of the official statements from Red Hat. Red Hat's statements weren't taken into consideration before, but now they're a sign of doom?
If they ... even allowed ANYONE ELSE that was not employed by Red Hat in 2014 to even come close to learning the secrets of rebuild, no backlash would have happened
I'm going to stop you there, because the CentOS maintainers kept that process out of public visibility long before Red Hat was ever involved. If you think users should know more about the process, then you are pointing fingers at the *wrong* people.
I don't want this to become a flame war. So rather than pointing fingers, let's focus on the fact that CentOS Stream promises to be developed in the open, resolving the problem that you're describing.
Red Hat is fixing the thing you're complaining about.
Red Hat is giving us the thing that has been requested more often, by more people, than any other change in CentOS, and the result is that the press is full of stories about users being angry, because five people on the mailing lists sent a lot of messages. (About half of the traffic in the threads on centos and centos-devel comes from five people, and various people replying to them.)
But no, as soon as Oracle started rebuilding RHEL source code Red Hat first made things difficult for everyone to create kernels (source code was not srpms anymore but tar?)
You're misinformed. Kernels are still built from SRPM, but the archive used is no longer an upstream release with a series of patches.
The reason for the change is not insidious. It's unfortunate that the pristine source + patches can't be maintained, I agree, but speaking as a developer: maintaining hundreds of patches that touch intersecting files and rebasing them all when earlier patches are updated is an incredibly difficult and time consuming task. And, if I remember correctly, applying all of those patches took almost as long as building the kernel. If it takes that long to just prepare the source code, that's a major hit to productivity when a developer needs to work on the code or build the SRPM to validate changes.
And, ultimately, there's very little value in shipping those patches when the vast majority of them are already in the current version of the upstream kernel, and they're merely backported to the older release that Red Hat supports. It's an entirely different story when distributions are shipping patches that they don't push upstream, but that's not generally what you see with the kernel package.
On 13.12.2020 11:48, Gordon Messmer wrote:
On 12/11/20 9:56 AM, Ljubomir Ljubojevic wrote:
And I will repeat that millions of CentOS users found free clone of RHEL trustworthy enough to use it for production, even without "official endorsement".
Exactly. That's why it's so weird that those people, today, think that CentOS Stream won't be usable, based on their interpretation of the official statements from Red Hat. Red Hat's statements weren't taken into consideration before, but now they're a sign of doom?
Who exactly said "doom"?
CentOS Stream won't match corresponding stable RHEL version, that's all. While CentOS was matching it, it was stable enough to use it safely on production.
Now that it becomes constant beta, it might be considered less stable, all the compatibility arguments have been uttered many a time.
Doom or no doom, everyone decides for oneself.
The primary problem is breach of trust. All the other consequences are mostly technical.
On 13/12/2020 6:48 π.μ., Gordon Messmer wrote:
Red Hat is giving us the thing that has been requested more often, by more people, than any other change in CentOS, and the result is that the press is full of stories about users being angry, because five people on the mailing lists sent a lot of messages. (About half of the traffic in the threads on centos and centos-devel comes from five people, and various people replying to them.)
Not really. I am afraid you are missing the point. Also see: 7500 sysadmins and growing are already explicitly rejecting the change.
...but speaking as a developer...
That's the problem: you are speaking as a developer, not as a sysadmin. CentOS is clearly for sysadmins, not for developers.
On 13/12/2020 10:22 π.μ., Nicolas Kovacs wrote:
For the last 16 years, the explicit scope of the CentOS project has been to rebuild RHEL "bug by bug". No more no less. A fact that has been stressed repeatedly by the maintainers on this list. So admins all over the world trusted this.
Words do have a meaning.
+1
And what is worse: RH are pushing their users to their competitors (read: OL).
RH are pulling their own eyes out... It's a shame.
And all that because they decided to stop supporting a quite extensive worldwide amount of orgs and sysadmins who need a safe and dependable production OS that does not cost a fortune, although many of those at some point in time might become RH support customers!
Now RH earned discontent and distrust from a large part of their FOSS community.
Such a sad end for CentOS... (In fact it is an end indeed.)
OL, Rocky Linux (when fully established) and other such projects (mentioned in various threads) will gain large parts of this extensive group. Some many end up in using CentOS Stream, but the core part of this vibrant community will probably lost by RH and the good old CentOS group (now in RH). Time will show.
That's a pity, because a large and conscious part of this community indeed has some affinity to Karanbir et al, greatly respecting their history and efforts...
RH (& CentOS) still has some small window of opportunity to announce full support of CentOS 8 to its EOL.
Cheers, Nick
On 12/13/20 5:48 AM, Gordon Messmer wrote:
On 12/11/20 9:56 AM, Ljubomir Ljubojevic wrote:
And I will repeat that millions of CentOS users found free clone of RHEL trustworthy enough to use it for production, even without "official endorsement".
Exactly. That's why it's so weird that those people, today, think that CentOS Stream won't be usable, based on their interpretation of the official statements from Red Hat. Red Hat's statements weren't taken into consideration before, but now they're a sign of doom?
Do not turn this upside down. Many obviously smarter then me saw this coming. Us more gullible believed when corporation told us nothing will change, while they took control of entire project. Death by thousand cuts is always more preferable by corporations, less bad PR. And they are not sign of doom but death, but of only this particular clone, others will take it's mantle. The reason I an agitated is I believed and with all my hearth supported this project, and now is owned by heartless profit-driven corporation....
If they ... even allowed ANYONE ELSE that was not employed by Red Hat in 2014 to even come close to learning the secrets of rebuild, no backlash would have happened
I'm going to stop you there, because the CentOS maintainers kept that process out of public visibility long before Red Hat was ever involved. If you think users should know more about the process, then you are pointing fingers at the *wrong* people.
I don't want this to become a flame war. So rather than pointing fingers, let's focus on the fact that CentOS Stream promises to be developed in the open, resolving the problem that you're describing.
Red Hat is fixing the thing you're complaining about.
Don't be silly. They wanted to control the process, and to prevent anyone else from cutting into their process. You should read their manifesto for stream, only RH employees will be able to change that code, others will only be able to report bugs/issues and to sit and watch what RH does with them. Only real difference with stream is for those few hundred developers that are developing software that runs on RHEL so their code can be deployed as soon as new point release is launched.
And even that RH does more for it's own gain, so that (opensurce?) projects/software important to RH can be deployed without delay, lest users ditch RH and go elsewhere.
Red Hat is giving us the thing that has been requested more often, by more people, than any other change in CentOS, and the result is that the press is full of stories about users being angry, because five people on the mailing lists sent a lot of messages. (About half of the traffic in the threads on centos and centos-devel comes from five people, and various people replying to them.)
When people are happy with something they do not voice their content on the mailing list, mailing list is only to voice your discontent. You heard about "silent majority", right? Ever though why it is called that?
But no, as soon as Oracle started rebuilding RHEL source code Red Hat first made things difficult for everyone to create kernels (source code was not srpms anymore but tar?)
You're misinformed. Kernels are still built from SRPM, but the archive used is no longer an upstream release with a series of patches.
Not misinformed, I was part of that discussion 9 years ago, but my memory was little rusty, but I was part of that discussion and It did not bother be to track down one of the comments about kernel tarballs being published as monolith code vs trackable patches, to make problems for builders of RHEL clones:
"There is the kernel tarball issue. That's understandable, but real. Red Hat is publishing their kernels as tarballs, rather than as the vanilla tarball with a list of ordered patches against it. This makes layering in, or out, specific patches much more awkward. Was Oracle being any better about this? Does anyone have actual pointers to their SRPM's?" (https://lists.centos.org/pipermail/centos-devel/2011-March/063442.html)
The reason for the change is not insidious. It's unfortunate that the pristine source + patches can't be maintained, I agree, but speaking as a developer: maintaining hundreds of patches that touch intersecting files and rebasing them all when earlier patches are updated is an incredibly difficult and time consuming task. And, if I remember correctly, applying all of those patches took almost as long as building the kernel. If it takes that long to just prepare the source code, that's a major hit to productivity when a developer needs to work on the code or build the SRPM to validate changes.
And, ultimately, there's very little value in shipping those patches when the vast majority of them are already in the current version of the upstream kernel, and they're merely backported to the older release that Red Hat supports. It's an entirely different story when distributions are shipping patches that they don't push upstream, but that's not generally what you see with the kernel package.
Point is that RH DID slow down build of clones due to this change, and for several months. I can not say about intent, but in hindsight it fits the pattern, I even thought the same back then (mail in link was direct reply to my mail).
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 12/13/20 2:45 AM, Ljubomir Ljubojevic wrote:
When people are happy with something they do not voice their content on the mailing list, mailing list is only to voice your discontent. You heard about "silent majority", right? Ever though why it is called that?
So, the majority of users are silent, because they're happy? Cool.
Point is that RH DID slow down build of clones due to this change
Your memory is still rusty. Early accusations were that this would impact developers (such as Oracle) who were adding additional patches to the kernel, or other maintenance. It never impacted "clones" like CentOS at all.
On Sun, Dec 13, 2020 at 11:56:05AM -0800, Gordon Messmer wrote:
Your memory is still rusty. Early accusations were that this would impact developers (such as Oracle) who were adding additional patches to the kernel, or other maintenance. It never impacted "clones" like CentOS at all.
And you're incorrect; CentOS publishes a centosplus kernel that was impacted to some extent by the kernel tarball changes I believe.
There are no absolutes in life, saying "never impacted" is almost surely untrue for at least one group of people.
John
On 12/13/20 8:56 PM, Gordon Messmer wrote:
On 12/13/20 2:45 AM, Ljubomir Ljubojevic wrote:
When people are happy with something they do not voice their content on the mailing list, mailing list is only to voice your discontent. You heard about "silent majority", right? Ever though why it is called that?
So, the majority of users are silent, because they're happy? Cool.
HAHAHAHAHA, what a wonderful imaginary world you live in :-D 530 negative commends on the blog, 7.800 signed the petition, 100+ negative mails on the CentOS mailing list, and at least 200 negative comments only in CentOS Facebook group, not counting comments in other 20-30 groups.
So yeah, lets go with only 5 complainers :-D
So long, there is nothing more to be said on this topic, except that I will be soon leaving Facebook group admin team.
Point is that RH DID slow down build of clones due to this change
Your memory is still rusty. Early accusations were that this would impact developers (such as Oracle) who were adding additional patches to the kernel, or other maintenance. It never impacted "clones" like CentOS at all.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 12/13/20 1:32 PM, Ljubomir Ljubojevic wrote:
On 12/13/20 8:56 PM, Gordon Messmer wrote:
On 12/13/20 2:45 AM, Ljubomir Ljubojevic wrote:
When people are happy with something they do not voice their content on the mailing list, mailing list is only to voice your discontent. You heard about "silent majority", right? Ever though why it is called that?
So, the majority of users are silent, because they're happy? Cool.
HAHAHAHAHA, what a wonderful imaginary world you live in:-D
I'm just trying to determine whether you were making the argument you intended to, because you are literally suggesting that the majority is silent, and the people who are silent are the ones that are happy with something.
I'm just trying to determine whether you were making the argument you intended to, because you are literally suggesting that the majority is silent, and the people who are silent are the ones that are happy with something.
Silence doesn't confer anything. People can be displeased, cautiously optimistic, and silent, all at the same time.
On Sun, 13 Dec 2020 at 19:56, Gordon Messmer gordon.messmer@gmail.com wrote:
So, the majority of users are silent, because they're happy? Cool.
Can't say I'm really appreciating the trolling in this list.
On Sun, Dec 13, 2020 at 10:06:50PM +0000, Simon Avery wrote:
Can't say I'm really appreciating the trolling in this list.
It's to be expected. What I find surprising is the shilling for RH and this decision that I am seeing. Oh well, such is life.
On 14/12/20 6:56 am, Gordon Messmer wrote:
On 12/13/20 2:45 AM, Ljubomir Ljubojevic wrote:
When people are happy with something they do not voice their content on the mailing list, mailing list is only to voice your discontent. You heard about "silent majority", right? Ever though why it is called that?
So, the majority of users are silent, because they're happy? Cool.
Not because they are happy, but rather because they've most likely moved on. The best protest is carried by moving feet.
I ditched CEntOS for Uuntu back in 2016 and haven't looked back. I only have one last machine still running CEntOS - the firewall. When that EOL's, mine will be a 100% Ubuntu shop. But, not knowing what would happen to Canonical in the future, I've also started toying with Arch and FreeBSD...
On Sat, Dec 12, 2020 at 11:48 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/11/20 9:56 AM, Ljubomir Ljubojevic wrote:
And I will repeat that millions of CentOS users found free clone of RHEL trustworthy enough to use it for production, even without "official endorsement".
Exactly. That's why it's so weird that those people, today, think that CentOS Stream won't be usable, based on their interpretation of the official statements from Red Hat. Red Hat's statements weren't taken into consideration before, but now they're a sign of doom?
If they ... even allowed ANYONE ELSE that was not employed by Red Hat in 2014 to even come close to learning the secrets of rebuild, no backlash would have happened
I'm going to stop you there, because the CentOS maintainers kept that process out of public visibility long before Red Hat was ever involved. If you think users should know more about the process, then you are pointing fingers at the *wrong* people.
I don't want this to become a flame war. So rather than pointing fingers, let's focus on the fact that CentOS Stream promises to be developed in the open, resolving the problem that you're describing.
Red Hat is fixing the thing you're complaining about.
Red Hat is giving us the thing that has been requested more often, by more people, than any other change in CentOS, and the result is that the press is full of stories about users being angry, because five people on the mailing lists sent a lot of messages. (About half of the traffic in the threads on centos and centos-devel comes from five people, and various people replying to them.)
As one of those "five people" I assure you, this is not just a few angry voices. If you, or anyone at Red Hat believe this is the case, you are very sadly mistaken.
Here is the problem: When IBM took over Red Hat, and hence CentOS, these words were posted on the CentOS Blog:
"What does this mean for Red Hat’s contributions to the CentOS project?
In short, nothing.
Red Hat always has and will continue to be a champion for open source and projects like CentOS. IBM is committed to Red Hat’s independence and role in open source software communities so that we can continue this work without interruption or changes.
Our mission, governance, and objectives remain the same. We will continue to execute the existing project roadmap."
This was *last year*. (CF https://blog.centos.org/2019/07/ibm-red-hat-and-centos/) Note the last sentence. The roadmap then had CentOS 8 supported through May 2029.
The simple fact is Red Hat reneged on a promise that hordes of us believed and made a lot of plans on. It is now going to be very expensive, and stress inducing to have to completely rethink everything we have done, and are doing.
You damn right we are angry.
And there's *a lot* more than five of us.
But no, as soon as Oracle started rebuilding RHEL source code Red Hat first made things difficult for everyone to create kernels (source code was not srpms anymore but tar?)
You're misinformed. Kernels are still built from SRPM, but the archive used is no longer an upstream release with a series of patches.
The reason for the change is not insidious. It's unfortunate that the pristine source + patches can't be maintained, I agree, but speaking as a developer: maintaining hundreds of patches that touch intersecting files and rebasing them all when earlier patches are updated is an incredibly difficult and time consuming task. And, if I remember correctly, applying all of those patches took almost as long as building the kernel. If it takes that long to just prepare the source code, that's a major hit to productivity when a developer needs to work on the code or build the SRPM to validate changes.
And, ultimately, there's very little value in shipping those patches when the vast majority of them are already in the current version of the upstream kernel, and they're merely backported to the older release that Red Hat supports. It's an entirely different story when distributions are shipping patches that they don't push upstream, but that's not generally what you see with the kernel package.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Am 13.12.2020 um 19:53 schrieb Phelps, Matthew mphelps@cfa.harvard.edu:
On Sat, Dec 12, 2020 at 11:48 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 12/11/20 9:56 AM, Ljubomir Ljubojevic wrote: And I will repeat that millions of CentOS users found free clone of RHEL trustworthy enough to use it for production, even without "official endorsement".
Exactly. That's why it's so weird that those people, today, think that CentOS Stream won't be usable, based on their interpretation of the official statements from Red Hat. Red Hat's statements weren't taken into consideration before, but now they're a sign of doom?
If they ... even allowed ANYONE ELSE that was not employed by Red Hat in 2014 to even come close to learning the secrets of rebuild, no backlash would have happened
I'm going to stop you there, because the CentOS maintainers kept that process out of public visibility long before Red Hat was ever involved. If you think users should know more about the process, then you are pointing fingers at the *wrong* people.
I don't want this to become a flame war. So rather than pointing fingers, let's focus on the fact that CentOS Stream promises to be developed in the open, resolving the problem that you're describing.
Red Hat is fixing the thing you're complaining about.
Red Hat is giving us the thing that has been requested more often, by more people, than any other change in CentOS, and the result is that the press is full of stories about users being angry, because five people on the mailing lists sent a lot of messages. (About half of the traffic in the threads on centos and centos-devel comes from five people, and various people replying to them.)
As one of those "five people" I assure you, this is not just a few angry voices. If you, or anyone at Red Hat believe this is the case, you are very sadly mistaken.
Here is the problem: When IBM took over Red Hat, and hence CentOS, these words were posted on the CentOS Blog:
"What does this mean for Red Hat’s contributions to the CentOS project?
In short, nothing.
Red Hat always has and will continue to be a champion for open source and projects like CentOS. IBM is committed to Red Hat’s independence and role in open source software communities so that we can continue this work without interruption or changes.
Our mission, governance, and objectives remain the same. We will continue to execute the existing project roadmap."
This was *last year*. (CF https://blog.centos.org/2019/07/ibm-red-hat-and-centos/) Note the last sentence. The roadmap then had CentOS 8 supported through May 2029.
The simple fact is Red Hat reneged on a promise that hordes of us believed and made a lot of plans on. It is now going to be very expensive, and stress inducing to have to completely rethink everything we have done, and are doing.
You damn right we are angry.
And there's *a lot* more than five of us.
Here is number six.
But no, as soon as Oracle started rebuilding RHEL source code Red Hat first made things difficult for everyone to create kernels (source code was not srpms anymore but tar?)
You're misinformed. Kernels are still built from SRPM, but the archive used is no longer an upstream release with a series of patches.
The reason for the change is not insidious. It's unfortunate that the pristine source + patches can't be maintained, I agree, but speaking as a developer: maintaining hundreds of patches that touch intersecting files and rebasing them all when earlier patches are updated is an incredibly difficult and time consuming task. And, if I remember correctly, applying all of those patches took almost as long as building the kernel. If it takes that long to just prepare the source code, that's a major hit to productivity when a developer needs to work on the code or build the SRPM to validate changes.
And, ultimately, there's very little value in shipping those patches when the vast majority of them are already in the current version of the upstream kernel, and they're merely backported to the older release that Red Hat supports. It's an entirely different story when distributions are shipping patches that they don't push upstream, but that's not generally what you see with the kernel package.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
--
*Matt Phelps*
*Information Technology Specialist, Systems Administrator*
(Computation Facility, Smithsonian Astrophysical Observatory)
Center for Astrophysics | Harvard & Smithsonian
60 Garden Street | MS 39 | Cambridge, MA 02138 email: mphelps@cfa.harvard.edu
cfa.harvard.edu | Facebook http://cfa.harvard.edu/facebook | Twitter http://cfa.harvard.edu/twitter | YouTube http://cfa.harvard.edu/youtube | Newsletter http://cfa.harvard.edu/newsletter _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
And there's *a lot* more than five of us.
Here is number six.
Just one of those groups energised from this decision is Rocky Linux. There are 4,606 people on their Slack right now, which did not even exist a week ago.
Yeah, it's more than five.
Am 13.12.2020 um 20:44 schrieb Simon Avery digdilem@gmail.com:
And there's *a lot* more than five of us.
Here is number six.
Just one of those groups energised from this decision is Rocky Linux. There are 4,606 people on their Slack right now, which did not even exist a week ago.
IIRC, one of the reasons cited that CentOS „merged“ with RedHat back then was that a lot of people were using CentOS, but there wasn’t enough money generated to pay the developers.
A lot of them were basically working for free.
That is never sustainable. At least not for a long time.
It’s also not often the case that you can split this kind of work into a thousand work-packages and have everybody just work 1/2 hour a day on it.
On Sun, 13 Dec 2020 21:05:42 +0100 Rainer Duffner rainer@ultra-secure.de wrote:
It’s also not often the case that you can split this kind of work into a thousand work-packages and have everybody just work 1/2 hour a day on it.
not like Debian for instance
d
On 12/13/20 1:25 PM, Dave Stevens wrote:
On Sun, 13 Dec 2020 21:05:42 +0100 Rainer Duffner rainer@ultra-secure.de wrote:
It’s also not often the case that you can split this kind of work into a thousand work-packages and have everybody just work 1/2 hour a day on it.
not like Debian for instance
d
The workflow is very different. For a primary distribution, updates to different packages happen at different times. Contributors can do that work when they have the time.
For a rebuild, work must happen as fast as possible after RHEL has released an update. Much harder for volunteers to contribute to.
There are other support roles that volunteers can hopefully do, but the core mission doesn't really align well with that.
On 12/13/20 3:25 PM, Dave Stevens wrote:
On Sun, 13 Dec 2020 21:05:42 +0100 Rainer Duffner rainer@ultra-secure.de wrote:
It’s also not often the case that you can split this kind of work into a thousand work-packages and have everybody just work 1/2 hour a day on it.
not like Debian for instance
No, not at all like Debian. Debian doesn't have to try to match the unattainable goal of 100% binary compatibility with an upstream source. I've seen a small part of this first-hand, and deducing the build order to gain binary compatibility is the one thing that can single-thread the build process quicker than anything else. RHEL doesn't even have the same need; an RHEL rebuild that didn't have the goal to be bug-compatible near the 100% level doesn't, either, and can be built by a lot of people.
Try it yourself: go back to CentOS 5.5 and attempt to rebuild the released sources for 5.6 and get a binary compatible build. I've done it myself for IA64; it was a pain.
All of the upstream distributions, Debian, Fedora, etc, have a lot of latitude that CentOS never has enjoyed.
On 12/17/2020 9:29 AM, Lamar Owen wrote:
On 12/13/20 3:25 PM, Dave Stevens wrote:
On Sun, 13 Dec 2020 21:05:42 +0100 Rainer Duffner rainer@ultra-secure.de wrote:
It’s also not often the case that you can split this kind of work into a thousand work-packages and have everybody just work 1/2 hour a day on it.
not like Debian for instance
No, not at all like Debian. Debian doesn't have to try to match the unattainable goal of 100% binary compatibility with an upstream source. I've seen a small part of this first-hand, and deducing the build order to gain binary compatibility is the one thing that can single-thread the build process quicker than anything else. RHEL doesn't even have the same need; an RHEL rebuild that didn't have the goal to be bug-compatible near the 100% level doesn't, either, and can be built by a lot of people.
Try it yourself: go back to CentOS 5.5 and attempt to rebuild the released sources for 5.6 and get a binary compatible build. I've done it myself for IA64; it was a pain.
All of the upstream distributions, Debian, Fedora, etc, have a lot of latitude that CentOS never has enjoyed.
Given that rebuilds /had/ been taking longer and longer lengths of time, it was clearly hop(e)xpected that bringing CentOS into the RedHat fold in an official partnership in 2014 would allow for direct access to the RHEL engineers and a path toward making the CentOS Project's rebuilds easier. Indeed, although it's been six years, moving to integrating EL/ELN/CS and RHEL in some way in the pipeline is precisely what had been desired... BUT, defining the problem out of existence while also defining "CentOS Linux" out of existence helps absolutely no one (on the outside, at least). And cutting the support period for 8 in half is so disruptive as to outweigh the progress in other areas.
-jc