Hi,
I am a grown-up kid :p.
As a Christmas gift from CentOS, we would like CentOS core developers to show us how they build a release from RedHat sources whenever RedHat releases a version or an rpm.
The documentation, scripts, and setup are what we are looking for.
This is not for me personally, but for people interested in cloning CentOS.
Christmas cheers :)
Thanks
--- Lee
On Wed, 16 Dec 2020 at 00:30, Thomas Stephen Lee lee.iitb@gmail.com wrote:
Hi,
I am a grown-up kid :p.
As a Christmas gift from CentOS, we would like CentOS core developers to show us how they build a release from RedHat sources whenever RedHat releases a version or an rpm.
The documentation, scripts, and setup are what we are looking for.
https://github.com/centos has all the ansible scripts used to set up the infrastructure. https://git.centos.org/projects/centos/ contains most of everything else. https://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional
Also on the wiki are a trove of various snippets of documents etc.
Now for documentation.. I don't think there is anything beyond that and has never been. Each person knows their job and does it. When they go out on break.. things tend to slow down for a reason. Making a cloned operating system usually requires a lot of concentration on why did it break this time when you did exactly all the same voodoo you did last time..
From my own study, basically you need to reinvent via guesswork what was
done inside Red Hat step by step. Did they have to compile rust1 then rust2 then rust3 to get thunderbird to work? Did they have to compile rust3 then rust1 then rust2 then rust3? Or some other combinatoric list.
Then we have modularity.. which tells you all that in its yaml files but needs an entire special infrastructure to do it.
This is not for me personally, but for people interested in cloning CentOS.
Christmas cheers :)
Thanks
Lee _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
Hi,
When it comes to modules the situation is clusterfu**.
That reason why making EL8 took us so long. I'm quite sure that without internal knowledge of modules (that only Fedora uses - and documentation sucks) amount of reverse engineering is tremendous. The best place to start is https://pagure.io/fm-orchestrator.
RHEL sources based/CentOS clones will also have different platform string, that will make the whole system reinstall itself xDDDDDDD. That's a reason why migrating to these solutions will be a pain in the neck.
Going back to src.rpms. If you are looking for src.rpms best bet is using original RHEL src.rpms, these are readily available when you have even minimal subscription. It's the much safer bet for long term system.
Imagine the following:
- RHEL made errata for package X version n release m. - CentOS stream have package X version n release m+10.
Which one will be available on git.centos.org?
Using git.centos.org is a bad idea for next gen rebuilds IMO.
Bests, Alex
On 12/16/20 12:13 PM, Stephen John Smoogen wrote:
On Wed, 16 Dec 2020 at 00:30, Thomas Stephen Lee <lee.iitb@gmail.com mailto:lee.iitb@gmail.com> wrote:
Hi, I am a grown-up kid :p. As a Christmas gift from CentOS, we would like CentOS core developers to show us how they build a release from RedHat sources whenever RedHat releases a version or an rpm. The documentation, scripts, and setup are what we are looking for.https://github.com/centos https://github.com/centos has all the ansible scripts used to set up the infrastructure. https://git.centos.org/projects/centos/ https://git.centos.org/projects/centos/ contains most of everything else. https://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional https://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional
Also on the wiki are a trove of various snippets of documents etc.
Now for documentation.. I don't think there is anything beyond that and has never been. Each person knows their job and does it. When they go out on break.. things tend to slow down for a reason. Making a cloned operating system usually requires a lot of concentration on why did it break this time when you did exactly all the same voodoo you did last time..
From my own study, basically you need to reinvent via guesswork what was done inside Red Hat step by step. Did they have to compile rust1 then rust2 then rust3 to get thunderbird to work? Did they have to compile rust3 then rust1 then rust2 then rust3? Or some other combinatoric list.
Then we have modularity.. which tells you all that in its yaml files but needs an entire special infrastructure to do it.
This is not for me personally, but for people interested in cloning CentOS. Christmas cheers :) Thanks --- Lee _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org <mailto:CentOS-devel@centos.org> https://lists.centos.org/mailman/listinfo/centos-devel <https://lists.centos.org/mailman/listinfo/centos-devel>-- Stephen J Smoogen.
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On Wed, 16 Dec 2020 at 06:46, aleksander.baranowski via CentOS-devel < centos-devel@centos.org> wrote:
Hi,
When it comes to modules the situation is clusterfu**.
That reason why making EL8 took us so long. I'm quite sure that without internal knowledge of modules (that only Fedora uses - and documentation sucks) amount of reverse engineering is tremendous. The best place to start is https://pagure.io/fm-orchestrator.
RHEL sources based/CentOS clones will also have different platform string, that will make the whole system reinstall itself xDDDDDDD. That's a reason why migrating to these solutions will be a pain in the neck.
Going back to src.rpms. If you are looking for src.rpms best bet is using original RHEL src.rpms, these are readily available when you have even minimal subscription. It's the much safer bet for long term system.
Imagine the following:
- RHEL made errata for package X version n release m.
- CentOS stream have package X version n release m+10.
Which one will be available on git.centos.org?
Both will be available on git.centos.org. All of the source code from RHEL-7 onward has been pushed to there... including buildroot only source code.
Using git.centos.org is a bad idea for next gen rebuilds IMO.
I am going to strongly recommend against this for 3 reasons: 1. It is a contractual problem to do this. [AKA Get a lawyer. I am not one and I am not going to try to argue it anymore than I am going to argue Rust language syntax I don't know. I will just say that two wrongs do not make a right.] 2. The src.rpms are not debranded. That is extra work a rebuilder has to do. The git source seems to be debranded by Red Hat. 3. Not all the src.rpms may be shipped with the developer etc. Red Hat Enterprise Linux 8 is not self-hosting. You need extra packages to build things.. the source for those should be on git.centos.org
That is all I am going to say on this.
On Wed, 16 Dec 2020 at 06:46, aleksander.baranowski via CentOS-devel < centos-devel@centos.org> wrote:
Hi,
When it comes to modules the situation is clusterfu**.
That reason why making EL8 took us so long. I'm quite sure that without internal knowledge of modules (that only Fedora uses - and documentation sucks) amount of reverse engineering is tremendous. The best place to start is https://pagure.io/fm-orchestrator.
RHEL sources based/CentOS clones will also have different platform string, that will make the whole system reinstall itself xDDDDDDD. That's a reason why migrating to these solutions will be a pain in the neck.
Going back to src.rpms. If you are looking for src.rpms best bet is using original RHEL src.rpms, these are readily available when you have even minimal subscription. It's the much safer bet for long term system.
Imagine the following:
- RHEL made errata for package X version n release m.
- CentOS stream have package X version n release m+10.
Which one will be available on git.centos.org?
Both will be available on git.centos.org. All of the source code from RHEL-7 onward has been pushed to there... including buildroot only source code.
Using git.centos.org is a bad idea for next gen rebuilds IMO.
I am going to strongly recommend against this for 3 reasons:
- It is a contractual problem to do this. [AKA Get a lawyer. I am not one
and I am not going to try to argue it anymore than I am going to argue Rust language syntax I don't know. I will just say that two wrongs do not make a right.] 2. The src.rpms are not debranded. That is extra work a rebuilder has to do. The git source seems to be debranded by Red Hat. 3. Not all the src.rpms may be shipped with the developer etc. Red Hat Enterprise Linux 8 is not self-hosting. You need extra packages to build
That's an interesting point which is, from a technical POV, really not nice. Just imagine an enterprise Linux distribution which was a) self hosting, and b) provides reproducible builds...
Simon
On 12/16/20 9:00 AM, Simon Matter wrote:
That's an interesting point which is, from a technical POV, really not nice. Just imagine an enterprise Linux distribution which was a) self hosting, and b) provides reproducible builds...
I've used Red Hat long enough to remember the old heavily customized Beehive builder. The mach and then mock chroot build systems were vast improvements when they became available, but there are problems even there.
Much depends upon build order. I experienced one instance of build order issues a few years back (2012 or 2013 or thereabouts) as I was rebuilding CentOS 5.6 for IA64 for an SGI Altix 350 system we have here at $dayjob. I remember all the issues on this list related to the concurrent lateness of CentOS 5.6 and 6.0, and I found out a piece of what caused 5.6 to take more time. There was one particular library (I don't remember which one right off, although I could probably go back and look at some point, but not today) that got an upgrade in 5.6; some upgraded packages in 5.6 needed the previous version of that library to build, and some other packages needed the upgraded library.
That library had to be rebuilt at a very specific place in the package build sequence for those other packages to even build correctly, and the buildrequires, if I remember correctly, were not versioned (that, in my opinion, is a packaging error, but at the same time I remember the day when versioned buildrequires didn't work; for that matter, do they work correctly now?). And there are certain core build dependencies that aren't called out in buildrequires anyway.
Now, this particular issue, once I found the package failing to rebuild, due to other dependencies involved I actually had to start the rebuild of the whole set of source packages over from scratch; I thought I was almost done, but in reality I had another full week of rebuilding to go (I was rebuilding on the Altix 350; it took a really long time to build some packages, and I think the full 5.6 build took about a month of machine time); the Altix was designed for compute performance but not I/O performance. This particular build order problem wasted over a week of machine time building packages.
There were other issues in rebuilding 5.6, but that's the one I ran up against and remember, and since I wasn't aiming for a redistributable binary-compatible-to-RHEL IA64 rebuild I didn't aim for the 100% binary compatibility (which does NOT mean identical binaries; it means identical versions of all required libraries and ABIs). The actual CentOS 5.6 build was I'm sure much much more difficult, since the CentOS team do far more testing that what I did; I just barely scratched the surface of how difficult that time in CentOS history really was (6.0, 5.6, and 4.10 all hit close together, if I remember correctly).
When the source distribution changed to git.centos.org that made it more difficult to ferret this build order out. Source RPMs contain the build timestamp; git.centos.org sources do not. But it can get more complicated; there have been instances, I believe, where a released package was built with a buildroot that contained unreleased versions of libraries; the released version of those libraries could be different. From a strict binary compatibility standpoint, any version bump of any required library breaks 100% compatibility. That's the second edge of the two-edged sword called shared libraries.
Yes, an enterprise Linux that is fully self-hosting and with reproducible builds is a laudable goal; is it a feasible goal, from a business point of view? I would say no, myself, since it provides no real return on investment. As the package set gets larger, the amount of work to verify self-hosting becomes that much more difficult, and costly.
BUT, in this one particular regard CentOS Stream will be far superior; you're not getting a big dump of packages or source commits to git.centos.org where the build order is unknown; with Stream you get the build order live, and that solves that particular problem (while creating a different problem, but I've already posted about kernel driver ABI fun). The tradeoff is better transparency at the cost of less binary compatibility (it's not to the level of Fedora, or Rawhide, or being a beta; it's just not what most CentOS users want, free RHEL with all features turned on; but did we ever really have that?). But as far as administration goes, CentOS Stream should be essentially identical to RHEL.
-
aleksander.baranowski -
Lamar Owen -
Simon Matter -
Stephen John Smoogen -
Thomas Stephen Lee