Hi
I've noticed that libssh inexplicably vanished from the ppc64le C7 extras repository, c.f., http://mirror.centos.org/altarch/7.4.1708/extras/ppc64/Packages/ .
The weird part is that this package is still available for ppc64le on http://mirror.centos.org/altarch/7.4.1708/extras/ppc64le/Packages/
The first time I noticed this was on Dec 10, though it might have been deleted earlier.
Is there any reason why this happened? I assume that libssh was updated and a rebuild for all arches attempted, but the ppc64 builder might have either had general problems or the build failed in some other way. Would fit in (kinda) with the outage notice from Dec 2nd.
I've tried tracking where "extras" builds are taking place. CBS seems to be a reasonable place for that, but the information for libssh seems to be outdated as per https://cbs.centos.org/koji/packageinfo?packageID=51
This page only lists rev 2, while the x86_64 and ppc64le repos both feature a package with rev 3, though.
Given that the ppc64le packages don't have a .centos suffix, am I right to assume that such packages are built internally by Red Hat and that this infrastructure is not transparent to users?
In any case, I'd appreciate a rebuild on ppc64.
Mihai
On 13/12/17 10:11, Mihai Moldovan wrote:
Hi
I've noticed that libssh inexplicably vanished from the ppc64le C7 extras repository, c.f., http://mirror.centos.org/altarch/7.4.1708/extras/ppc64/Packages/ .
The weird part is that this package is still available for ppc64le on http://mirror.centos.org/altarch/7.4.1708/extras/ppc64le/Packages/
The first time I noticed this was on Dec 10, though it might have been deleted earlier.
Is there any reason why this happened? I assume that libssh was updated and a rebuild for all arches attempted, but the ppc64 builder might have either had general problems or the build failed in some other way. Would fit in (kinda) with the outage notice from Dec 2nd.
I've tried tracking where "extras" builds are taking place. CBS seems to be a reasonable place for that, but the information for libssh seems to be outdated as per https://cbs.centos.org/koji/packageinfo?packageID=51
This page only lists rev 2, while the x86_64 and ppc64le repos both feature a package with rev 3, though.
Given that the ppc64le packages don't have a .centos suffix, am I right to assume that such packages are built internally by Red Hat and that this infrastructure is not transparent to users?
In any case, I'd appreciate a rebuild on ppc64.
The first thing to consider would be the Extras repository : normally, packages rebuilt and pushed in that repository are rebuilt pkgs that landed in "upstream" Extras repo. AFAIK (but happy to be corrected if I'm wrong) there is no ppc64 Extras channel.
Now we can though try to rebuild that libssh and push it through Extras if that's possible
WRT "infrastructure" : CBS is *not* used at all to build distribution components, so don't search there for packages that land in os/updates/extras repository. Those packages are all rebuilt in a different setup.
Let's try to sync with James (ppc64/ppc64le maintainer) to see what can be done for that libssh pkg.
* On 12/13/2017 01:54 PM, Fabian Arrotin wrote:
The first thing to consider would be the Extras repository : normally, packages rebuilt and pushed in that repository are rebuilt pkgs that landed in "upstream" Extras repo. AFAIK (but happy to be corrected if I'm wrong) there is no ppc64 Extras channel.
I cannot tell. The only thing I know for sure is that there were libssh ppc64 packages prior to (likely) Dec 7.
Now we can though try to rebuild that libssh and push it through Extras if that's possible
That would be nice!
WRT "infrastructure" : CBS is *not* used at all to build distribution components, so don't search there for packages that land in os/updates/extras repository. Those packages are all rebuilt in a different setup.
I suspected that since most packages are just copied directly from RHEL without a further rebuild, the building infrastructure might not be publicly available.
Nothing I could look up to figure out what happened...
Let's try to sync with James (ppc64/ppc64le maintainer) to see what can be done for that libssh pkg.
Thanks.
Mihai
Since my initial report, either even more packages mysteriously vanished or I simply didn't notice the highly diverging number of packages in the ppc64 and ppc64le repositories.
However, as of right now, both repositories have only 24 packages in common.
274(!) packages are missing from the ppc64 repository but are available in the ppc64le repository.
Not all of these packages are actually unique packages. Instead, some are merely older versions of the same package (for instance, ansiable has four packages with three being outdated versions).
The trend, though, seems to be that any package that has more recent builds than Dec 07 is deleted from the ppc64 repository (including old versions), but the newer version never makes it into this repository.
My best guesses currently are that either deletion works, but pushing packages to this repository is broken for some reason, or that the package builds failed (which seems unlikely, unless your systems delete older packages automatically without waiting for a successful new build).
Something definitely looks odd there.
Mihai
On 15/12/17 04:52, Mihai Moldovan wrote:
Since my initial report, either even more packages mysteriously vanished or I simply didn't notice the highly diverging number of packages in the ppc64 and ppc64le repositories.
However, as of right now, both repositories have only 24 packages in common.
274(!) packages are missing from the ppc64 repository but are available in the ppc64le repository.
So let's dive into that assumption : when you say that pkgs are missing from the Extras repository, are you using the upstream Extras repository to compare against ? As said in previous mail, I don't think that there is an Extras channel for ppc64 (but that exists for ppc64le) but again, that's something also the ppc64/ppc64le maintainer would have to answer.
BTW if the CentOS Extras policy should indeed be (and we tried to have a rule enforcement during CBS meetings about this, but no consensus) to only include packages from upstream Extras repository, then my understanding is that we 'd even have to delete packages there not in the upstream extras
But that would not be possible, as that would also mean removing all centos pkgs like centos-release-* for the SIGs content
For the record of that libssh pkg for ppc64, it was only included in 7.2.1511 os repo for ppc64 (we have a daily snapshot of mirror.centos.org content) :
/20151216/altarch/7.2.1511/os/ppc64/Packages/libssh-0.6.4-4.el7.ppc64.rpm
but nothing after that, and even in the base distro it was replaced by libssh2, which is still there : http://mirror.centos.org/altarch/7.4.1708/os/ppc64/Packages/libssh2-1.4.3-10...
So I guess that some pkgs from upstream Extras repo were still needed the previous libssh pkg, so rebuilt in the Extras repo instead, but once again, something we *never* had (so no, it didn't "vanish" from the ppc64 repo)
On 15/12/17 09:47, Fabian Arrotin wrote:
On 15/12/17 04:52, Mihai Moldovan wrote:
Since my initial report, either even more packages mysteriously vanished or I simply didn't notice the highly diverging number of packages in the ppc64 and ppc64le repositories.
However, as of right now, both repositories have only 24 packages in common.
274(!) packages are missing from the ppc64 repository but are available in the ppc64le repository.
So let's dive into that assumption : when you say that pkgs are missing from the Extras repository, are you using the upstream Extras repository to compare against ? As said in previous mail, I don't think that there is an Extras channel for ppc64 (but that exists for ppc64le) but again, that's something also the ppc64/ppc64le maintainer would have to answer.
BTW if the CentOS Extras policy should indeed be (and we tried to have a rule enforcement during CBS meetings about this, but no consensus) to only include packages from upstream Extras repository, then my understanding is that we 'd even have to delete packages there not in the upstream extras
But that would not be possible, as that would also mean removing all centos pkgs like centos-release-* for the SIGs content
For the record of that libssh pkg for ppc64, it was only included in 7.2.1511 os repo for ppc64 (we have a daily snapshot of mirror.centos.org content) :
/20151216/altarch/7.2.1511/os/ppc64/Packages/libssh-0.6.4-4.el7.ppc64.rpm
but nothing after that, and even in the base distro it was replaced by libssh2, which is still there : http://mirror.centos.org/altarch/7.4.1708/os/ppc64/Packages/libssh2-1.4.3-10...
So I guess that some pkgs from upstream Extras repo were still needed the previous libssh pkg, so rebuilt in the Extras repo instead, but once again, something we *never* had (so no, it didn't "vanish" from the ppc64 repo)
Just for people reading it, but not following discussions in #centos-devel :
We confirmed with Mihai that we never shipped that libssh pkg in centos extras for ppc64, but that it was initially in /os/ for 7.2.1511.
When that package was removed from /os/ it was then available in Epel, but recently, it seems such libssh pkg was also removed from Epel, because it entered Extras, but only for ppc64le/x86_64 ..
That explains so the confusion, but from my PoV, I guess the pkg can still be either back in Epel for ppc64, or we can try to build it for ppc64, despite the fact that it doensn't exist 'upstream' (lack of Extras repo for ppc64)
On 15 December 2017 at 07:59, Fabian Arrotin arrfab@centos.org wrote:
On 15/12/17 09:47, Fabian Arrotin wrote:
On 15/12/17 04:52, Mihai Moldovan wrote:
Since my initial report, either even more packages mysteriously vanished or I simply didn't notice the highly diverging number of packages in the ppc64 and ppc64le repositories.
However, as of right now, both repositories have only 24 packages in common.
274(!) packages are missing from the ppc64 repository but are available in the ppc64le repository.
So let's dive into that assumption : when you say that pkgs are missing from the Extras repository, are you using the upstream Extras repository to compare against ? As said in previous mail, I don't think that there is an Extras channel for ppc64 (but that exists for ppc64le) but again, that's something also the ppc64/ppc64le maintainer would have to answer.
BTW if the CentOS Extras policy should indeed be (and we tried to have a rule enforcement during CBS meetings about this, but no consensus) to only include packages from upstream Extras repository, then my understanding is that we 'd even have to delete packages there not in the upstream extras
But that would not be possible, as that would also mean removing all centos pkgs like centos-release-* for the SIGs content
For the record of that libssh pkg for ppc64, it was only included in 7.2.1511 os repo for ppc64 (we have a daily snapshot of mirror.centos.org content) :
/20151216/altarch/7.2.1511/os/ppc64/Packages/libssh-0.6.4-4.el7.ppc64.rpm
but nothing after that, and even in the base distro it was replaced by libssh2, which is still there : http://mirror.centos.org/altarch/7.4.1708/os/ppc64/Packages/libssh2-1.4.3-10...
So I guess that some pkgs from upstream Extras repo were still needed the previous libssh pkg, so rebuilt in the Extras repo instead, but once again, something we *never* had (so no, it didn't "vanish" from the ppc64 repo)
Just for people reading it, but not following discussions in #centos-devel :
We confirmed with Mihai that we never shipped that libssh pkg in centos extras for ppc64, but that it was initially in /os/ for 7.2.1511.
When that package was removed from /os/ it was then available in Epel, but recently, it seems such libssh pkg was also removed from Epel, because it entered Extras, but only for ppc64le/x86_64 ..
That explains so the confusion, but from my PoV, I guess the pkg can still be either back in Epel for ppc64, or we can try to build it for ppc64, despite the fact that it doensn't exist 'upstream' (lack of Extras repo for ppc64)
PPC64 is a positive pain in the butt to deal with because various packages will not compile at all, will compile for it but not work correctly (too much software written with the all the worlds little ended), or needing special care because it is in certain architectures but not all. When we put stuff in ppc in EPEL it gets built for all the architectures which means we have to make sure that the version is a lower NEVR from what is in CentOS/RHEL for that version so that we don't replace the upstream version. [it two versions have the same NEVR then yum/dnf will choose whichever one is lower by other heuristics which can make EPEL the one installed when the RHEL one is the one supported.] This is extra steps and work as packages in RHEL get updated more than previously.
I don't have a good solution on the top of my brain.. I am open to suggestions to try and come up with one.
On Dec 13, 2017, at 6:12 PM, Mihai Moldovan ionic@ionic.de wrote:
- On 12/13/2017 01:54 PM, Fabian Arrotin wrote:
The first thing to consider would be the Extras repository : normally, packages rebuilt and pushed in that repository are rebuilt pkgs that landed in "upstream" Extras repo. AFAIK (but happy to be corrected if I'm wrong) there is no ppc64 Extras channel.
I cannot tell. The only thing I know for sure is that there were libssh ppc64 packages prior to (likely) Dec 7.
Now we can though try to rebuild that libssh and push it through Extras if that's possible
That would be nice!
I’ve built libssh-0.7.1-3.el7.ppc64.rpm and libssh-devel-0.7.1-3.el7.ppc64.rpm. They will be in the next ppc64 Extras sign and push.
There is no upstream RHEL 7 Extras for ppc64. There is also no ppc64 golang port which makes it impossible to compile a bunch of the Extras content that exists for x86_64 and ppc64le.
But if you help me identify any C or Python code in ppc64le Extras, I will attempt to build, sign and publish for ppc64.
-James
* On 12/15/2017 09:33 PM, James O'Connor wrote:
I’ve built libssh-0.7.1-3.el7.ppc64.rpm and libssh-devel-0.7.1-3.el7.ppc64.rpm. They will be in the next ppc64 Extras sign and push.
There is no upstream RHEL 7 Extras for ppc64. There is also no ppc64 golang port which makes it impossible to compile a bunch of the Extras content that exists for x86_64 and ppc64le.
Okay, that's good to know.
As previously said, I've been probably barking up the wrong tree. It looks like ppc64 extras never contained libssh or the other python packages - instead they were part of EPEL 7, but the packages were deleted there for some unknown reason.
Maybe because the EPEL maintainers saw that the software is available in the RHEL extras channel (and by extensions, CentOS 7's extras channel) - at least for x86_64 (and incidentally ppc64le).
If RedHat itself has no extras channel for ppc64, I wonder whether CentOS should have one in the first place - given that CentOS' goal is to stay close to RHEL. Maybe it would make sense to drop that repository completely (until RH introduces an extra channel for ppc64) and hope that EPEL will fill the void.
Mihai
On 12/18/2017 03:41 AM, Mihai Moldovan wrote:
- On 12/15/2017 09:33 PM, James O'Connor wrote:
I’ve built libssh-0.7.1-3.el7.ppc64.rpm and libssh-devel-0.7.1-3.el7.ppc64.rpm. They will be in the next ppc64 Extras sign and push.
There is no upstream RHEL 7 Extras for ppc64. There is also no ppc64 golang port which makes it impossible to compile a bunch of the Extras content that exists for x86_64 and ppc64le.
Okay, that's good to know.
As previously said, I've been probably barking up the wrong tree. It looks like ppc64 extras never contained libssh or the other python packages - instead they were part of EPEL 7, but the packages were deleted there for some unknown reason.
Maybe because the EPEL maintainers saw that the software is available in the RHEL extras channel (and by extensions, CentOS 7's extras channel) - at least for x86_64 (and incidentally ppc64le).
If RedHat itself has no extras channel for ppc64, I wonder whether CentOS should have one in the first place - given that CentOS' goal is to stay close to RHEL. Maybe it would make sense to drop that repository completely (until RH introduces an extra channel for ppc64) and hope that EPEL will fill the void.
Well .. extras is also for other things we want to add to CentOS which is not in RHEL and not an upgrade or replacement to things in the base OS. So, if there are things one wants as an 'add on', extras is where it will go.
There is nothing wrong with adding things on top of CentOS Linux. BUT .. major things should be done in a SIG. (https://wiki.centos.org/SpecialInterestGroup). I would not be opposed to building anything in extras for x86_64 (that will build) for the AltArches.
-
Fabian Arrotin -
James O'Connor -
Johnny Hughes -
Mihai Moldovan -
Stephen John Smoogen