On Fri, May 19, 2023 at 7:00 AM Sandro Bonazzola sbonazzo@redhat.com wrote:

Il giorno ven 19 mag 2023 alle ore 15:52 Troy Dawson tdawson@redhat.com ha scritto:

...

Those don't sound like CentOS packages. Centos packages do not have dist tags of .el9s or .el8s Please check the repo's those are coming from and contact the respective repository owners.

Those are from CentOS SIGs, https://cbs.centos.org/koji/buildinfo?buildID=39630 https://cbs.centos.org/koji/buildinfo?buildID=42936

Ahh ... ok. That makes more sense. Then you DID contact the respective repo owners. :)

...

On Fri, May 19, 2023 at 1:48 AM Sandro Bonazzola sbonazzo@redhat.com wrote:

...

Good morning. In oVirt CI we are seeing errors like: [MIRROR] ovn22.09-host-22.09.0-31.el9s.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: 84909c4cb450c019b923f703efa9bddaa4bc070fa4866eea3551c5205ebe287f(sha256) Expected: 234e5bbdbde4426d8236c5f2d1373fb1379ca2d1149da84b5b831a5950a10c2b(sha256)

Similar issue on el8 as well: [MIRROR] openvswitch-selinux-extra-policy-1.0-29.el8s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 3bde93c064d8e1142338e572e3731f733eb1c1e0a06523b9711adefcd6abb716(sha256) Expected: 960a0b5500729627959ecedb74f6e61a1075f082977fe177c7e855c402e81434(sha256)

Perhaps packages got signed but repository metadata not refreshed?

--

Sandro Bonazzola

MANAGER, SOFTWARE ENGINEERING - Red Hat In-Vehicle Operating System

Red Hat EMEA https://www.redhat.com/

sbonazzo@redhat.com https://www.redhat.com/

*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

---

CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

---

CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

--

Sandro Bonazzola

MANAGER, SOFTWARE ENGINEERING - Red Hat In-Vehicle Operating System

Red Hat EMEA https://www.redhat.com/

sbonazzo@redhat.com https://www.redhat.com/

*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

---

CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

On 19/05/2023 15:59, Sandro Bonazzola wrote:

Il giorno ven 19 mag 2023 alle ore 15:52 Troy Dawson <tdawson@redhat.com mailto:tdawson@redhat.com> ha scritto:

Those don't sound like CentOS packages.
Centos packages do not have dist tags of .el9s  or .el8s
Please check the repo's those are coming from and contact the
respective repository owners.

Those are from CentOS SIGs, https://cbs.centos.org/koji/buildinfo?buildID=39630 https://cbs.centos.org/koji/buildinfo?buildID=39630 https://cbs.centos.org/koji/buildinfo?buildID=42936 https://cbs.centos.org/koji/buildinfo?buildID=42936

Can you elaborate where you're seeing that ? you mentioned oVirt CI so outside of centos infra (or outside of CBS) How do you test , from which repo, etc ? It's true that packages in -testing are now signed (it was announced https://lists.centos.org/pipermail/centos-devel/2023-May/142901.html) but is that for -release ? what about mirrormanager ?

On 22/05/2023 08:53, YATIN KAREL wrote:

Hi,

Seeing similar issues in Openstack CI[1].

For example:- $ sudo dnf install --repofrompath=tmp,https://buildlogs.centos.org/centos/9-stream/nfv/x86_64/openvswitch-2/ https://buildlogs.centos.org/centos/9-stream/nfv/x86_64/openvswitch-2/ --enablerepo=tmp --disablerepo=* openvswitch3.1 Added tmp repo from https://buildlogs.centos.org/centos/9-stream/nfv/x86_64/openvswitch-2/ https://buildlogs.centos.org/centos/9-stream/nfv/x86_64/openvswitch-2/ tmp          11 kB/s | 3.0 kB     00:00 Dependencies resolved. ================================================================================================================  Package                                     Architecture      Version                     Repository      Size ================================================================================================================ Installing:  openvswitch3.1                              x86_64  3.1.0-23.el9s               tmp            6.8 M Installing dependencies:  openvswitch-selinux-extra-policy            noarch  1.0-31.el9s                 tmp             14 k

Transaction Summary

Install  2 Packages

Total download size: 6.8 M Installed size: 22 M Is this ok [y/N]: y Downloading Packages: [MIRROR] openvswitch-selinux-extra-policy-1.0-31.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 8e2f3b2089ea33e20e3b2a76190f9facae77772fdb9dd72e9c3d3b426fc2edb9(sha256)  Expected: 17e5200b2c4045b2060e094ff383317f5c55744eb482d91b1ae026d473b83dc2(sha256) [MIRROR] openvswitch-selinux-extra-policy-1.0-31.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 8e2f3b2089ea33e20e3b2a76190f9facae77772fdb9dd72e9c3d3b426fc2edb9(sha256)  Expected: 17e5200b2c4045b2060e094ff383317f5c55744eb482d91b1ae026d473b83dc2(sha256) [MIRROR] openvswitch-selinux-extra-policy-1.0-31.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 8e2f3b2089ea33e20e3b2a76190f9facae77772fdb9dd72e9c3d3b426fc2edb9(sha256)  Expected: 17e5200b2c4045b2060e094ff383317f5c55744eb482d91b1ae026d473b83dc2(sha256) [MIRROR] openvswitch-selinux-extra-policy-1.0-31.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 8e2f3b2089ea33e20e3b2a76190f9facae77772fdb9dd72e9c3d3b426fc2edb9(sha256)  Expected: 17e5200b2c4045b2060e094ff383317f5c55744eb482d91b1ae026d473b83dc2(sha256) [FAILED] openvswitch-selinux-extra-policy-1.0-31.el9s.noarch.rpm: No more mirrors to try - All mirrors were already tried without success (2/2): openvswitch3.1-3.1.0-23.el9s.  1% [-       ]  68 kB/s | 109 kB     01:41 ETA The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: Error downloading packages:   openvswitch-selinux-extra-policy-1.0-31.el9s.noarch: Cannot download, all mirrors were already tried without success

[1] https://bugs.launchpad.net/tripleo/+bug/2020304 https://bugs.launchpad.net/tripleo/+bug/2020304

Tracked in https://pagure.io/centos-infra/issue/1161

Seems to be related with same ENVR packages but obviously changed (as they are now signed) and buildlogs.centos.org has its own CDN , keeping (by default as set by sponsor/vendor) everything in cache for 12 days. Taken actions will be tracked in ticket (for people interested)

For the record, we are also observing the same problem in CloudSIG openstack-antelope testing repo:

[MIRROR] libsodium-1.0.18-7.el9s.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: 1f2a5abe61bdd68bcbee962cf9a3b81158d1e5b987b24f421cb902924d75bb89(sha256) Expected: 8147c75c0454e7e58f5ac3a351e08b8b734ac3ddef803885c085515523ad8327(sha256) (14/194): openstack-barbican-worker-16.0.0-2.el9s.noarch.rpm

2.1 kB/s | 8.4 kB 00:04 [MIRROR] python-oslo-cache-lang-3.3.1-1.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 7344f476bdf799f62457a6a34bfee00a845656eca4b19429d25e97dfde1c2b38(sha256) Expected: 125622fdeb3a2e02aa4cdcbc25a9a631677aba4c4de97d823ddb3c5230ecb714(sha256) [MIRROR] python-oslo-concurrency-lang-5.1.1-2.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: cd53374a430a54df57a6482c2805e931dfd0030778a1a16fcf6e8b97a2dda69c(sha256) Expected: e0c44f79afc2568b30e85f3aeb128f375129694c77166bfa07a73c3bacd6bb53(sha256) [MIRROR] python-oslo-cache-lang-3.3.1-1.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 7344f476bdf799f62457a6a34bfee00a845656eca4b19429d25e97dfde1c2b38(sha256) Expected: 125622fdeb3a2e02aa4cdcbc25a9a631677aba4c4de97d823ddb3c5230ecb714(sha256) [FAILED] python-oslo-cache-lang-3.3.1-1.el9s.noarch.rpm: No more mirrors to try - All mirrors were already tried without success

My recommendation for users hitting this so far in CloudSIG or NFV SIG is to move to release repos unless you want to explicitly test new packages recently added to -testing repo.

Alfredo

On Mon, May 22, 2023 at 10:16 AM Alfredo Moralejo Alonso < amoralej@redhat.com> wrote:

Hi,

Apparently untagging/tagging packages in the -testing tag is helping to fix the issue in some packages (ovn22.12, openvswitch-selinux-extra-policy, ovn22.09) but not openvswitch3.1-3.1.0-23.el9s.

I'm currently untagging it to test the previous build.

Note that in NFV SIG we are building the same packages in C9S and RHEL9 buildroots with different disttag el9s vs el9. May this be related?

Alfredo

On Mon, May 22, 2023 at 8:53 AM YATIN KAREL yatinkarel@gmail.com wrote:

...

Hi,

Seeing similar issues in Openstack CI[1].

For example:- $ sudo dnf install --repofrompath=tmp, https://buildlogs.centos.org/centos/9-stream/nfv/x86_64/openvswitch-2/ --enablerepo=tmp --disablerepo=* openvswitch3.1 Added tmp repo from https://buildlogs.centos.org/centos/9-stream/nfv/x86_64/openvswitch-2/ tmp 11 kB/s | 3.0 kB 00:00 Dependencies resolved.

================================================================================================================ Package Architecture Version Repository Size

================================================================================================================ Installing: openvswitch3.1 x86_64 3.1.0-23.el9s tmp 6.8 M Installing dependencies: openvswitch-selinux-extra-policy noarch 1.0-31.el9s tmp 14 k

Transaction Summary

================================================================================================================ Install 2 Packages

Total download size: 6.8 M Installed size: 22 M Is this ok [y/N]: y Downloading Packages: [MIRROR] openvswitch-selinux-extra-policy-1.0-31.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 8e2f3b2089ea33e20e3b2a76190f9facae77772fdb9dd72e9c3d3b426fc2edb9(sha256) Expected: 17e5200b2c4045b2060e094ff383317f5c55744eb482d91b1ae026d473b83dc2(sha256) [MIRROR] openvswitch-selinux-extra-policy-1.0-31.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 8e2f3b2089ea33e20e3b2a76190f9facae77772fdb9dd72e9c3d3b426fc2edb9(sha256) Expected: 17e5200b2c4045b2060e094ff383317f5c55744eb482d91b1ae026d473b83dc2(sha256) [MIRROR] openvswitch-selinux-extra-policy-1.0-31.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 8e2f3b2089ea33e20e3b2a76190f9facae77772fdb9dd72e9c3d3b426fc2edb9(sha256) Expected: 17e5200b2c4045b2060e094ff383317f5c55744eb482d91b1ae026d473b83dc2(sha256) [MIRROR] openvswitch-selinux-extra-policy-1.0-31.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 8e2f3b2089ea33e20e3b2a76190f9facae77772fdb9dd72e9c3d3b426fc2edb9(sha256) Expected: 17e5200b2c4045b2060e094ff383317f5c55744eb482d91b1ae026d473b83dc2(sha256) [FAILED] openvswitch-selinux-extra-policy-1.0-31.el9s.noarch.rpm: No more mirrors to try - All mirrors were already tried without success (2/2): openvswitch3.1-3.1.0-23.el9s. 1% [- ] 68 kB/s | 109 kB 01:41 ETA The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: Error downloading packages: openvswitch-selinux-extra-policy-1.0-31.el9s.noarch: Cannot download, all mirrors were already tried without success

[1] https://bugs.launchpad.net/tripleo/+bug/2020304

Thanks and Regards Yatin Karel

On Fri, May 19, 2023 at 7:38 PM Fabian Arrotin arrfab@centos.org wrote:

...

On 19/05/2023 15:59, Sandro Bonazzola wrote:

...

Il giorno ven 19 mag 2023 alle ore 15:52 Troy Dawson <

tdawson@redhat.com

...

mailto:tdawson@redhat.com> ha scritto:

Those don't sound like CentOS packages.
Centos packages do not have dist tags of .el9s  or .el8s
Please check the repo's those are coming from and contact the
respective repository owners.

Those are from CentOS SIGs, https://cbs.centos.org/koji/buildinfo?buildID=39630 https://cbs.centos.org/koji/buildinfo?buildID=39630 https://cbs.centos.org/koji/buildinfo?buildID=42936 https://cbs.centos.org/koji/buildinfo?buildID=42936

Can you elaborate where you're seeing that ? you mentioned oVirt CI so outside of centos infra (or outside of CBS) How do you test , from which repo, etc ? It's true that packages in -testing are now signed (it was announced https://lists.centos.org/pipermail/centos-devel/2023-May/142901.html) but is that for -release ? what about mirrormanager ?

-- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

---

CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

-- Yatin Karel _______________________________________________ CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel

On Mon, May 22, 2023 at 2:32 PM Fabian Arrotin arrfab@centos.org wrote:

On 22/05/2023 11:13, Alfredo Moralejo Alonso wrote:

...

For the record, we are also observing the same problem in CloudSIG openstack-antelope testing repo:

[MIRROR] libsodium-1.0.18-7.el9s.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated: 1f2a5abe61bdd68bcbee962cf9a3b81158d1e5b987b24f421cb902924d75bb89(sha256) Expected: 8147c75c0454e7e58f5ac3a351e08b8b734ac3ddef803885c085515523ad8327(sha256) (14/194): openstack-barbican-worker-16.0.0-2.el9s.noarch.rpm

          2.1 kB/s | 8.4 kB     00:04

[MIRROR] python-oslo-cache-lang-3.3.1-1.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 7344f476bdf799f62457a6a34bfee00a845656eca4b19429d25e97dfde1c2b38(sha256) Expected: 125622fdeb3a2e02aa4cdcbc25a9a631677aba4c4de97d823ddb3c5230ecb714(sha256) [MIRROR] python-oslo-concurrency-lang-5.1.1-2.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: cd53374a430a54df57a6482c2805e931dfd0030778a1a16fcf6e8b97a2dda69c(sha256) Expected: e0c44f79afc2568b30e85f3aeb128f375129694c77166bfa07a73c3bacd6bb53(sha256) [MIRROR] python-oslo-cache-lang-3.3.1-1.el9s.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 7344f476bdf799f62457a6a34bfee00a845656eca4b19429d25e97dfde1c2b38(sha256) Expected: 125622fdeb3a2e02aa4cdcbc25a9a631677aba4c4de97d823ddb3c5230ecb714(sha256) [FAILED] python-oslo-cache-lang-3.3.1-1.el9s.noarch.rpm: No more mirrors to try - All mirrors were already tried without success

My recommendation for users hitting this so far in CloudSIG or NFV SIG is to move to release repos unless you want to explicitly test new packages recently added to -testing repo.

Alfredo

After some troubleshooting with Alfredo and Tomas Kopecek (thanks again !) we found the issue so it should already be ok for some tags but signing queue is being processed for *all* -testing tags to ensure it would be reflecting signed packages.

Now, I see everything fine for testing repos in cloudsig and nfvsig.

Thanks!

More information in the https://pagure.io/centos-infra/issue/1161 ticket

-- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

---

CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel