On 18/11/2025 20:25, Fabian Arrotin wrote:
On 18/11/2025 10:30, Fabian Arrotin wrote:
<snip>
It seems we have an issue with the releng process that is supposed to automatically sign and push some tags. I'll investigate but it was already a long day and while it worked in our .stg. env, and that it worked also for *some* messages sent to the mqtt bus , it doesn't always work and kojihub doesn't have any output about what's failing.
I'll keep you all informed when it will be all back to normal. In the meantime, should you have an urgent need for tag[s] to be processed, we can manually kick the script (through same message bus , which is why it seems related to kojihub plugin using *same* python code ....)
Status update : while we have to take a deep dive into this, I implemented a workaround so that tagged builds are still processed, but with a small delay of max 15min. One process is now querying for last tagged builds in some tags, and sending notifications on the bus and so signing+push is still happening but not in "real time". That's the workaround but that gives us time to replicate the initial issue, fix it and deploy it. In parallel we suffered from a strange behaviour with newer postgresql and we had some other things to solve (ansible deployed exact same settings as it was on previous instance, only difference is a jump from rhel8 to rhel9 for the hub, as all builders were already on el9)
If you want to track the message bus issue, it's now tracked under https://gitlab.com/CentOS/infra/tracker/-/issues/1808
Apart from that, it seems new infra/builders are good enough and that all is working so the service interruption for https://cbs.centos.org (accepting builds and processing these) was itself below 1h when migration happened yesterday.
I'll keep you informed about status when signing service automation will be back at full steam