Hi all,
this Sunday, our zabbix monitoring platform informed us that there were plenty of nodes unreachable in community cage, and after some investigation it's a whole rack that was cut off access from network switches. That causes issues for multiple workloads, including duffy CI infra env, CentOS Stream env (aka https://testing.stream.centos.org) or storage/mirror servers used for https://cbs.centos.org.
As we don't know when the root cause will be investigated by the network team managing the network infra, we have disabled the koji builders, so don't even try to submit builds to cbs.centos.org as they'd just be stuck waiting for builders to be available (themselves disabled until we have access to internal mirror)
On 01/10/2023 22:42, Fabian Arrotin wrote:
Hi all,
this Sunday, our zabbix monitoring platform informed us that there were plenty of nodes unreachable in community cage, and after some investigation it's a whole rack that was cut off access from network switches. That causes issues for multiple workloads, including duffy CI infra env, CentOS Stream env (aka https://testing.stream.centos.org) or storage/mirror servers used for https://cbs.centos.org.
As we don't know when the root cause will be investigated by the network team managing the network infra, we have disabled the koji builders, so don't even try to submit builds to cbs.centos.org as they'd just be stuck waiting for builders to be available (themselves disabled until we have access to internal mirror)
Just to let you know that we were able (thanks @Michael from RH Network team) to identify and fix the issue so the whole CBS build env is now back online and kojid workers are processing tasks that were in the build queue
-
Fabian Arrotin