[CentOS-devel] unplanned outage : community cage network issue

Mon Oct 2 09:33:25 UTC 2023
Fabian Arrotin <arrfab at centos.org>

On 01/10/2023 22:42, Fabian Arrotin wrote:
> Hi all,
> 
> this Sunday, our zabbix monitoring platform informed us that there were 
> plenty of nodes unreachable in community cage, and after some 
> investigation it's a whole rack that was cut off access from network 
> switches.
> That causes issues for multiple workloads, including duffy CI infra env, 
> CentOS Stream env (aka https://testing.stream.centos.org) or 
> storage/mirror servers used for https://cbs.centos.org.
> 
> As we don't know when the root cause will be investigated by the network 
> team managing the network infra, we have disabled the koji builders, so 
> don't even try to submit builds to cbs.centos.org as they'd just be 
> stuck waiting for builders to be available (themselves disabled until we 
> have access to internal mirror)
> 

Just to let you know that we were able (thanks @Michael from RH Network 
team) to identify and fix the issue so the whole CBS build env is now 
back online and kojid workers are processing tasks that were in the 
build queue

-- 
Fabian Arrotin
The CentOS Project | https://www.centos.org
gpg key: 17F3B7A1 | @arrfab[@fosstodon.org]

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20231002/68b78379/attachment.sig>