On 22/04/2025 14:06, Fabian Arrotin wrote:
We recently discovered that number of duffy ec2 instances (aws side) is much higher than what Duffy thinks is provisioned so at some point, it lost track of really deployed ec2 and we need to reconcile DB and reality)
We just need to :
stop duffy delete all duffy ec2 instances clean-up duffy DB restart duffy (and it will reprovision from scratch/zero)
Once restarted, you'll be able to resume your ci jobs and requesting duffy nodes
Maintenance is scheduled for """"Wednesday April 23rd, 11:30 am UTC time"""". You can convert to local time with $(date -d '2025-04-23 11:30 UTC')
So just sending this in advance so that you can either pause your builds, or just inject logic into your provisioning scripts for duffy nodes and just retry until service is available again.
Just to add that we'll also decommission ppc64le architecture in Duffy pool, due to upcoming DC move (and Duffy will entirely run from the cloud, where it's not possible to request ppc64le arch anyway - see https://pagure.io/centos-infra/issue/1590 )