On 22/04/2025 14:06, Fabian Arrotin wrote:
We recently discovered that number of duffy ec2 instances (aws side) is much higher than what Duffy thinks is provisioned so at some point, it lost track of really deployed ec2 and we need to reconcile DB and reality)
We just need to :
stop duffy delete all duffy ec2 instances clean-up duffy DB restart duffy (and it will reprovision from scratch/zero)
Once restarted, you'll be able to resume your ci jobs and requesting duffy nodes
Maintenance is scheduled for """"Wednesday April 23rd, 11:30 am UTC time"""". You can convert to local time with $(date -d '2025-04-23 11:30 UTC')
So just sending this in advance so that you can either pause your builds, or just inject logic into your provisioning scripts for duffy nodes and just retry until service is available again.
Just to let you know that we're fully back in action, and with a clean state (both Duffy DB and AWS side) now.