Good Morning!
I was recently seeing quite a few failures on the CI jobs of the Automotive SIG. Diving into it a little bit, it looks like duffy errors with: { "error": { "detail": "can't reserve nodes: quantity=1 pool='metal-ec2-c5n-centos-9s-x86_64'" } } and { "error": { "detail": "can't reserve nodes: quantity=1 pool='virt-ec2-c6g-centos-9s-aarch64'" }
So I check the pools: $ duffy client list-pools { "action": "get", "pools": [ { "name": "virt-ec2-t2-centos-9s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-9s-aarch64", "fill_level": 5 }, { "name": "metal-ec2-c5n-centos-9s-x86_64", "fill_level": 6 }, { "name": "virt-ec2-t2-centos-10s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-10s-aarch64", "fill_level": 2 }, { "name": "metal-ec2-c5n-centos-10s-x86_64", "fill_level": 3 } ] }
Which seems fine from here. Do we have monitoring of how busy the nodes are in the pools? Also: is there a way to have duffy client request-session have a --retry option so that it can keep on retrying until a node becomes available? (Maybe combined with a --timeout?) (happy to work with claude on the patches if that's deemed helpful)
Thanks in advance :) Pierre
On 20/10/2025 17:51, Pierre-Yves Chibon wrote:
Good Morning!
I was recently seeing quite a few failures on the CI jobs of the Automotive SIG. Diving into it a little bit, it looks like duffy errors with: { "error": { "detail": "can't reserve nodes: quantity=1 pool='metal-ec2-c5n-centos-9s-x86_64'" } } and { "error": { "detail": "can't reserve nodes: quantity=1 pool='virt-ec2-c6g-centos-9s-aarch64'" }
So I check the pools: $ duffy client list-pools { "action": "get", "pools": [ { "name": "virt-ec2-t2-centos-9s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-9s-aarch64", "fill_level": 5 }, { "name": "metal-ec2-c5n-centos-9s-x86_64", "fill_level": 6 }, { "name": "virt-ec2-t2-centos-10s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-10s-aarch64", "fill_level": 2 }, { "name": "metal-ec2-c5n-centos-10s-x86_64", "fill_level": 3 } ] }
Which seems fine from here. Do we have monitoring of how busy the nodes are in the pools? Also: is there a way to have duffy client request-session have a --retry option so that it can keep on retrying until a node becomes available? (Maybe combined with a --timeout?) (happy to work with claude on the patches if that's deemed helpful)
Thanks in advance :) Pierre
Welcome to us-east-1 outage day ... https://health.aws.amazon.com/health/status
For the --retry option, you can eventually open a PR against duffy code itself, and Nils will probably review it if he has time ;-)
On Mon, Oct 20, 2025 at 06:02:37PM +0200, Fabian Arrotin wrote:
On 20/10/2025 17:51, Pierre-Yves Chibon wrote:
Good Morning!
I was recently seeing quite a few failures on the CI jobs of the Automotive SIG. Diving into it a little bit, it looks like duffy errors with: { "error": { "detail": "can't reserve nodes: quantity=1 pool='metal-ec2-c5n-centos-9s-x86_64'" } } and { "error": { "detail": "can't reserve nodes: quantity=1 pool='virt-ec2-c6g-centos-9s-aarch64'" }
So I check the pools: $ duffy client list-pools { "action": "get", "pools": [ { "name": "virt-ec2-t2-centos-9s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-9s-aarch64", "fill_level": 5 }, { "name": "metal-ec2-c5n-centos-9s-x86_64", "fill_level": 6 }, { "name": "virt-ec2-t2-centos-10s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-10s-aarch64", "fill_level": 2 }, { "name": "metal-ec2-c5n-centos-10s-x86_64", "fill_level": 3 } ] }
Which seems fine from here. Do we have monitoring of how busy the nodes are in the pools? Also: is there a way to have duffy client request-session have a --retry option so that it can keep on retrying until a node becomes available? (Maybe combined with a --timeout?) (happy to work with claude on the patches if that's deemed helpful)
Thanks in advance :) Pierre
Welcome to us-east-1 outage day ... https://health.aws.amazon.com/health/status
Doh! didn't think of it :( Alright, I guess I'm done with that for today then :)
For the --retry option, you can eventually open a PR against duffy code itself, and Nils will probably review it if he has time ;-)
I was hoping you could point me to the sources :]
Thanks ! Pierre
On 20/10/2025 18:11, Pierre-Yves Chibon wrote:
On Mon, Oct 20, 2025 at 06:02:37PM +0200, Fabian Arrotin wrote:
On 20/10/2025 17:51, Pierre-Yves Chibon wrote:
Good Morning!
I was recently seeing quite a few failures on the CI jobs of the Automotive SIG. Diving into it a little bit, it looks like duffy errors with: { "error": { "detail": "can't reserve nodes: quantity=1 pool='metal-ec2-c5n-centos-9s-x86_64'" } } and { "error": { "detail": "can't reserve nodes: quantity=1 pool='virt-ec2-c6g-centos-9s-aarch64'" }
So I check the pools: $ duffy client list-pools { "action": "get", "pools": [ { "name": "virt-ec2-t2-centos-9s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-9s-aarch64", "fill_level": 5 }, { "name": "metal-ec2-c5n-centos-9s-x86_64", "fill_level": 6 }, { "name": "virt-ec2-t2-centos-10s-x86_64", "fill_level": 8 }, { "name": "virt-ec2-c6g-centos-10s-aarch64", "fill_level": 2 }, { "name": "metal-ec2-c5n-centos-10s-x86_64", "fill_level": 3 } ] }
Which seems fine from here. Do we have monitoring of how busy the nodes are in the pools? Also: is there a way to have duffy client request-session have a --retry option so that it can keep on retrying until a node becomes available? (Maybe combined with a --timeout?) (happy to work with claude on the patches if that's deemed helpful)
Thanks in advance :) Pierre
Welcome to us-east-1 outage day ... https://health.aws.amazon.com/health/status
Doh! didn't think of it :( Alright, I guess I'm done with that for today then :)
For the --retry option, you can eventually open a PR against duffy code itself, and Nils will probably review it if he has time ;-)
I was hoping you could point me to the sources :]
Thanks ! Pierre
https://github.com/centos/duffy
On Mon, Oct 20, 2025 at 06:26:17PM +0200, Fabian Arrotin wrote:
On 20/10/2025 18:11, Pierre-Yves Chibon wrote:
On Mon, Oct 20, 2025 at 06:02:37PM +0200, Fabian Arrotin wrote:
On 20/10/2025 17:51, Pierre-Yves Chibon wrote:
Also: is there a way to have duffy client request-session have a --retry option so that it can keep on retrying until a node becomes available? (Maybe combined with a --timeout?) (happy to work with claude on the patches if that's deemed helpful)
For the --retry option, you can eventually open a PR against duffy code itself, and Nils will probably review it if he has time ;-)
I was hoping you could point me to the sources :]
Here is the PR: https://github.com/CentOS/duffy/pull/1255
I took the opportunity to scratch a couple of other itches I had (like -h in addition to --help) :)
Pierre
On 20/10/2025 18:02, Fabian Arrotin wrote:
On 20/10/2025 17:51, Pierre-Yves Chibon wrote:
Good Morning!
I was recently seeing quite a few failures on the CI jobs of the Automotive SIG. Diving into it a little bit, it looks like duffy errors with: { "error": { "detail": "can't reserve nodes: quantity=1 pool='metal-ec2- c5n-centos-9s-x86_64'" } }
<snip>
Welcome to us-east-1 outage day ... https://health.aws.amazon.com/health/status
I had a quick look this morning and it seems all back :
[+] Duffy Pool status : Ready | Deployed | Provisioning : [metal-ec2-c5n-centos-10s-x86_64]: 3 | 0 | 0 [metal-ec2-c5n-centos-9s-x86_64]: 6 | 0 | 0 [virt-ec2-c6g-centos-10s-aarch64]: 2 | 0 | 0 [virt-ec2-c6g-centos-9s-aarch64]: 5 | 0 | 0 [virt-ec2-t2-centos-10s-x86_64]: 8 | 0 | 0 [virt-ec2-t2-centos-9s-x86_64]: 8 | 0 | 0
-
Fabian Arrotin -
Pierre-Yves Chibon