Good Morning!
I was recently seeing quite a few failures on the CI jobs of the Automotive SIG.
Diving into it a little bit, it looks like duffy errors with:
{
"error": {
"detail": "can't reserve nodes: quantity=1 pool='metal-ec2-c5n-centos-9s-x86_64'"
}
}
and
{
"error": {
"detail": "can't reserve nodes: quantity=1 pool='virt-ec2-c6g-centos-9s-aarch64'"
}
So I check the pools:
$ duffy client list-pools
{
"action": "get",
"pools": [
{
"name": "virt-ec2-t2-centos-9s-x86_64",
"fill_level": 8
},
{
"name": "virt-ec2-c6g-centos-9s-aarch64",
"fill_level": 5
},
{
"name": "metal-ec2-c5n-centos-9s-x86_64",
"fill_level": 6
},
{
"name": "virt-ec2-t2-centos-10s-x86_64",
"fill_level": 8
},
{
"name": "virt-ec2-c6g-centos-10s-aarch64",
"fill_level": 2
},
{
"name": "metal-ec2-c5n-centos-10s-x86_64",
"fill_level": 3
}
]
}
Which seems fine from here. Do we have monitoring of how busy the nodes are in
the pools?
Also: is there a way to have duffy client request-session have a --retry option
so that it can keep on retrying until a node becomes available? (Maybe combined
with a --timeout?)
(happy to work with claude on the patches if that's deemed helpful)
Thanks in advance :)
Pierre
