Hi guys,
apologies if you are already working on this, but we keep getting 'Insufficient Nodes in READY State' so none of our jobs work.
More info here: https://bugs.centos.org/view.php?id=15185
On 16/08/18 13:25, Jaime Melis wrote:
Hi guys,
apologies if you are already working on this, but we keep getting 'Insufficient Nodes in READY State' so none of our jobs work.
More info here: https://bugs.centos.org/view.php?id=15185
Yes, I've currently "paused" Jenkins to not accept jobs and let the Ready Pool to go back to normal state (and so have enough nodes ) I'll open the Jenkins gate for jobs when it will be ready
On 16/08/18 13:50, Fabian Arrotin wrote:
On 16/08/18 13:25, Jaime Melis wrote:
Hi guys,
apologies if you are already working on this, but we keep getting 'Insufficient Nodes in READY State' so none of our jobs work.
More info here: https://bugs.centos.org/view.php?id=15185
Yes, I've currently "paused" Jenkins to not accept jobs and let the Ready Pool to go back to normal state (and so have enough nodes ) I'll open the Jenkins gate for jobs when it will be ready
I have now reopened Jenkins for jobs and I'll keep an eye on that. I'll also sync with Brian later about a fix for this (two issues happened at almost same time and so monitoring didn't trigger - monitoring itself having issue - )
On 16/08/18 13:50, Fabian Arrotin wrote:
On 16/08/18 13:25, Jaime Melis wrote:
Hi guys,
apologies if you are already working on this, but we keep getting 'Insufficient Nodes in READY State' so none of our jobs work.
More info here: https://bugs.centos.org/view.php?id=15185
Yes, I've currently "paused" Jenkins to not accept jobs and let the Ready Pool to go back to normal state (and so have enough nodes ) I'll open the Jenkins gate for jobs when it will be ready
Just as a follow-up : it seems *plenty* of jobs are now happening on bare-metal, for very short jobs, meaning that such jobs in a single queue are eating the whole pool faster than the workers can provision/reinstall the bare-metal (~12min on such kind of hardware) nodes.
It's maybe time to push such workload on the Cloud.cico (openstack) side, in Virtual Machines, instead of bare-metal if such jobs don't need really hardware for multi-tenants tests ?
On 08/16/2018 08:21 AM, Fabian Arrotin wrote:
It's maybe time to push such workload on the Cloud.cico (openstack) side, in Virtual Machines, instead of bare-metal if such jobs don't need really hardware for multi-tenants tests ?
I'd be happy to be migrated - my job just fires up containers (currently via Docker, and I've been trying without success to get my jobs to run in podman) so it should work just fine virtualized.
Do I need to take any action, or is that an admin task?
On 08/16/2018 08:23 AM, Randy Barlow wrote:
I'd be happy to be migrated - my job just fires up containers (currently via Docker, and I've been trying without success to get my jobs to run in podman) so it should work just fine virtualized.
Haha, for the sake of clarity I suppose I should mention that by "my job" I mean Bodhi:
On 16/08/18 14:23, Randy Barlow wrote:
On 08/16/2018 08:21 AM, Fabian Arrotin wrote:
It's maybe time to push such workload on the Cloud.cico (openstack) side, in Virtual Machines, instead of bare-metal if such jobs don't need really hardware for multi-tenants tests ?
I'd be happy to be migrated - my job just fires up containers (currently via Docker, and I've been trying without success to get my jobs to run in podman) so it should work just fine virtualized.
Do I need to take any action, or is that an admin task?
I discussed that already last week with Brian and as a starting point we'd like that to be "opt-in", aka people would need to request a VM (instead of bare-metal node by default). Proper code in Duffy to accept such request has to land first though, and we'll announce when it will be there.
After some time, (and also after being announced) the goal would probably be to have vm as the default and so bare-metal as "opt-in" ... Let's first though have that plumbed into Duffy api code and so projects willing to switch to just adapt their ci jobs to request that .
-
Fabian Arrotin -
Jaime Melis -
Randy Barlow