On Tue, Oct 6, 2020 at 12:50 PM Katerina Foniok <kkanova at redhat.com> wrote: > > So, I can see that access to Vault was disabled on purpose, so it probably doesn't relate to the outage. Sorry for the hoax. > > We also can see this error message in our jobs: >> >> "msg": "Exceeded maximum allowed fail nodes limit, please release other machines to continue" > > Example of the job is here. So when you mark a node fail (usually when the job fails), the node stays around for 12 hours in case someone wants to check manually on what went wrong. Keeping too many nodes in fail state becomes a bottleneck for duffy pool as it means those nodes can't be reprovisioned for the next round of jobs (for 12 hours). We have a limit on how many can be in the fail state. This is expected and you would have seen it on calling node/fail API which should ideally be called when the job failed. So error could be something else > Thank you for taking a look, > Katka > > On Tue, Oct 6, 2020 at 9:04 AM Katerina Foniok <kkanova at redhat.com> wrote: >> >> Thank you, the `devtools-ci-slave04` is running again but it seems that our jobs can not get credentials from the vault now. Can it be related to the outage? >> >> On Tue, Oct 6, 2020 at 8:43 AM Vipul Siddharth <vipul at redhat.com> wrote: >>> >>> On Tue, Oct 6, 2020 at 11:40 AM Katerina Foniok <kkanova at redhat.com> wrote: >>> > >>> > Hello guys, >>> > >>> > our jobs on ci.centos.org are pending because the devtools-ci-slave04 is offline. Can someone take a look, please? >>> fixed >>> > One of the affected jobs is here. >>> > Thank you! >>> > >>> > Have a great day, >>> > Katka >>> > _______________________________________________ >>> > CI-users mailing list >>> > CI-users at centos.org >>> > https://lists.centos.org/mailman/listinfo/ci-users >>> >>> >>> >>> -- >>> Vipul Siddharth >>> He/His/Him >>> Fedora | CentOS CI Infrastructure Team >>> >>> _______________________________________________ >>> CI-users mailing list >>> CI-users at centos.org >>> https://lists.centos.org/mailman/listinfo/ci-users >>> > _______________________________________________ > CI-users mailing list > CI-users at centos.org > https://lists.centos.org/mailman/listinfo/ci-users -- Vipul Siddharth He/His/Him Fedora | CentOS CI Infrastructure Team