Ah, ok, thank you very much for clarifying! On Tue, Oct 6, 2020 at 9:42 AM Vipul Siddharth <vipul at redhat.com> wrote: > On Tue, Oct 6, 2020 at 12:50 PM Katerina Foniok <kkanova at redhat.com> > wrote: > > > > So, I can see that access to Vault was disabled on purpose, so it > probably doesn't relate to the outage. Sorry for the hoax. > > > > We also can see this error message in our jobs: > >> > >> "msg": "Exceeded maximum allowed fail nodes limit, please release other > machines to continue" > > > > Example of the job is here. > So when you mark a node fail (usually when the job fails), the node > stays around for 12 hours in case someone wants to check manually on > what went wrong. > Keeping too many nodes in fail state becomes a bottleneck for duffy > pool as it means those nodes can't be reprovisioned for the next round > of jobs (for 12 hours). > We have a limit on how many can be in the fail state. > This is expected and you would have seen it on calling node/fail API > which should ideally be called when the job failed. So error could be > something else > > > Thank you for taking a look, > > Katka > > > > On Tue, Oct 6, 2020 at 9:04 AM Katerina Foniok <kkanova at redhat.com> > wrote: > >> > >> Thank you, the `devtools-ci-slave04` is running again but it seems that > our jobs can not get credentials from the vault now. Can it be related to > the outage? > >> > >> On Tue, Oct 6, 2020 at 8:43 AM Vipul Siddharth <vipul at redhat.com> > wrote: > >>> > >>> On Tue, Oct 6, 2020 at 11:40 AM Katerina Foniok <kkanova at redhat.com> > wrote: > >>> > > >>> > Hello guys, > >>> > > >>> > our jobs on ci.centos.org are pending because the > devtools-ci-slave04 is offline. Can someone take a look, please? > >>> fixed > >>> > One of the affected jobs is here. > >>> > Thank you! > >>> > > >>> > Have a great day, > >>> > Katka > >>> > _______________________________________________ > >>> > CI-users mailing list > >>> > CI-users at centos.org > >>> > https://lists.centos.org/mailman/listinfo/ci-users > >>> > >>> > >>> > >>> -- > >>> Vipul Siddharth > >>> He/His/Him > >>> Fedora | CentOS CI Infrastructure Team > >>> > >>> _______________________________________________ > >>> CI-users mailing list > >>> CI-users at centos.org > >>> https://lists.centos.org/mailman/listinfo/ci-users > >>> > > _______________________________________________ > > CI-users mailing list > > CI-users at centos.org > > https://lists.centos.org/mailman/listinfo/ci-users > > > > -- > Vipul Siddharth > He/His/Him > Fedora | CentOS CI Infrastructure Team > > _______________________________________________ > CI-users mailing list > CI-users at centos.org > https://lists.centos.org/mailman/listinfo/ci-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/ci-users/attachments/20201006/ad9601ae/attachment-0003.html>