Hi Vipul, I ended up killing the jobs manually because they were taking very long. Can you re-enable the slave now? Thanks a lot for the swift support Vipul. Cheers, Jaime On Thu, 12 Dec 2019 at 10:39, Vipul Siddharth <vipul at redhat.com> wrote: > > On Thu, Dec 12, 2019 at 3:06 PM Jaime Melis <jmelis at redhat.com> wrote: > > > > Hi Vipul, > > > > The backup is done, can you please put the slave in maintenance mode? > Done > Please let me know once you are done cleaning the workspace. > Once again, thank you for stepping in for this. > > > > On Thu, 12 Dec 2019 at 10:14, Vipul Siddharth <vipul at redhat.com> wrote: > > > > > > On Thu, Dec 12, 2019 at 2:40 PM Jaime Melis <jmelis at redhat.com> wrote: > > > > > > > > I'm backing it up manually, so no need for step 2. > > > > > > > > Can I ping you once the backup is done so you can put the slave in > > > > maintenance mode? > > > yes > > > please do :) > > > > > > > > > > > > On Thu, 12 Dec 2019 at 10:05, Vipul Siddharth <vipul at redhat.com> wrote: > > > > > > > > > > On Thu, Dec 12, 2019 at 2:32 PM Jaime Melis <jmelis at redhat.com> wrote: > > > > > > > > > > > > Hi Vipul, > > > > > > > > > > > > The slave is full. Basically too many workspaces with a fair amount of > > > > > > data in them: > > > > > > > > > > > > 15.5 GiB /workspace > > > > > > > > > > > > And these are the top 10 dirs under /workspace: > > > > > > > > > > > > 547.9 MiB /devtools-openshift-jenkins-s2i-config-fabric8-push-prcheck at 2 > > > > > > 547.3 MiB /devtools-openshift-jenkins-s2i-config-fabric8-push-prcheck > > > > > > 464.2 MiB /devtools-kubernetes-model at 2 > > > > > > 460.9 MiB /devtools-kubernetes-model-fabric8-push-build-master > > > > > > 422.0 MiB /devtools-openshift-jenkins-s2i-config-fabric8-push-build-master > > > > > > 362.3 MiB /devtools-kubernetes-client-fabric8-push-build-master > > > > > > 241.8 MiB /devtools-kubernetes-client-fabric8-push-prcheck at 4 > > > > > > 236.0 MiB /devtools-kubernetes-client-fabric8-push-prcheck at 6 > > > > > > 236.0 MiB /devtools-kubernetes-client-fabric8-push-prcheck at 5 > > > > > > 206.5 MiB /devtools-contract-test-consumer-fabric8-wit-fabric8-auth > > > > > > > > > > > > So, I would propose to delete all those, in order to do that I would > > > > > > like to request the following: > > > > > > > > > > > > 1. put the slave in maintenance mode > > > > > > 2. take snapshot of the slave's disk > > > > > > 3. we will delete all the workspaces > > > > > > 4. re-enable the slave node > > > > > > > > > > > > Can you do 1, 2 and 4, and coordinate with me to do 3? > > > > > Thank you Jaime for the help, > > > > > I am looking into it > > > > > Appreciate your help > > > > > > > > > > > > Thanks, > > > > > > Jaime > > > > > > > > > > > > > > > > > > On Thu, 12 Dec 2019 at 09:41, Vipul Siddharth <vipul at redhat.com> wrote: > > > > > > > > > > > > > > The Jenkins master has good space left, it could be that the slave > > > > > > > node itself is full. > > > > > > > Can you please share the build link? > > > > > > > > > > > > > > On Thu, Dec 12, 2019 at 1:53 PM Katerina Foniok <kkanova at redhat.com> wrote: > > > > > > > > > > > > > > > > Hello guys, > > > > > > > > > > > > > > > > from yesterday evening we are experiencing failures in our jobs on ci.centos.org saying No space left on device. It's blocking our PR checks. > > > > > > > > > > > > > > > > Could you please take a look at what's happening there? Thank you! > > > > > > > > > > > > > > > > Have a nice day, > > > > > > > > Katka > > > > > > > > _______________________________________________ > > > > > > > > Ci-users mailing list > > > > > > > > Ci-users at centos.org > > > > > > > > https://lists.centos.org/mailman/listinfo/ci-users > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Vipul Siddharth > > > > > > > He/His/Him > > > > > > > Fedora | CentOS CI Infrastructure Team > > > > > > > Red Hat > > > > > > > w: vipul.dev > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Ci-users mailing list > > > > > > > Ci-users at centos.org > > > > > > > https://lists.centos.org/mailman/listinfo/ci-users > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Jaime Melis > > > > > > Application SRE team, Service Delivery > > > > > > Red Hat > > > > > > jmelis at redhat.com > > > > > > > > > > > > > > > > > > > > > -- > > > > > Vipul Siddharth > > > > > He/His/Him > > > > > Fedora | CentOS CI Infrastructure Team > > > > > Red Hat > > > > > w: vipul.dev > > > > > > > > > > > > > > > > > -- > > > > Jaime Melis > > > > Application SRE team, Service Delivery > > > > Red Hat > > > > jmelis at redhat.com > > > > > > > > > > > > > -- > > > Vipul Siddharth > > > He/His/Him > > > Fedora | CentOS CI Infrastructure Team > > > Red Hat > > > w: vipul.dev > > > > > > > > > -- > > Jaime Melis > > Application SRE team, Service Delivery > > Red Hat > > jmelis at redhat.com > > > > > -- > Vipul Siddharth > He/His/Him > Fedora | CentOS CI Infrastructure Team > Red Hat > w: vipul.dev > -- Jaime Melis Application SRE team, Service Delivery Red Hat jmelis at redhat.com