[Ci-users] No space left on device

Thank you very much guys for your quick response and solving the issue!

On Thu, Dec 12, 2019 at 11:17 AM Vipul Siddharth <vipul at redhat.com> wrote:

> On Thu, Dec 12, 2019 at 3:34 PM Jaime Melis <jmelis at redhat.com> wrote:
> >
> > Hi Vipul,
> >
> > I ended up killing the jobs manually because they were taking very long.
> >
> > Can you re-enable the slave now?
> Done :)
> >
> > Thanks a lot for the swift support Vipul.
> I am glad this was done with not a lot of delay :)
> >
> > Cheers,
> > Jaime
> >
> > On Thu, 12 Dec 2019 at 10:39, Vipul Siddharth <vipul at redhat.com> wrote:
> > >
> > > On Thu, Dec 12, 2019 at 3:06 PM Jaime Melis <jmelis at redhat.com> wrote:
> > > >
> > > > Hi Vipul,
> > > >
> > > > The backup is done, can you please put the slave in maintenance mode?
> > > Done
> > > Please let me know once you are done cleaning the workspace.
> > > Once again, thank you for stepping in for this.
> > > >
> > > > On Thu, 12 Dec 2019 at 10:14, Vipul Siddharth <vipul at redhat.com>
> wrote:
> > > > >
> > > > > On Thu, Dec 12, 2019 at 2:40 PM Jaime Melis <jmelis at redhat.com>
> wrote:
> > > > > >
> > > > > > I'm backing it up manually, so no need for step 2.
> > > > > >
> > > > > > Can I ping you once the backup is done so you can put the slave
> in
> > > > > > maintenance mode?
> > > > > yes
> > > > > please do :)
> > > > > >
> > > > > >
> > > > > > On Thu, 12 Dec 2019 at 10:05, Vipul Siddharth <vipul at redhat.com>
> wrote:
> > > > > > >
> > > > > > > On Thu, Dec 12, 2019 at 2:32 PM Jaime Melis <jmelis at redhat.com>
> wrote:
> > > > > > > >
> > > > > > > > Hi Vipul,
> > > > > > > >
> > > > > > > > The slave is full. Basically too many workspaces with a fair
> amount of
> > > > > > > > data in them:
> > > > > > > >
> > > > > > > > 15.5 GiB /workspace
> > > > > > > >
> > > > > > > > And these are the top 10 dirs under /workspace:
> > > > > > > >
> > > > > > > > 547.9 MiB
> /devtools-openshift-jenkins-s2i-config-fabric8-push-prcheck at 2
> > > > > > > > 547.3 MiB
> /devtools-openshift-jenkins-s2i-config-fabric8-push-prcheck
> > > > > > > > 464.2 MiB /devtools-kubernetes-model at 2
> > > > > > > > 460.9 MiB
> /devtools-kubernetes-model-fabric8-push-build-master
> > > > > > > > 422.0 MiB
> /devtools-openshift-jenkins-s2i-config-fabric8-push-build-master
> > > > > > > > 362.3 MiB
> /devtools-kubernetes-client-fabric8-push-build-master
> > > > > > > > 241.8 MiB /devtools-kubernetes-client-fabric8-push-prcheck at 4
> > > > > > > > 236.0 MiB /devtools-kubernetes-client-fabric8-push-prcheck at 6
> > > > > > > > 236.0 MiB /devtools-kubernetes-client-fabric8-push-prcheck at 5
> > > > > > > > 206.5 MiB
> /devtools-contract-test-consumer-fabric8-wit-fabric8-auth
> > > > > > > >
> > > > > > > > So, I would propose to delete all those, in order to do that
> I would
> > > > > > > > like to request the following:
> > > > > > > >
> > > > > > > > 1. put the slave in maintenance mode
> > > > > > > > 2. take snapshot of the slave's disk
> > > > > > > > 3. we will delete all the workspaces
> > > > > > > > 4. re-enable the slave node
> > > > > > > >
> > > > > > > > Can you do 1, 2 and 4, and coordinate with me to do 3?
> > > > > > > Thank you Jaime for the help,
> > > > > > > I am looking into it
> > > > > > > Appreciate your help
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jaime
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, 12 Dec 2019 at 09:41, Vipul Siddharth <
> vipul at redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > The Jenkins master has good space left, it could be that
> the slave
> > > > > > > > > node itself is full.
> > > > > > > > > Can you please share the build link?
> > > > > > > > >
> > > > > > > > > On Thu, Dec 12, 2019 at 1:53 PM Katerina Foniok <
> kkanova at redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hello guys,
> > > > > > > > > >
> > > > > > > > > > from yesterday evening we are experiencing failures in
> our jobs on ci.centos.org saying No space left on device. It's blocking
> our PR checks.
> > > > > > > > > >
> > > > > > > > > > Could you please take a look at what's happening there?
> Thank you!
> > > > > > > > > >
> > > > > > > > > > Have a nice day,
> > > > > > > > > > Katka
> > > > > > > > > > _______________________________________________
> > > > > > > > > > Ci-users mailing list
> > > > > > > > > > Ci-users at centos.org
> > > > > > > > > > https://lists.centos.org/mailman/listinfo/ci-users
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Vipul Siddharth
> > > > > > > > > He/His/Him
> > > > > > > > > Fedora | CentOS CI Infrastructure Team
> > > > > > > > > Red Hat
> > > > > > > > > w: vipul.dev
> > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > Ci-users mailing list
> > > > > > > > > Ci-users at centos.org
> > > > > > > > > https://lists.centos.org/mailman/listinfo/ci-users
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Jaime Melis
> > > > > > > > Application SRE team, Service Delivery
> > > > > > > > Red Hat
> > > > > > > > jmelis at redhat.com
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Vipul Siddharth
> > > > > > > He/His/Him
> > > > > > > Fedora | CentOS CI Infrastructure Team
> > > > > > > Red Hat
> > > > > > > w: vipul.dev
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Jaime Melis
> > > > > > Application SRE team, Service Delivery
> > > > > > Red Hat
> > > > > > jmelis at redhat.com
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Vipul Siddharth
> > > > > He/His/Him
> > > > > Fedora | CentOS CI Infrastructure Team
> > > > > Red Hat
> > > > > w: vipul.dev
> > > > >
> > > >
> > > >
> > > > --
> > > > Jaime Melis
> > > > Application SRE team, Service Delivery
> > > > Red Hat
> > > > jmelis at redhat.com
> > > >
> > >
> > >
> > > --
> > > Vipul Siddharth
> > > He/His/Him
> > > Fedora | CentOS CI Infrastructure Team
> > > Red Hat
> > > w: vipul.dev
> > >
> >
> >
> > --
> > Jaime Melis
> > Application SRE team, Service Delivery
> > Red Hat
> > jmelis at redhat.com
> >
>
>
> --
> Vipul Siddharth
> He/His/Him
> Fedora | CentOS CI Infrastructure Team
> Red Hat
> w: vipul.dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20191212/f366a581/attachment-0002.html>