From frantisek at sumsal.cz  Fri Feb  7 16:34:46 2020
From: frantisek at sumsal.cz (=?UTF-8?B?RnJhbnRpxaFlayDFoHVtxaFhbA==?=)
Date: Fri, 7 Feb 2020 17:34:46 +0100
Subject: [Ci-users] Missing recent builds
Message-ID: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>

Hello,

I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
removed, even though they shouldn't be, according to the retention policy (which is set
to keep builds for 90 days in all affected jobs). As it's not the first time this happened
I'm curious if there's some other retention policy we should be aware of, as this is kind of
annoying - the missing jobs are a few days old, some of them failed, but apart from not
being able to check the logs, we can't even reschedule them, because they don't exist
anymore.

Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
master.

PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:

Thank you,
Frantisek

[0] systemd-ci-slave01

-- 
PGP Key ID: 0xFB738CE27B634E4B



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200207/60e6de13/attachment.sig>

From vipul at redhat.com  Fri Feb  7 16:43:56 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Fri, 7 Feb 2020 22:13:56 +0530
Subject: [Ci-users] Missing recent builds
In-Reply-To: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
Message-ID: <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>

When /var/lib/jenkins gets full (or about to), I leave just the last
20 builds and prune rest to clear up some space
Since we advise to copy important things (like builds to artifacts), I
didn't know they were important (leaving 20 to go back and address
some issues)
In case you don't want me messing with them, please let me know.
Apologies for those.

On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>
> Hello,
>
> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
> removed, even though they shouldn't be, according to the retention policy (which is set
> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
> I'm curious if there's some other retention policy we should be aware of, as this is kind of
> annoying - the missing jobs are a few days old, some of them failed, but apart from not
> being able to check the logs, we can't even reschedule them, because they don't exist
> anymore.
>
> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
> master.
>
> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
>
> Thank you,
> Frantisek
>
> [0] systemd-ci-slave01
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>
>
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev


From frantisek at sumsal.cz  Fri Feb  7 17:15:48 2020
From: frantisek at sumsal.cz (=?UTF-8?B?RnJhbnRpxaFlayDFoHVtxaFhbA==?=)
Date: Fri, 7 Feb 2020 18:15:48 +0100
Subject: [Ci-users] Missing recent builds
In-Reply-To: <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
 <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
Message-ID: <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>

I see, thanks a lot for the explanation!

I guess the issue here is that in systemd we usually generate several 
dozens of builds per day, so leaving just last 20 cuts into the 'recent
history'. However, I absolutely get the space issue, so I'll reconfigure
all jobs to keep the artifacts for 30 days, and the jobs themselves for
60 days (and they shouldn't take much space and would help us to reschedule
them if they're needed again).

Anyway, if the need arises and a cleanup is necessary, could you remove
just the artifacts? It would leave us with the option of being able to
easily reschedule the job (as doing that manually is next to impossible
due to sheer amount of variables the GitHub plugin sets). 

Thanks!

On 2/7/20 5:43 PM, Vipul Siddharth wrote:
> When /var/lib/jenkins gets full (or about to), I leave just the last
> 20 builds and prune rest to clear up some space
> Since we advise to copy important things (like builds to artifacts), I
> didn't know they were important (leaving 20 to go back and address
> some issues)
> In case you don't want me messing with them, please let me know.
> Apologies for those.
> 
> On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>>
>> Hello,
>>
>> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
>> removed, even though they shouldn't be, according to the retention policy (which is set
>> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
>> I'm curious if there's some other retention policy we should be aware of, as this is kind of
>> annoying - the missing jobs are a few days old, some of them failed, but apart from not
>> being able to check the logs, we can't even reschedule them, because they don't exist
>> anymore.
>>
>> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
>> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
>> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
>> master.
>>
>> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
>>
>> Thank you,
>> Frantisek
>>
>> [0] systemd-ci-slave01
>>
>> --
>> PGP Key ID: 0xFB738CE27B634E4B
>>
>>
>>
> 
> 

-- 
PGP Key ID: 0xFB738CE27B634E4B

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200207/c985eddf/attachment.sig>

From vipul at redhat.com  Fri Feb  7 17:20:06 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Fri, 7 Feb 2020 22:50:06 +0530
Subject: [Ci-users] Missing recent builds
In-Reply-To: <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
 <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
 <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>
Message-ID: <CANGkBUCdqMq3woMfjaq+mwJS0CPtZjKXPu2fWr7P-dCKhoahfg@mail.gmail.com>

Sure, thank you for understanding :)

On Fri, Feb 7, 2020 at 10:46 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>
> I see, thanks a lot for the explanation!
>
> I guess the issue here is that in systemd we usually generate several
> dozens of builds per day, so leaving just last 20 cuts into the 'recent
> history'. However, I absolutely get the space issue, so I'll reconfigure
> all jobs to keep the artifacts for 30 days, and the jobs themselves for
> 60 days (and they shouldn't take much space and would help us to reschedule
> them if they're needed again).
>
> Anyway, if the need arises and a cleanup is necessary, could you remove
> just the artifacts? It would leave us with the option of being able to
> easily reschedule the job (as doing that manually is next to impossible
> due to sheer amount of variables the GitHub plugin sets).
>
> Thanks!
>
> On 2/7/20 5:43 PM, Vipul Siddharth wrote:
> > When /var/lib/jenkins gets full (or about to), I leave just the last
> > 20 builds and prune rest to clear up some space
> > Since we advise to copy important things (like builds to artifacts), I
> > didn't know they were important (leaving 20 to go back and address
> > some issues)
> > In case you don't want me messing with them, please let me know.
> > Apologies for those.
> >
> > On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
> >>
> >> Hello,
> >>
> >> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
> >> removed, even though they shouldn't be, according to the retention policy (which is set
> >> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
> >> I'm curious if there's some other retention policy we should be aware of, as this is kind of
> >> annoying - the missing jobs are a few days old, some of them failed, but apart from not
> >> being able to check the logs, we can't even reschedule them, because they don't exist
> >> anymore.
> >>
> >> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
> >> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
> >> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
> >> master.
> >>
> >> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
> >>
> >> Thank you,
> >> Frantisek
> >>
> >> [0] systemd-ci-slave01
> >>
> >> --
> >> PGP Key ID: 0xFB738CE27B634E4B
> >>
> >>
> >>
> >
> >
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev


From kkanova at redhat.com  Wed Feb 26 13:53:01 2020
From: kkanova at redhat.com (Katerina Foniok)
Date: Wed, 26 Feb 2020 14:53:01 +0100
Subject: [Ci-users] No space left on device
Message-ID: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>

Hello guys,

our tests failed on `No space left on device` [1]. Could you please take a
look at that?

Thank you for your effort.
Have a nice day,
Katka

[1]
https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200226/8316b356/attachment.html>

From vipul at redhat.com  Wed Feb 26 14:23:05 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Wed, 26 Feb 2020 19:53:05 +0530
Subject: [Ci-users] No space left on device
In-Reply-To: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
References: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
Message-ID: <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>

On Wed, Feb 26, 2020 at 7:23 PM Katerina Foniok <kkanova at redhat.com> wrote:
>
> Hello guys,
>
> our tests failed on `No space left on device` [1]. Could you please take a look at that?
Hi Katerina,
I have been cleaning slave04 around twice a day and now it has come to
a point where just parallel cleaning is not working.
You (or someone who has access to slave04) can  either clean
/home/devtools/workspace or work with me to tell me what can be
deleted and I will do it.
Sadly I didn't know whom to reach out to ask for this, so this has to
come to this :)
I have written your name in my self maintained "project to people" mapping :)

Please let me know which one works for you and I am just a ping away
to clean things for you
>
> Thank you for your effort.
> Have a nice day,
> Katka
>
> [1] https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users



-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev


From mafriedm at redhat.com  Wed Feb 26 16:34:21 2020
From: mafriedm at redhat.com (Maor Friedman)
Date: Wed, 26 Feb 2020 18:34:21 +0200
Subject: [Ci-users] No space left on device
In-Reply-To: <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>
References: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
 <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>
Message-ID: <CADUYHM-W3v3wDXAKpu+1EXXGQksp_oeAmOVKOyrwvnp8n-v+VQ@mail.gmail.com>

This has been resolved with the help of Vipul!

Thanks!

On Wed, Feb 26, 2020 at 4:24 PM Vipul Siddharth <vipul at redhat.com> wrote:

> On Wed, Feb 26, 2020 at 7:23 PM Katerina Foniok <kkanova at redhat.com>
> wrote:
> >
> > Hello guys,
> >
> > our tests failed on `No space left on device` [1]. Could you please take
> a look at that?
> Hi Katerina,
> I have been cleaning slave04 around twice a day and now it has come to
> a point where just parallel cleaning is not working.
> You (or someone who has access to slave04) can  either clean
> /home/devtools/workspace or work with me to tell me what can be
> deleted and I will do it.
> Sadly I didn't know whom to reach out to ask for this, so this has to
> come to this :)
> I have written your name in my self maintained "project to people" mapping
> :)
>
> Please let me know which one works for you and I am just a ping away
> to clean things for you
> >
> > Thank you for your effort.
> > Have a nice day,
> > Katka
> >
> > [1]
> https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
> > _______________________________________________
> > CI-users mailing list
> > CI-users at centos.org
> > https://lists.centos.org/mailman/listinfo/ci-users
>
>
>
> --
> Vipul Siddharth
> He/His/Him
> Fedora | CentOS CI Infrastructure Team
> Red Hat
> w: vipul.dev
>
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users
>
>

-- 
Maor Friedman
Senior Software Engineer, Application SRE team, Service Delivery
Red Hat
maor at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200226/810d6ac8/attachment.html>

From frantisek at sumsal.cz  Fri Feb  7 16:34:46 2020
From: frantisek at sumsal.cz (=?UTF-8?B?RnJhbnRpxaFlayDFoHVtxaFhbA==?=)
Date: Fri, 7 Feb 2020 17:34:46 +0100
Subject: [Ci-users] Missing recent builds
Message-ID: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>

Hello,

I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
removed, even though they shouldn't be, according to the retention policy (which is set
to keep builds for 90 days in all affected jobs). As it's not the first time this happened
I'm curious if there's some other retention policy we should be aware of, as this is kind of
annoying - the missing jobs are a few days old, some of them failed, but apart from not
being able to check the logs, we can't even reschedule them, because they don't exist
anymore.

Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
master.

PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:

Thank you,
Frantisek

[0] systemd-ci-slave01

-- 
PGP Key ID: 0xFB738CE27B634E4B



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200207/60e6de13/attachment-0002.sig>

From vipul at redhat.com  Fri Feb  7 16:43:56 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Fri, 7 Feb 2020 22:13:56 +0530
Subject: [Ci-users] Missing recent builds
In-Reply-To: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
Message-ID: <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>

When /var/lib/jenkins gets full (or about to), I leave just the last
20 builds and prune rest to clear up some space
Since we advise to copy important things (like builds to artifacts), I
didn't know they were important (leaving 20 to go back and address
some issues)
In case you don't want me messing with them, please let me know.
Apologies for those.

On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>
> Hello,
>
> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
> removed, even though they shouldn't be, according to the retention policy (which is set
> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
> I'm curious if there's some other retention policy we should be aware of, as this is kind of
> annoying - the missing jobs are a few days old, some of them failed, but apart from not
> being able to check the logs, we can't even reschedule them, because they don't exist
> anymore.
>
> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
> master.
>
> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
>
> Thank you,
> Frantisek
>
> [0] systemd-ci-slave01
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>
>
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From frantisek at sumsal.cz  Fri Feb  7 17:15:48 2020
From: frantisek at sumsal.cz (=?UTF-8?B?RnJhbnRpxaFlayDFoHVtxaFhbA==?=)
Date: Fri, 7 Feb 2020 18:15:48 +0100
Subject: [Ci-users] Missing recent builds
In-Reply-To: <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
 <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
Message-ID: <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>

I see, thanks a lot for the explanation!

I guess the issue here is that in systemd we usually generate several 
dozens of builds per day, so leaving just last 20 cuts into the 'recent
history'. However, I absolutely get the space issue, so I'll reconfigure
all jobs to keep the artifacts for 30 days, and the jobs themselves for
60 days (and they shouldn't take much space and would help us to reschedule
them if they're needed again).

Anyway, if the need arises and a cleanup is necessary, could you remove
just the artifacts? It would leave us with the option of being able to
easily reschedule the job (as doing that manually is next to impossible
due to sheer amount of variables the GitHub plugin sets). 

Thanks!

On 2/7/20 5:43 PM, Vipul Siddharth wrote:
> When /var/lib/jenkins gets full (or about to), I leave just the last
> 20 builds and prune rest to clear up some space
> Since we advise to copy important things (like builds to artifacts), I
> didn't know they were important (leaving 20 to go back and address
> some issues)
> In case you don't want me messing with them, please let me know.
> Apologies for those.
> 
> On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>>
>> Hello,
>>
>> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
>> removed, even though they shouldn't be, according to the retention policy (which is set
>> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
>> I'm curious if there's some other retention policy we should be aware of, as this is kind of
>> annoying - the missing jobs are a few days old, some of them failed, but apart from not
>> being able to check the logs, we can't even reschedule them, because they don't exist
>> anymore.
>>
>> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
>> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
>> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
>> master.
>>
>> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
>>
>> Thank you,
>> Frantisek
>>
>> [0] systemd-ci-slave01
>>
>> --
>> PGP Key ID: 0xFB738CE27B634E4B
>>
>>
>>
> 
> 

-- 
PGP Key ID: 0xFB738CE27B634E4B

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200207/c985eddf/attachment-0002.sig>

From vipul at redhat.com  Fri Feb  7 17:20:06 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Fri, 7 Feb 2020 22:50:06 +0530
Subject: [Ci-users] Missing recent builds
In-Reply-To: <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
 <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
 <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>
Message-ID: <CANGkBUCdqMq3woMfjaq+mwJS0CPtZjKXPu2fWr7P-dCKhoahfg@mail.gmail.com>

Sure, thank you for understanding :)

On Fri, Feb 7, 2020 at 10:46 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>
> I see, thanks a lot for the explanation!
>
> I guess the issue here is that in systemd we usually generate several
> dozens of builds per day, so leaving just last 20 cuts into the 'recent
> history'. However, I absolutely get the space issue, so I'll reconfigure
> all jobs to keep the artifacts for 30 days, and the jobs themselves for
> 60 days (and they shouldn't take much space and would help us to reschedule
> them if they're needed again).
>
> Anyway, if the need arises and a cleanup is necessary, could you remove
> just the artifacts? It would leave us with the option of being able to
> easily reschedule the job (as doing that manually is next to impossible
> due to sheer amount of variables the GitHub plugin sets).
>
> Thanks!
>
> On 2/7/20 5:43 PM, Vipul Siddharth wrote:
> > When /var/lib/jenkins gets full (or about to), I leave just the last
> > 20 builds and prune rest to clear up some space
> > Since we advise to copy important things (like builds to artifacts), I
> > didn't know they were important (leaving 20 to go back and address
> > some issues)
> > In case you don't want me messing with them, please let me know.
> > Apologies for those.
> >
> > On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
> >>
> >> Hello,
> >>
> >> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
> >> removed, even though they shouldn't be, according to the retention policy (which is set
> >> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
> >> I'm curious if there's some other retention policy we should be aware of, as this is kind of
> >> annoying - the missing jobs are a few days old, some of them failed, but apart from not
> >> being able to check the logs, we can't even reschedule them, because they don't exist
> >> anymore.
> >>
> >> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
> >> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
> >> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
> >> master.
> >>
> >> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
> >>
> >> Thank you,
> >> Frantisek
> >>
> >> [0] systemd-ci-slave01
> >>
> >> --
> >> PGP Key ID: 0xFB738CE27B634E4B
> >>
> >>
> >>
> >
> >
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From kkanova at redhat.com  Wed Feb 26 13:53:01 2020
From: kkanova at redhat.com (Katerina Foniok)
Date: Wed, 26 Feb 2020 14:53:01 +0100
Subject: [Ci-users] No space left on device
Message-ID: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>

Hello guys,

our tests failed on `No space left on device` [1]. Could you please take a
look at that?

Thank you for your effort.
Have a nice day,
Katka

[1]
https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200226/8316b356/attachment-0002.html>

From vipul at redhat.com  Wed Feb 26 14:23:05 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Wed, 26 Feb 2020 19:53:05 +0530
Subject: [Ci-users] No space left on device
In-Reply-To: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
References: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
Message-ID: <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>

On Wed, Feb 26, 2020 at 7:23 PM Katerina Foniok <kkanova at redhat.com> wrote:
>
> Hello guys,
>
> our tests failed on `No space left on device` [1]. Could you please take a look at that?
Hi Katerina,
I have been cleaning slave04 around twice a day and now it has come to
a point where just parallel cleaning is not working.
You (or someone who has access to slave04) can  either clean
/home/devtools/workspace or work with me to tell me what can be
deleted and I will do it.
Sadly I didn't know whom to reach out to ask for this, so this has to
come to this :)
I have written your name in my self maintained "project to people" mapping :)

Please let me know which one works for you and I am just a ping away
to clean things for you
>
> Thank you for your effort.
> Have a nice day,
> Katka
>
> [1] https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users



-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From mafriedm at redhat.com  Wed Feb 26 16:34:21 2020
From: mafriedm at redhat.com (Maor Friedman)
Date: Wed, 26 Feb 2020 18:34:21 +0200
Subject: [Ci-users] No space left on device
In-Reply-To: <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>
References: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
 <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>
Message-ID: <CADUYHM-W3v3wDXAKpu+1EXXGQksp_oeAmOVKOyrwvnp8n-v+VQ@mail.gmail.com>

This has been resolved with the help of Vipul!

Thanks!

On Wed, Feb 26, 2020 at 4:24 PM Vipul Siddharth <vipul at redhat.com> wrote:

> On Wed, Feb 26, 2020 at 7:23 PM Katerina Foniok <kkanova at redhat.com>
> wrote:
> >
> > Hello guys,
> >
> > our tests failed on `No space left on device` [1]. Could you please take
> a look at that?
> Hi Katerina,
> I have been cleaning slave04 around twice a day and now it has come to
> a point where just parallel cleaning is not working.
> You (or someone who has access to slave04) can  either clean
> /home/devtools/workspace or work with me to tell me what can be
> deleted and I will do it.
> Sadly I didn't know whom to reach out to ask for this, so this has to
> come to this :)
> I have written your name in my self maintained "project to people" mapping
> :)
>
> Please let me know which one works for you and I am just a ping away
> to clean things for you
> >
> > Thank you for your effort.
> > Have a nice day,
> > Katka
> >
> > [1]
> https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
> > _______________________________________________
> > CI-users mailing list
> > CI-users at centos.org
> > https://lists.centos.org/mailman/listinfo/ci-users
>
>
>
> --
> Vipul Siddharth
> He/His/Him
> Fedora | CentOS CI Infrastructure Team
> Red Hat
> w: vipul.dev
>
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users
>
>

-- 
Maor Friedman
Senior Software Engineer, Application SRE team, Service Delivery
Red Hat
maor at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200226/810d6ac8/attachment-0002.html>

From frantisek at sumsal.cz  Fri Feb  7 16:34:46 2020
From: frantisek at sumsal.cz (=?UTF-8?B?RnJhbnRpxaFlayDFoHVtxaFhbA==?=)
Date: Fri, 7 Feb 2020 17:34:46 +0100
Subject: [Ci-users] Missing recent builds
Message-ID: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>

Hello,

I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
removed, even though they shouldn't be, according to the retention policy (which is set
to keep builds for 90 days in all affected jobs). As it's not the first time this happened
I'm curious if there's some other retention policy we should be aware of, as this is kind of
annoying - the missing jobs are a few days old, some of them failed, but apart from not
being able to check the logs, we can't even reschedule them, because they don't exist
anymore.

Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
master.

PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:

Thank you,
Frantisek

[0] systemd-ci-slave01

-- 
PGP Key ID: 0xFB738CE27B634E4B



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200207/60e6de13/attachment-0003.sig>

From vipul at redhat.com  Fri Feb  7 16:43:56 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Fri, 7 Feb 2020 22:13:56 +0530
Subject: [Ci-users] Missing recent builds
In-Reply-To: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
Message-ID: <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>

When /var/lib/jenkins gets full (or about to), I leave just the last
20 builds and prune rest to clear up some space
Since we advise to copy important things (like builds to artifacts), I
didn't know they were important (leaving 20 to go back and address
some issues)
In case you don't want me messing with them, please let me know.
Apologies for those.

On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>
> Hello,
>
> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
> removed, even though they shouldn't be, according to the retention policy (which is set
> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
> I'm curious if there's some other retention policy we should be aware of, as this is kind of
> annoying - the missing jobs are a few days old, some of them failed, but apart from not
> being able to check the logs, we can't even reschedule them, because they don't exist
> anymore.
>
> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
> master.
>
> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
>
> Thank you,
> Frantisek
>
> [0] systemd-ci-slave01
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>
>
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From frantisek at sumsal.cz  Fri Feb  7 17:15:48 2020
From: frantisek at sumsal.cz (=?UTF-8?B?RnJhbnRpxaFlayDFoHVtxaFhbA==?=)
Date: Fri, 7 Feb 2020 18:15:48 +0100
Subject: [Ci-users] Missing recent builds
In-Reply-To: <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
 <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
Message-ID: <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>

I see, thanks a lot for the explanation!

I guess the issue here is that in systemd we usually generate several 
dozens of builds per day, so leaving just last 20 cuts into the 'recent
history'. However, I absolutely get the space issue, so I'll reconfigure
all jobs to keep the artifacts for 30 days, and the jobs themselves for
60 days (and they shouldn't take much space and would help us to reschedule
them if they're needed again).

Anyway, if the need arises and a cleanup is necessary, could you remove
just the artifacts? It would leave us with the option of being able to
easily reschedule the job (as doing that manually is next to impossible
due to sheer amount of variables the GitHub plugin sets). 

Thanks!

On 2/7/20 5:43 PM, Vipul Siddharth wrote:
> When /var/lib/jenkins gets full (or about to), I leave just the last
> 20 builds and prune rest to clear up some space
> Since we advise to copy important things (like builds to artifacts), I
> didn't know they were important (leaving 20 to go back and address
> some issues)
> In case you don't want me messing with them, please let me know.
> Apologies for those.
> 
> On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>>
>> Hello,
>>
>> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
>> removed, even though they shouldn't be, according to the retention policy (which is set
>> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
>> I'm curious if there's some other retention policy we should be aware of, as this is kind of
>> annoying - the missing jobs are a few days old, some of them failed, but apart from not
>> being able to check the logs, we can't even reschedule them, because they don't exist
>> anymore.
>>
>> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
>> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
>> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
>> master.
>>
>> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
>>
>> Thank you,
>> Frantisek
>>
>> [0] systemd-ci-slave01
>>
>> --
>> PGP Key ID: 0xFB738CE27B634E4B
>>
>>
>>
> 
> 

-- 
PGP Key ID: 0xFB738CE27B634E4B

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200207/c985eddf/attachment-0003.sig>

From vipul at redhat.com  Fri Feb  7 17:20:06 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Fri, 7 Feb 2020 22:50:06 +0530
Subject: [Ci-users] Missing recent builds
In-Reply-To: <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
 <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
 <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>
Message-ID: <CANGkBUCdqMq3woMfjaq+mwJS0CPtZjKXPu2fWr7P-dCKhoahfg@mail.gmail.com>

Sure, thank you for understanding :)

On Fri, Feb 7, 2020 at 10:46 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>
> I see, thanks a lot for the explanation!
>
> I guess the issue here is that in systemd we usually generate several
> dozens of builds per day, so leaving just last 20 cuts into the 'recent
> history'. However, I absolutely get the space issue, so I'll reconfigure
> all jobs to keep the artifacts for 30 days, and the jobs themselves for
> 60 days (and they shouldn't take much space and would help us to reschedule
> them if they're needed again).
>
> Anyway, if the need arises and a cleanup is necessary, could you remove
> just the artifacts? It would leave us with the option of being able to
> easily reschedule the job (as doing that manually is next to impossible
> due to sheer amount of variables the GitHub plugin sets).
>
> Thanks!
>
> On 2/7/20 5:43 PM, Vipul Siddharth wrote:
> > When /var/lib/jenkins gets full (or about to), I leave just the last
> > 20 builds and prune rest to clear up some space
> > Since we advise to copy important things (like builds to artifacts), I
> > didn't know they were important (leaving 20 to go back and address
> > some issues)
> > In case you don't want me messing with them, please let me know.
> > Apologies for those.
> >
> > On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
> >>
> >> Hello,
> >>
> >> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
> >> removed, even though they shouldn't be, according to the retention policy (which is set
> >> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
> >> I'm curious if there's some other retention policy we should be aware of, as this is kind of
> >> annoying - the missing jobs are a few days old, some of them failed, but apart from not
> >> being able to check the logs, we can't even reschedule them, because they don't exist
> >> anymore.
> >>
> >> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
> >> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
> >> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
> >> master.
> >>
> >> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
> >>
> >> Thank you,
> >> Frantisek
> >>
> >> [0] systemd-ci-slave01
> >>
> >> --
> >> PGP Key ID: 0xFB738CE27B634E4B
> >>
> >>
> >>
> >
> >
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From kkanova at redhat.com  Wed Feb 26 13:53:01 2020
From: kkanova at redhat.com (Katerina Foniok)
Date: Wed, 26 Feb 2020 14:53:01 +0100
Subject: [Ci-users] No space left on device
Message-ID: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>

Hello guys,

our tests failed on `No space left on device` [1]. Could you please take a
look at that?

Thank you for your effort.
Have a nice day,
Katka

[1]
https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200226/8316b356/attachment-0003.html>

From vipul at redhat.com  Wed Feb 26 14:23:05 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Wed, 26 Feb 2020 19:53:05 +0530
Subject: [Ci-users] No space left on device
In-Reply-To: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
References: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
Message-ID: <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>

On Wed, Feb 26, 2020 at 7:23 PM Katerina Foniok <kkanova at redhat.com> wrote:
>
> Hello guys,
>
> our tests failed on `No space left on device` [1]. Could you please take a look at that?
Hi Katerina,
I have been cleaning slave04 around twice a day and now it has come to
a point where just parallel cleaning is not working.
You (or someone who has access to slave04) can  either clean
/home/devtools/workspace or work with me to tell me what can be
deleted and I will do it.
Sadly I didn't know whom to reach out to ask for this, so this has to
come to this :)
I have written your name in my self maintained "project to people" mapping :)

Please let me know which one works for you and I am just a ping away
to clean things for you
>
> Thank you for your effort.
> Have a nice day,
> Katka
>
> [1] https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users



-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From mafriedm at redhat.com  Wed Feb 26 16:34:21 2020
From: mafriedm at redhat.com (Maor Friedman)
Date: Wed, 26 Feb 2020 18:34:21 +0200
Subject: [Ci-users] No space left on device
In-Reply-To: <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>
References: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
 <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>
Message-ID: <CADUYHM-W3v3wDXAKpu+1EXXGQksp_oeAmOVKOyrwvnp8n-v+VQ@mail.gmail.com>

This has been resolved with the help of Vipul!

Thanks!

On Wed, Feb 26, 2020 at 4:24 PM Vipul Siddharth <vipul at redhat.com> wrote:

> On Wed, Feb 26, 2020 at 7:23 PM Katerina Foniok <kkanova at redhat.com>
> wrote:
> >
> > Hello guys,
> >
> > our tests failed on `No space left on device` [1]. Could you please take
> a look at that?
> Hi Katerina,
> I have been cleaning slave04 around twice a day and now it has come to
> a point where just parallel cleaning is not working.
> You (or someone who has access to slave04) can  either clean
> /home/devtools/workspace or work with me to tell me what can be
> deleted and I will do it.
> Sadly I didn't know whom to reach out to ask for this, so this has to
> come to this :)
> I have written your name in my self maintained "project to people" mapping
> :)
>
> Please let me know which one works for you and I am just a ping away
> to clean things for you
> >
> > Thank you for your effort.
> > Have a nice day,
> > Katka
> >
> > [1]
> https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
> > _______________________________________________
> > CI-users mailing list
> > CI-users at centos.org
> > https://lists.centos.org/mailman/listinfo/ci-users
>
>
>
> --
> Vipul Siddharth
> He/His/Him
> Fedora | CentOS CI Infrastructure Team
> Red Hat
> w: vipul.dev
>
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users
>
>

-- 
Maor Friedman
Senior Software Engineer, Application SRE team, Service Delivery
Red Hat
maor at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200226/810d6ac8/attachment-0003.html>

From frantisek at sumsal.cz  Fri Feb  7 16:34:46 2020
From: frantisek at sumsal.cz (=?UTF-8?B?RnJhbnRpxaFlayDFoHVtxaFhbA==?=)
Date: Fri, 7 Feb 2020 17:34:46 +0100
Subject: [Ci-users] Missing recent builds
Message-ID: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>

Hello,

I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
removed, even though they shouldn't be, according to the retention policy (which is set
to keep builds for 90 days in all affected jobs). As it's not the first time this happened
I'm curious if there's some other retention policy we should be aware of, as this is kind of
annoying - the missing jobs are a few days old, some of them failed, but apart from not
being able to check the logs, we can't even reschedule them, because they don't exist
anymore.

Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
master.

PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:

Thank you,
Frantisek

[0] systemd-ci-slave01

-- 
PGP Key ID: 0xFB738CE27B634E4B



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200207/60e6de13/attachment-0004.sig>

From vipul at redhat.com  Fri Feb  7 16:43:56 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Fri, 7 Feb 2020 22:13:56 +0530
Subject: [Ci-users] Missing recent builds
In-Reply-To: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
Message-ID: <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>

When /var/lib/jenkins gets full (or about to), I leave just the last
20 builds and prune rest to clear up some space
Since we advise to copy important things (like builds to artifacts), I
didn't know they were important (leaving 20 to go back and address
some issues)
In case you don't want me messing with them, please let me know.
Apologies for those.

On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>
> Hello,
>
> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
> removed, even though they shouldn't be, according to the retention policy (which is set
> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
> I'm curious if there's some other retention policy we should be aware of, as this is kind of
> annoying - the missing jobs are a few days old, some of them failed, but apart from not
> being able to check the logs, we can't even reschedule them, because they don't exist
> anymore.
>
> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
> master.
>
> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
>
> Thank you,
> Frantisek
>
> [0] systemd-ci-slave01
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>
>
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From frantisek at sumsal.cz  Fri Feb  7 17:15:48 2020
From: frantisek at sumsal.cz (=?UTF-8?B?RnJhbnRpxaFlayDFoHVtxaFhbA==?=)
Date: Fri, 7 Feb 2020 18:15:48 +0100
Subject: [Ci-users] Missing recent builds
In-Reply-To: <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
 <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
Message-ID: <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>

I see, thanks a lot for the explanation!

I guess the issue here is that in systemd we usually generate several 
dozens of builds per day, so leaving just last 20 cuts into the 'recent
history'. However, I absolutely get the space issue, so I'll reconfigure
all jobs to keep the artifacts for 30 days, and the jobs themselves for
60 days (and they shouldn't take much space and would help us to reschedule
them if they're needed again).

Anyway, if the need arises and a cleanup is necessary, could you remove
just the artifacts? It would leave us with the option of being able to
easily reschedule the job (as doing that manually is next to impossible
due to sheer amount of variables the GitHub plugin sets). 

Thanks!

On 2/7/20 5:43 PM, Vipul Siddharth wrote:
> When /var/lib/jenkins gets full (or about to), I leave just the last
> 20 builds and prune rest to clear up some space
> Since we advise to copy important things (like builds to artifacts), I
> didn't know they were important (leaving 20 to go back and address
> some issues)
> In case you don't want me messing with them, please let me know.
> Apologies for those.
> 
> On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>>
>> Hello,
>>
>> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
>> removed, even though they shouldn't be, according to the retention policy (which is set
>> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
>> I'm curious if there's some other retention policy we should be aware of, as this is kind of
>> annoying - the missing jobs are a few days old, some of them failed, but apart from not
>> being able to check the logs, we can't even reschedule them, because they don't exist
>> anymore.
>>
>> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
>> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
>> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
>> master.
>>
>> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
>>
>> Thank you,
>> Frantisek
>>
>> [0] systemd-ci-slave01
>>
>> --
>> PGP Key ID: 0xFB738CE27B634E4B
>>
>>
>>
> 
> 

-- 
PGP Key ID: 0xFB738CE27B634E4B

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200207/c985eddf/attachment-0004.sig>

From vipul at redhat.com  Fri Feb  7 17:20:06 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Fri, 7 Feb 2020 22:50:06 +0530
Subject: [Ci-users] Missing recent builds
In-Reply-To: <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
 <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
 <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>
Message-ID: <CANGkBUCdqMq3woMfjaq+mwJS0CPtZjKXPu2fWr7P-dCKhoahfg@mail.gmail.com>

Sure, thank you for understanding :)

On Fri, Feb 7, 2020 at 10:46 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>
> I see, thanks a lot for the explanation!
>
> I guess the issue here is that in systemd we usually generate several
> dozens of builds per day, so leaving just last 20 cuts into the 'recent
> history'. However, I absolutely get the space issue, so I'll reconfigure
> all jobs to keep the artifacts for 30 days, and the jobs themselves for
> 60 days (and they shouldn't take much space and would help us to reschedule
> them if they're needed again).
>
> Anyway, if the need arises and a cleanup is necessary, could you remove
> just the artifacts? It would leave us with the option of being able to
> easily reschedule the job (as doing that manually is next to impossible
> due to sheer amount of variables the GitHub plugin sets).
>
> Thanks!
>
> On 2/7/20 5:43 PM, Vipul Siddharth wrote:
> > When /var/lib/jenkins gets full (or about to), I leave just the last
> > 20 builds and prune rest to clear up some space
> > Since we advise to copy important things (like builds to artifacts), I
> > didn't know they were important (leaving 20 to go back and address
> > some issues)
> > In case you don't want me messing with them, please let me know.
> > Apologies for those.
> >
> > On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
> >>
> >> Hello,
> >>
> >> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
> >> removed, even though they shouldn't be, according to the retention policy (which is set
> >> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
> >> I'm curious if there's some other retention policy we should be aware of, as this is kind of
> >> annoying - the missing jobs are a few days old, some of them failed, but apart from not
> >> being able to check the logs, we can't even reschedule them, because they don't exist
> >> anymore.
> >>
> >> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
> >> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
> >> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
> >> master.
> >>
> >> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
> >>
> >> Thank you,
> >> Frantisek
> >>
> >> [0] systemd-ci-slave01
> >>
> >> --
> >> PGP Key ID: 0xFB738CE27B634E4B
> >>
> >>
> >>
> >
> >
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From kkanova at redhat.com  Wed Feb 26 13:53:01 2020
From: kkanova at redhat.com (Katerina Foniok)
Date: Wed, 26 Feb 2020 14:53:01 +0100
Subject: [Ci-users] No space left on device
Message-ID: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>

Hello guys,

our tests failed on `No space left on device` [1]. Could you please take a
look at that?

Thank you for your effort.
Have a nice day,
Katka

[1]
https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200226/8316b356/attachment-0004.html>

From vipul at redhat.com  Wed Feb 26 14:23:05 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Wed, 26 Feb 2020 19:53:05 +0530
Subject: [Ci-users] No space left on device
In-Reply-To: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
References: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
Message-ID: <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>

On Wed, Feb 26, 2020 at 7:23 PM Katerina Foniok <kkanova at redhat.com> wrote:
>
> Hello guys,
>
> our tests failed on `No space left on device` [1]. Could you please take a look at that?
Hi Katerina,
I have been cleaning slave04 around twice a day and now it has come to
a point where just parallel cleaning is not working.
You (or someone who has access to slave04) can  either clean
/home/devtools/workspace or work with me to tell me what can be
deleted and I will do it.
Sadly I didn't know whom to reach out to ask for this, so this has to
come to this :)
I have written your name in my self maintained "project to people" mapping :)

Please let me know which one works for you and I am just a ping away
to clean things for you
>
> Thank you for your effort.
> Have a nice day,
> Katka
>
> [1] https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users



-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From mafriedm at redhat.com  Wed Feb 26 16:34:21 2020
From: mafriedm at redhat.com (Maor Friedman)
Date: Wed, 26 Feb 2020 18:34:21 +0200
Subject: [Ci-users] No space left on device
In-Reply-To: <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>
References: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
 <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>
Message-ID: <CADUYHM-W3v3wDXAKpu+1EXXGQksp_oeAmOVKOyrwvnp8n-v+VQ@mail.gmail.com>

This has been resolved with the help of Vipul!

Thanks!

On Wed, Feb 26, 2020 at 4:24 PM Vipul Siddharth <vipul at redhat.com> wrote:

> On Wed, Feb 26, 2020 at 7:23 PM Katerina Foniok <kkanova at redhat.com>
> wrote:
> >
> > Hello guys,
> >
> > our tests failed on `No space left on device` [1]. Could you please take
> a look at that?
> Hi Katerina,
> I have been cleaning slave04 around twice a day and now it has come to
> a point where just parallel cleaning is not working.
> You (or someone who has access to slave04) can  either clean
> /home/devtools/workspace or work with me to tell me what can be
> deleted and I will do it.
> Sadly I didn't know whom to reach out to ask for this, so this has to
> come to this :)
> I have written your name in my self maintained "project to people" mapping
> :)
>
> Please let me know which one works for you and I am just a ping away
> to clean things for you
> >
> > Thank you for your effort.
> > Have a nice day,
> > Katka
> >
> > [1]
> https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
> > _______________________________________________
> > CI-users mailing list
> > CI-users at centos.org
> > https://lists.centos.org/mailman/listinfo/ci-users
>
>
>
> --
> Vipul Siddharth
> He/His/Him
> Fedora | CentOS CI Infrastructure Team
> Red Hat
> w: vipul.dev
>
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users
>
>

-- 
Maor Friedman
Senior Software Engineer, Application SRE team, Service Delivery
Red Hat
maor at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200226/810d6ac8/attachment-0004.html>

From frantisek at sumsal.cz  Fri Feb  7 16:34:46 2020
From: frantisek at sumsal.cz (=?UTF-8?B?RnJhbnRpxaFlayDFoHVtxaFhbA==?=)
Date: Fri, 7 Feb 2020 17:34:46 +0100
Subject: [Ci-users] Missing recent builds
Message-ID: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>

Hello,

I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
removed, even though they shouldn't be, according to the retention policy (which is set
to keep builds for 90 days in all affected jobs). As it's not the first time this happened
I'm curious if there's some other retention policy we should be aware of, as this is kind of
annoying - the missing jobs are a few days old, some of them failed, but apart from not
being able to check the logs, we can't even reschedule them, because they don't exist
anymore.

Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
master.

PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:

Thank you,
Frantisek

[0] systemd-ci-slave01

-- 
PGP Key ID: 0xFB738CE27B634E4B



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200207/60e6de13/attachment-0005.sig>

From vipul at redhat.com  Fri Feb  7 16:43:56 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Fri, 7 Feb 2020 22:13:56 +0530
Subject: [Ci-users] Missing recent builds
In-Reply-To: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
Message-ID: <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>

When /var/lib/jenkins gets full (or about to), I leave just the last
20 builds and prune rest to clear up some space
Since we advise to copy important things (like builds to artifacts), I
didn't know they were important (leaving 20 to go back and address
some issues)
In case you don't want me messing with them, please let me know.
Apologies for those.

On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>
> Hello,
>
> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
> removed, even though they shouldn't be, according to the retention policy (which is set
> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
> I'm curious if there's some other retention policy we should be aware of, as this is kind of
> annoying - the missing jobs are a few days old, some of them failed, but apart from not
> being able to check the logs, we can't even reschedule them, because they don't exist
> anymore.
>
> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
> master.
>
> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
>
> Thank you,
> Frantisek
>
> [0] systemd-ci-slave01
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>
>
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From frantisek at sumsal.cz  Fri Feb  7 17:15:48 2020
From: frantisek at sumsal.cz (=?UTF-8?B?RnJhbnRpxaFlayDFoHVtxaFhbA==?=)
Date: Fri, 7 Feb 2020 18:15:48 +0100
Subject: [Ci-users] Missing recent builds
In-Reply-To: <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
 <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
Message-ID: <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>

I see, thanks a lot for the explanation!

I guess the issue here is that in systemd we usually generate several 
dozens of builds per day, so leaving just last 20 cuts into the 'recent
history'. However, I absolutely get the space issue, so I'll reconfigure
all jobs to keep the artifacts for 30 days, and the jobs themselves for
60 days (and they shouldn't take much space and would help us to reschedule
them if they're needed again).

Anyway, if the need arises and a cleanup is necessary, could you remove
just the artifacts? It would leave us with the option of being able to
easily reschedule the job (as doing that manually is next to impossible
due to sheer amount of variables the GitHub plugin sets). 

Thanks!

On 2/7/20 5:43 PM, Vipul Siddharth wrote:
> When /var/lib/jenkins gets full (or about to), I leave just the last
> 20 builds and prune rest to clear up some space
> Since we advise to copy important things (like builds to artifacts), I
> didn't know they were important (leaving 20 to go back and address
> some issues)
> In case you don't want me messing with them, please let me know.
> Apologies for those.
> 
> On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>>
>> Hello,
>>
>> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
>> removed, even though they shouldn't be, according to the retention policy (which is set
>> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
>> I'm curious if there's some other retention policy we should be aware of, as this is kind of
>> annoying - the missing jobs are a few days old, some of them failed, but apart from not
>> being able to check the logs, we can't even reschedule them, because they don't exist
>> anymore.
>>
>> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
>> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
>> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
>> master.
>>
>> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
>>
>> Thank you,
>> Frantisek
>>
>> [0] systemd-ci-slave01
>>
>> --
>> PGP Key ID: 0xFB738CE27B634E4B
>>
>>
>>
> 
> 

-- 
PGP Key ID: 0xFB738CE27B634E4B

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200207/c985eddf/attachment-0005.sig>

From vipul at redhat.com  Fri Feb  7 17:20:06 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Fri, 7 Feb 2020 22:50:06 +0530
Subject: [Ci-users] Missing recent builds
In-Reply-To: <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>
References: <7e0487c3-3c74-bdd8-e11a-cb35a2206bde@sumsal.cz>
 <CANGkBUCejkTUOhWE=QWDz2M8dY3WR=VdSTJgCD2ogJ5o7gOseQ@mail.gmail.com>
 <ecb340fd-466c-a710-6777-98e825d16917@sumsal.cz>
Message-ID: <CANGkBUCdqMq3woMfjaq+mwJS0CPtZjKXPu2fWr7P-dCKhoahfg@mail.gmail.com>

Sure, thank you for understanding :)

On Fri, Feb 7, 2020 at 10:46 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
>
> I see, thanks a lot for the explanation!
>
> I guess the issue here is that in systemd we usually generate several
> dozens of builds per day, so leaving just last 20 cuts into the 'recent
> history'. However, I absolutely get the space issue, so I'll reconfigure
> all jobs to keep the artifacts for 30 days, and the jobs themselves for
> 60 days (and they shouldn't take much space and would help us to reschedule
> them if they're needed again).
>
> Anyway, if the need arises and a cleanup is necessary, could you remove
> just the artifacts? It would leave us with the option of being able to
> easily reschedule the job (as doing that manually is next to impossible
> due to sheer amount of variables the GitHub plugin sets).
>
> Thanks!
>
> On 2/7/20 5:43 PM, Vipul Siddharth wrote:
> > When /var/lib/jenkins gets full (or about to), I leave just the last
> > 20 builds and prune rest to clear up some space
> > Since we advise to copy important things (like builds to artifacts), I
> > didn't know they were important (leaving 20 to go back and address
> > some issues)
> > In case you don't want me messing with them, please let me know.
> > Apologies for those.
> >
> > On Fri, Feb 7, 2020 at 10:04 PM Franti?ek ?um?al <frantisek at sumsal.cz> wrote:
> >>
> >> Hello,
> >>
> >> I just noticed that many of the recent builds from our Jenkins slave[0] were unexpectedly
> >> removed, even though they shouldn't be, according to the retention policy (which is set
> >> to keep builds for 90 days in all affected jobs). As it's not the first time this happened
> >> I'm curious if there's some other retention policy we should be aware of, as this is kind of
> >> annoying - the missing jobs are a few days old, some of them failed, but apart from not
> >> being able to check the logs, we can't even reschedule them, because they don't exist
> >> anymore.
> >>
> >> Note: the retention policy is currently set unnecessarily high - that's a remnant from the last
> >> time this issue occurred, as I thought Jenkins is simply miscalculating the retention policy
> >> configuration. Cutting it to 45 or even 30 days should be an issue to conserve space on the
> >> master.
> >>
> >> PS: sorry for the double post, Vipul, I accidentally dropped the ML from To:
> >>
> >> Thank you,
> >> Frantisek
> >>
> >> [0] systemd-ci-slave01
> >>
> >> --
> >> PGP Key ID: 0xFB738CE27B634E4B
> >>
> >>
> >>
> >
> >
>
> --
> PGP Key ID: 0xFB738CE27B634E4B
>


-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From kkanova at redhat.com  Wed Feb 26 13:53:01 2020
From: kkanova at redhat.com (Katerina Foniok)
Date: Wed, 26 Feb 2020 14:53:01 +0100
Subject: [Ci-users] No space left on device
Message-ID: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>

Hello guys,

our tests failed on `No space left on device` [1]. Could you please take a
look at that?

Thank you for your effort.
Have a nice day,
Katka

[1]
https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200226/8316b356/attachment-0005.html>

From vipul at redhat.com  Wed Feb 26 14:23:05 2020
From: vipul at redhat.com (Vipul Siddharth)
Date: Wed, 26 Feb 2020 19:53:05 +0530
Subject: [Ci-users] No space left on device
In-Reply-To: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
References: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
Message-ID: <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>

On Wed, Feb 26, 2020 at 7:23 PM Katerina Foniok <kkanova at redhat.com> wrote:
>
> Hello guys,
>
> our tests failed on `No space left on device` [1]. Could you please take a look at that?
Hi Katerina,
I have been cleaning slave04 around twice a day and now it has come to
a point where just parallel cleaning is not working.
You (or someone who has access to slave04) can  either clean
/home/devtools/workspace or work with me to tell me what can be
deleted and I will do it.
Sadly I didn't know whom to reach out to ask for this, so this has to
come to this :)
I have written your name in my self maintained "project to people" mapping :)

Please let me know which one works for you and I am just a ping away
to clean things for you
>
> Thank you for your effort.
> Have a nice day,
> Katka
>
> [1] https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users



-- 
Vipul Siddharth
He/His/Him
Fedora | CentOS CI Infrastructure Team
Red Hat
w: vipul.dev



From mafriedm at redhat.com  Wed Feb 26 16:34:21 2020
From: mafriedm at redhat.com (Maor Friedman)
Date: Wed, 26 Feb 2020 18:34:21 +0200
Subject: [Ci-users] No space left on device
In-Reply-To: <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>
References: <CAAGHeEgiYEMXqZ9VrPmnRiW-=y1-xhAcym=T4=wZw+hPpH-RAw@mail.gmail.com>
 <CANGkBUCozrBox1i_Aq0KzkPRhMRxtrd6zDoXeB5aJh64JyqvSA@mail.gmail.com>
Message-ID: <CADUYHM-W3v3wDXAKpu+1EXXGQksp_oeAmOVKOyrwvnp8n-v+VQ@mail.gmail.com>

This has been resolved with the help of Vipul!

Thanks!

On Wed, Feb 26, 2020 at 4:24 PM Vipul Siddharth <vipul at redhat.com> wrote:

> On Wed, Feb 26, 2020 at 7:23 PM Katerina Foniok <kkanova at redhat.com>
> wrote:
> >
> > Hello guys,
> >
> > our tests failed on `No space left on device` [1]. Could you please take
> a look at that?
> Hi Katerina,
> I have been cleaning slave04 around twice a day and now it has come to
> a point where just parallel cleaning is not working.
> You (or someone who has access to slave04) can  either clean
> /home/devtools/workspace or work with me to tell me what can be
> deleted and I will do it.
> Sadly I didn't know whom to reach out to ask for this, so this has to
> come to this :)
> I have written your name in my self maintained "project to people" mapping
> :)
>
> Please let me know which one works for you and I am just a ping away
> to clean things for you
> >
> > Thank you for your effort.
> > Have a nice day,
> > Katka
> >
> > [1]
> https://ci.centos.org/view/Devtools/job/devtools-rh-che-rh-che-prcheck-dev.rdu2c.fabric8.io/2298/console
> > _______________________________________________
> > CI-users mailing list
> > CI-users at centos.org
> > https://lists.centos.org/mailman/listinfo/ci-users
>
>
>
> --
> Vipul Siddharth
> He/His/Him
> Fedora | CentOS CI Infrastructure Team
> Red Hat
> w: vipul.dev
>
> _______________________________________________
> CI-users mailing list
> CI-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users
>
>

-- 
Maor Friedman
Senior Software Engineer, Application SRE team, Service Delivery
Red Hat
maor at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200226/810d6ac8/attachment-0005.html>