[Ci-users] apps.ci.centos.org registry service degradtaion
Brian Stinson
brian at bstinson.com
Thu Sep 13 20:03:51 UTC 2018
On Sep 13 12:28, Brian Stinson wrote:
>
> Hi Folks,
>
> I'm seeing a few issues with the internal registry on apps.ci.centos.org
>
>
> I'm going to work over the next couple of hours to get some things moved
> around and redeployed. You may notice some trouble pushing/pulling
> containers in the meantime.
>
> I'll give updates as they happen.
>
> Cheers!
>
> --Brian
> _______________________________________________
> Ci-users mailing list
> Ci-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users
Hi All,
We should be back now, here's what happened:
- The registry service was hosted on one of our infra nodes, backed by
local storage
- As part of regular cleanup cronjobs we *usually* try to prune out any
orphaned image layers from the registry (if the layers are still in
use, or still referenced in a tag we keep them around)
- The pruning process got stuck, and left the registry process in
uninterruptable sleep.
To fix this we:
- Converted the registry to use shared storage so it can be
scheduled/rescheduled across any of the infra nodes
- Restarted the infra node that was having trouble
- Ran a manual pruning
- Restored the registry service
If you notice any trouble because of this please let me know here or in
#centos-devel on freenode
Thanks all!
--
Brian Stinson
CentOS CI Infrastructure Team
More information about the Ci-users
mailing list