On Sep 13 12:28, Brian Stinson wrote:
Hi Folks,
I'm seeing a few issues with the internal registry on apps.ci.centos.org
I'm going to work over the next couple of hours to get some things moved around and redeployed. You may notice some trouble pushing/pulling containers in the meantime.
I'll give updates as they happen.
Cheers!
--Brian _______________________________________________ Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Hi All,
We should be back now, here's what happened:
- The registry service was hosted on one of our infra nodes, backed by local storage
- As part of regular cleanup cronjobs we *usually* try to prune out any orphaned image layers from the registry (if the layers are still in use, or still referenced in a tag we keep them around)
- The pruning process got stuck, and left the registry process in uninterruptable sleep.
To fix this we:
- Converted the registry to use shared storage so it can be scheduled/rescheduled across any of the infra nodes
- Restarted the infra node that was having trouble
- Ran a manual pruning
- Restored the registry service
If you notice any trouble because of this please let me know here or in #centos-devel on freenode
Thanks all!
-- Brian Stinson CentOS CI Infrastructure Team