On Sep 13 12:28, Brian Stinson wrote: > > Hi Folks, > > I'm seeing a few issues with the internal registry on apps.ci.centos.org > > > I'm going to work over the next couple of hours to get some things moved > around and redeployed. You may notice some trouble pushing/pulling > containers in the meantime. > > I'll give updates as they happen. > > Cheers! > > --Brian > _______________________________________________ > Ci-users mailing list > Ci-users at centos.org > https://lists.centos.org/mailman/listinfo/ci-users Hi All, We should be back now, here's what happened: - The registry service was hosted on one of our infra nodes, backed by local storage - As part of regular cleanup cronjobs we *usually* try to prune out any orphaned image layers from the registry (if the layers are still in use, or still referenced in a tag we keep them around) - The pruning process got stuck, and left the registry process in uninterruptable sleep. To fix this we: - Converted the registry to use shared storage so it can be scheduled/rescheduled across any of the infra nodes - Restarted the infra node that was having trouble - Ran a manual pruning - Restored the registry service If you notice any trouble because of this please let me know here or in #centos-devel on freenode Thanks all! -- Brian Stinson CentOS CI Infrastructure Team