On Mon, Dec 5, 2022 at 9:10 AM Fabian Arrotin <arrfab at centos.org> wrote: > > On 05/12/2022 12:45, Peter Georg wrote: > > On 02/12/2022 09.13, Fabian Arrotin wrote: > >> Just for awareness : I noticed this morning through monitoring that > >> the cache/proxy used by kojid builders (needed to reach out to > >> gitlab/etc) was down during the night, so while kojihub itself was > >> available, the kojid nodes weren't even checking in. > >> > >> I quickly reconfigured (with ansible) another caching proxy and so > >> restored connectivity. If you noticed some build issues during some > >> hours, please resubmit these, but it seems most were just queued and > >> so picked up "normally" as soon as connection was restored. > > > > Since approx. 1st of December I've been noticing issues running cbs > > commands within GitLab CI. Most of the cbs commands are running into a > > time out, no connection can be established. No tasks are ever created. > > This issue still exists. > > > > Might this issue be related to the cache/proxy used for the kojid nodes? > > No, not related, but last week we also suffered from massive remote koji > api calls , that was taking cbs/koji down on its knees. > After having discussed with Fedora infra team colleagues, they said they > had the same issue the week[s] before and had to just drop IP ranges to > ensure koji.fedoraproject.org was back online. > I quickly took same emergency measure (had really other things to do) to > restore cbs.centos.org availability and so just also dropped some > ranges. Probably gitlab is hosted in one of these ranges ? > > In all cases, it's better to open an infra ticket (in parallel, as it's > also good to discuss on the centos-devel list too) > GitLab.com is hosted in Google Cloud on Google Kubernetes Engine. If you blocked GCP IP addresses, then yes, that would happen. -- 真実はいつも一つ！/ Always, there's only one truth!