CI-users September 2018

ci-users@lists.centos.org

18 participants
16 discussions

[apps.ci.centos.org] Service degradation 27-Sept-2018 to 28-Sept-2018
by Brian Stinson 28 Sep '18

28 Sep '18

Summary: A large subset of the application nodes in apps.ci.centos.org were placed in an unschedulable state around 13h00 UTC on September 27th. Nodes were rebooted and service was partially restored, but new behavior was exhibited overnight. Pods were able to schedule on the nodes but DNS was not functional. DNS service was restored at around 15h00 UTC on September 28th. Timeline: 27-Sept-2018 13h00 UTC - 28-Sept-2018 15h00 UTC Root Cause: A previously applied update (applied around 17-August) to selinux-policy caused some files to be relabeled. We did not, at the time, schedule a reboot, but routine restarts of the docker service caused the nodes to enter a degraded state. Further file relabels caused the node boot process to complete, but also in a degraded state. Recovery: Completed the rest of the pending updates, and rebooted the nodes to clear the node-schedulable degradation. (27-Sept) Triggered a full autorelabel and rebooted the nodes to clear the node-boot degradation. (28-Sept) Preventative Measures: - Consider rebooting the nodes more often, perhaps on a regular schedule to catch OS upgrade problems - Complete the openshift-monitoring EPIC in the CI backlog, which will add better checks for DNS. Thank you very much for your patience during this outage. -- Brian Stinson CentOS CI Infrastructure Team

1 0

Adding users to the Admin list for github PR whitelist
by Jakub Hrozek 28 Sep '18

28 Sep '18

Hi, I’m sure I’m doing something silly, but I’m not able to add two new SSSD team members to the Admin list on ci.centos.org so that the CI runs on their PRs without having to be explicitly approved. Here’s what I did: - I went to: https://ci.centos.org/ - log in as the sssd user - click on the ‘sssd’ project - click on the ‘sssd-CentOS7’ project - click on configure on the left - select the “Build Triggers” tab - there is a box called “Admin list” which contains usernames - add the usernames there - click “Save” and “Apply” on the lower bar But still, both users’ PRs are being “gated” by the centos-ci user. Am I missing something? Does something need to be configured on the github side as well perhaps? btw everything works for users who were added some time ago to the whitelist..but I’ve inherited the maintenance of the SSSD ci.centos.org so this is the first time I’m adding new developers myself. Thank you for your help.

1 0

DNS resolution issues in OCP
by Jonathan Lebon 27 Sep '18

27 Sep '18

Anyone else encountering DNS resolution issues? Seeing this coming out of one of our apps: ``` Traceback (most recent call last): File "/usr/lib/python3.5/site-packages/urllib3/connection.py", line 141, in _new_conn (self.host, self.port), self.timeout, **extra_kw) File "/usr/lib/python3.5/site-packages/urllib3/util/connection.py", line 60, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/lib64/python3.5/socket.py", line 733, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -3] Temporary failure in name resolution During handling of the above exception, another exception occurred: ... Traceback (most recent call last): File "/usr/lib/python3.5/site-packages/requests/adapters.py", line 440, in send timeout=timeout File "/usr/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/lib/python3.5/site-packages/urllib3/util/retry.py", line 388, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /user (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7febf8618b00>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)) ``` And indeed, running a `getent ahostsv4 api.github.com` returns rc=2 in a running container. Even `ci.centos.org` is not resolving.

1 0

Triggering job after PR close
by Tibor Dancs 25 Sep '18

25 Sep '18

Hello, We are facing a curious issue. We have a job that needs to be triggered when PR is closed (either merge or close). Unfortunately Jenkins Github Pull Request Builder plugin does not support that event (yet?) so there are no triggers available for PR close (even though it has triggers for commit). Do you have any experience with this particular problem? Have you ever handled these github events or made a webtask hooking those webhooks? Any information is welcome. Tibor Dancs

2 1

Need centos-ci approval to collaborate for minishift/minishift-fedora-iso
by Praveen Kumar 24 Sep '18

24 Sep '18

Hi, I followed all the steps provided to centos wiki[0] and now the only thing remaining is adding `centos-ci` as a collaborator for minishift/minishift-fedora-iso[1] repo. I already added `centos-ci` user to collaborator but approval is pending, can someone do it soon. [0] https://wiki.centos.org/QaWiki/CI/GithubIntegration [1] https://github.com/minishift/minishift-fedora-iso -- Praveen Kumar http://fedoraproject.org/wiki/User:Kumarpraveen http://fedoraproject.org/ http://kumar-pravin.blogspot.com

1 0

How to delete old jobs?
by Randy Barlow 17 Sep '18

17 Sep '18

I recently replaced a job with some new jobs in my JJB template, but the old job remains in the CI system (and still runs): https://ci.centos.org/job/bodhi-bodhi/ How do I delete this no-longer-needed job?

2 2

Can jobs triggered by github triggered jobs set status on github easily?
by Randy Barlow 15 Sep '18

15 Sep '18

Greetings! I'm interested in sprucing up my JJB template[0] so that the following happens: 0) When a pull request is opened, a job is triggered that builds a container with the contents of the pull request. The results of this are reported to the pull request. 1) Success of that first job triggers a variety of jobs that all use that container and test different things. I would like these to run in parallel and I would like them to each also report back to the pull request. I would like each of these to have their own "status-context" that describes what they tested, if that makes sense. I see that there are a variety of ways to have jobs trigger other jobs, but what I don't know is how the subsequent jobs can set the "status-context" since they wouldn't be triggered from the pull request itself (outside of me manually using curl to GitHub, which I'd rather not do). Is there a nice clean/recommended way to achieve this with JJB? [0] https://github.com/fedora-infra/bodhi/blob/develop/devel/ci/githubprb-proje…

2 1

showing more than one check in github ui from ci.centos.org
by Bamacharan Kundu 14 Sep '18

14 Sep '18

Hi, We are running unit tests and functional tests on ci.centos.org[1][2] for container-pipeline-service[3] repository. Both the jobs are set to be triggered on PR raised on the repository. Now whenever a PR is raised, both jobs runs and overrides the check status for each other. So, we can see status of only one check at one time. Is there a way we can show both the checks on github UI, with their status? [1]https://ci.centos.org/view/Container/job/centos-container-pipeline-servic… [2]https://ci.centos.org/view/Container/job/centos-container-pipeline-servic… [3]https://github.com/CentOS/container-pipeline-service Regards Bamacharan Kundu

4 5

apps.ci.centos.org registry service degradtaion
by Brian Stinson 13 Sep '18

13 Sep '18

Hi Folks, I'm seeing a few issues with the internal registry on apps.ci.centos.org I'm going to work over the next couple of hours to get some things moved around and redeployed. You may notice some trouble pushing/pulling containers in the meantime. I'll give updates as they happen. Cheers! --Brian

1 1

Unplanned duffy outage ongoing
by Brian Stinson 13 Sep '18

13 Sep '18

Hi All, We're currently experiencing an isolated outage in duffy. I've paused Jenkins temporarily while the pools catch back up. --Brian

1 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users September 2018