CI-users June 2017

ci-users@lists.centos.org

20 participants
18 discussions

Unexpected outage 17:00 UTC Today - Service Restored
by Brian Stinson 14 Jun '17

14 Jun '17

Hi Folks, Jenkins was leaking file descriptors and hit a limit today at 17:00 UTC, service was degraded for about 10 minutes, and service was fully restored at around 17:24. I've increased the open-files limit for jenkins and am working on tuning the garbage collector to mitigate this in the future. Thanks for your patience, and apologies for any inconvenience. -- Brian Stinson

5 13

Devicemapper error with Docker
by Ratnadeep Debnath 12 Jun '17

12 Jun '17

Hi all, We have been experiencing some devicemapper issues with Docker since the last few weeks on CentOS CI and CentOS in general. This issue is reported here: https://github.com/moby/moby/issues/23089, however the workaround does not work. Did anyone face similar issue? Does anyone know how to workaround it. Debug info ======== [root@nxx ~]# docker --version Docker version 1.12.6, build 1398f24/1.12.6 [root@nxx ~]# systemctl status docker -l ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Mon 2017-06-12 08:14:41 BST; 1min 22s ago Docs: http://docs.docker.com Process: 300 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE) Main PID: 300 (code=exited, status=1/FAILURE) Jun 12 08:14:40 n41.pufty.ci.centos.org systemd[1]: Starting Docker Application Container Engine... Jun 12 08:14:40 n41.pufty.ci.centos.org dockerd-current[300]: time="2017-06-12T08:14:40.732651146+01:00" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]" Jun 12 08:14:40 n41.pufty.ci.centos.org dockerd-current[300]: time="2017-06-12T08:14:40.742206601+01:00" level=info msg="libcontainerd: new containerd process, pid: 308" Jun 12 08:14:41 n41.pufty.ci.centos.org dockerd-current[300]: time="2017-06-12T08:14:41.758615859+01:00" level=warning msg="devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section." Jun 12 08:14:41 n41.pufty.ci.centos.org dockerd-current[300]: time="2017-06-12T08:14:41.759174439+01:00" level=error msg="[graphdriver] prior storage driver \"devicemapper\" failed: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Can't set cookie dm_task_set_cookie failed" Jun 12 08:14:41 n41.pufty.ci.centos.org dockerd-current[300]: time="2017-06-12T08:14:41.759576207+01:00" level=fatal msg="Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Can't set cookie dm_task_set_cookie failed" Jun 12 08:14:41 n41.pufty.ci.centos.org systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE Jun 12 08:14:41 n41.pufty.ci.centos.org systemd[1]: Failed to start Docker Application Container Engine. Jun 12 08:14:41 n41.pufty.ci.centos.org systemd[1]: Unit docker.service entered failed state. Jun 12 08:14:41 n41.pufty.ci.centos.org systemd[1]: docker.service failed. Thanks, rtnpro -- Ratnadeep Debnath, https://www.waartaa.com GPG Fingerprint: 033C 8041 A0E9 CDBA 2E02 B785 2119 5486 F245 DFD6

1 0

Jenkins Gerrit Trigger problem, please investigate
by Attila Darazs 09 Jun '17

09 Jun '17

Dear CentOS CI admins, We realized that some of our jobs are not triggering properly when we submit changes to review.openstack.org. None of our jobs that is using this gerrit server triggers any more. We did not change anything in the jobs configs recently, the jobs are set up to use the "rdo-ci-openstack.org" gerrit server and trigger on any change. What's weird is that if we go to https://ci.centos.org/gerrit_manual_trigger/ then we can manually trigger the jobs that were supposed to be running automatically. Example test change: https://review.openstack.org/472267 Submitted: 2017-06-08 13:44 UTC One of the gates jobs that were supposed to trigger, but nothing happened: https://ci.centos.org/view/rdo/view/tripleo-gate/job/tripleo-quickstart-gat… Manually triggered at (with rdo-ci-openstack.org) 2017-06-08 13:58 UTC Triggered Builds 472267,1 tripleo-quickstart-gate-ocata-delorean-quick-basic #530 tripleo-quickstart-gate-master-delorean-quick-basic #2290 tripleo-quickstart-gate-newton-delorean-quick-basic #1453 As far as I can tell this was the last of our jobs that was still properly triggered by gerrit, 11 days ago: https://ci.centos.org/job/tripleo-quickstart-extras-gate-newton-delorean-fu… Please look into this, it seems to me that somehow the connection that listens to Gerrit events is down and it's important for us to have these gates running. Thanks, Attila

1 1

Unplanned Saferestart 14:35 UTC
by Brian Stinson 08 Jun '17

08 Jun '17

Hi Folks, To handle the Gerrit Triggering issue, I have issued a saferestart starting at 14:35 today. I expect this to finish up in the next hour or so. In the meantime jobs will build up in the queue and be executed shortly after jenkins returns. Thanks for your patience. -- Brian Stinson

2 2

Ongoing delayed triggers, and Github PR updates in CICO
by Brian Stinson 08 Jun '17

08 Jun '17

Hi Folks, Over the past few days, we've heard reports of delayed Github job triggers and PR status updates. Quite a few jobs are polling PRs instead of relying on the webhooks such that we are exhausting our Github API rate limit. We'll be reaching out to some of you individually if your jobs are affecting this, but I thought I'd give a single update on the situation here. If you have questions, please let us know. Cheers! -- Brian Stinson

1 0

Weird issue in CentOS CI GUI and jobs are taking longer time
by Budh Gurung 07 Jun '17

07 Jun '17

Hi Folks, Thanks for the wonderful CI platform. Recently, we are noticing that few of our jobs [1] are taking way longer than the usual time. Also, it is not notifying back to the PR which triggered those jobs. I noticed that on the list of executors view [2], one/many of the jobs are in the state of progress (see screenshot 1) and on clicking on the progress bar to go to the respective job [3](screenshot 2), that job seems to have completed quite a long back. We are also experiencing jobs taking a longer time in recent time. Anyone experiencing same? Any idea what is happening here? Regards, Budh Ram Gurung Software Engineer - Devtools

1 1

irregular execution of NM jobs
by Vladimir Benes 02 Jun '17

02 Jun '17

Hi all, not sure if it's expected but I see some irregularity in my jobs execution https://ci.centos.org/view/NetworkManager/job/NetworkManager-nm-1-4/ and others since job 55 it was executed every day at the same time but since then (May 25th) it was executed just two times. Any hints? CI itself doesn't seem to be super busy. thanks, Vladimir

2 1

Splitting a job and making dependencies
by Václav Pavlín 02 Jun '17

02 Jun '17

Hi all, I am very new to Jenkins, so please bear with me:) We have this JJB config: https://github.com/almighty/almighty-jobs/blob/master/devtools-ci-index.yaml Where there is this job-template: https://github.com/almighty/almighty-jobs/blob/master/devtools-ci-index.yam… The script basically copies content of git cloned directory and runs build script. If that run is successful it deploys the resulting docker image to OpenShift. We want to deploy images by tag - which corresponds to git commit hash, so it gets templated in by oc process. The problem is that there are templates which have more parameters and we have taken care of this with a small script in this repostory: https://github.com/openshiftio/saas/ So my goal is to use that saas tool to do the deploy instead of plain oc process > oc apply. As we have multiple templates which have different scripts, I'd like to avoid code duplication - I don't want to copy&paste the oc process/oc apply or the script calls around in the file. So my thinking to solve this was to create some kind of hierarchy - I found something about dependencies and something called multijob where a job would trigger a build phase (build a docker image) and a deploy phase (oc apply). The key part is that if any of the phases fail, whole job fails. Also, deploy phase must not be triggered when build phase fails. What would be the correct approach with JJB to solve this? Is this wrong thinking? Am I missing something from the Jenkins point of view? I know this would be ideally solved by pipelines, but I don't think that is an option at the moment. Looking forward to comments and suggestions, Vašek -- Red Hat Developer Tools team Brno, Czech Republic Phone: +420 739 666 824

2 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users June 2017