June 2017 - CI-users - lists.centos.org

BREAKING CHANGE: Jenkins downtime 30-Jun-2017 20:00 EDT (01-Jul-2017 00:00 UTC)
by Brian Stinson 01 Jul '17

01 Jul '17

Hi Folks, We will be updating Jenkins to the latest LTS release and moving the service to a more performant node on June 30th 2017 at 20:00 EDT This will require a full restart , and a move of the jenkins volume which means jobs caught in the queue during the quiet period will _NOT_ be requeued. If there is anything we can do to make this process smoother please let us know. As usual we will have a 1hr quiet period before this to allow any other jobs to finish. Feel free to find us on #centos-devel on IRC or here on the ci-users list for questions or comments. Cheers! -- Brian Stinson brian(a)bstinson.com

1 2

using Jenkinsfiles in cico
by Karanbir Singh 29 Jun '17

29 Jun '17

Hi What do we need in order to support Jenkinsfiles in projects at cico ? regards, -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc

2 2

CI specific meeting
by Karanbir Singh 23 Jun '17

23 Jun '17

Hi Guys Since there is enough going on around CI itself, do we want to consider hosting a weekly or fortnightly meeting on irc, specifically driven towards ci.centos.org ? regards, -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc

7 9

[RFE] cobertura plugin
by Randy Barlow 23 Jun '17

23 Jun '17

Hello! I would love to be able to use the Cobertura Jenkins plugin[0] for my Bodhi tests. Any other coverage plugin that can consume my nose-cov coverage.xml output would be fine as well. Here[1] is an example of using it with JJB. This is certainly squarely in the "nice to have" category so no big deal if it doesn't happen. Thanks for considering! [0] https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin [1] https://github.com/RedHatQE/cinch/blob/ab535b6a61ac0d540d5c5cc8cabde9069c24…

2 1

Self Introduction: Sinny Kumari
by Sinny Kumari 21 Jun '17

21 Jun '17

Hello everyone! I am Sinny Kumari working at Red Hat and have recently joined this Mailing List. I have contributed in Fedora by packaging and testing. I am also part of Alternative Architectures group in Fedora and take care of some AltArches specific packages like librtas, libservicelog, opencryptoki, etc. Recently, I have also started working with Atomic-wg[1] group and working on enabling Atomic Host for aarch64 and ppc64le arches in Fedora. Meanwhile, also monitoring closely work happening around Fedora CI efforts in order to get it expanded to AltArches. So far, my contribution is around Fedora and would love to expand it to CentOS. I have talked to Brian Stinson and would like to help with expanding CentOS CI to Alternative Arches. I am also looking forward to help in general with other CentOS projects. [1] https://fedoraproject.org/wiki/Atomic_WG Thanks, Sinny Kumari ( https://sinny.io/ )

2 1

accessing duffy from apps.ci
by Karanbir Singh 21 Jun '17

21 Jun '17

hi, Ari mentioned that there were some problems getting to duffy from the apps.ci infra, can we get some more details around that ? Regards -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc

3 8

state of openstack on power and aarch64
by Karanbir Singh 20 Jun '17

20 Jun '17

hi guys, I know we've got a few of the RDO guys on here, so asking here. Is RDO on ppc8le at a state where we can deploy it ? could we then make it be a part of the cico cloud ? if so, is there a good way to request nodes on specific hypervisor's ? eg. if someone wanted a cloud vm, they get x86_64 only, and if they ask for it - ppc8le. Same question for aarch64 regards -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc

1 0

How to abort running CI jobs
by Vinzenz Feenstra 19 Jun '17

19 Jun '17

Hi, Our team uses the CentOS CI infrastructure together within GitHub PR to check those to be in shape. At times it happens that due to programming mistakes we would like to cancel/abort jobs. As those runs take usually up to 45 minutes or even longer or sometimes are just stuck because they are waiting for user input that ofc never is going to happen. Is there some way to stop them? In this particular case I am talking about the leapp-to project @ github ( https://github.com/leapp-to/prototype ) Thanks -- Vinzenz Feenstra Senior Software Developer Red Hat Czech

5 8

Helping with CentOS CI
by Karanbir Singh 16 Jun '17

16 Jun '17

Hi Guys, Ari and some of the other people in his team have offered to help with running and maintenance around CentOS Ci, which would be very welcome. In order to get the ball rolling, Ari can you and the guys maybe go ahead and post some details on yourself, the work you are looking to do and what sort of things you had in mind w.r.t helping on the CentOS CI bits ? regards, -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc

3 3

safe restart coming
by Karanbir Singh 14 Jun '17

14 Jun '17

hi Ari has disabled the jobs that were causing the problems for now. I've setup the jenkins instance to saferestart and judging by the openstack jobs running, this is going to happen in about 2 hrs time from now. that translates to : 02:50 14th Jun 2017 UTC 23:50 14th Jun 2017 UTC If the system load goes up again, I will disable the JMS plugins and do another restart, but hopefully there wont be a need for it. Regards -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc

1 0

Unexpected outage 17:00 UTC Today - Service Restored
by Brian Stinson 14 Jun '17

14 Jun '17

Hi Folks, Jenkins was leaking file descriptors and hit a limit today at 17:00 UTC, service was degraded for about 10 minutes, and service was fully restored at around 17:24. I've increased the open-files limit for jenkins and am working on tuning the garbage collector to mitigate this in the future. Thanks for your patience, and apologies for any inconvenience. -- Brian Stinson

5 13

Devicemapper error with Docker
by Ratnadeep Debnath 12 Jun '17

12 Jun '17

Hi all, We have been experiencing some devicemapper issues with Docker since the last few weeks on CentOS CI and CentOS in general. This issue is reported here: https://github.com/moby/moby/issues/23089, however the workaround does not work. Did anyone face similar issue? Does anyone know how to workaround it. Debug info ======== [root@nxx ~]# docker --version Docker version 1.12.6, build 1398f24/1.12.6 [root@nxx ~]# systemctl status docker -l ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Mon 2017-06-12 08:14:41 BST; 1min 22s ago Docs: http://docs.docker.com Process: 300 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY (code=exited, status=1/FAILURE) Main PID: 300 (code=exited, status=1/FAILURE) Jun 12 08:14:40 n41.pufty.ci.centos.org systemd[1]: Starting Docker Application Container Engine... Jun 12 08:14:40 n41.pufty.ci.centos.org dockerd-current[300]: time="2017-06-12T08:14:40.732651146+01:00" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]" Jun 12 08:14:40 n41.pufty.ci.centos.org dockerd-current[300]: time="2017-06-12T08:14:40.742206601+01:00" level=info msg="libcontainerd: new containerd process, pid: 308" Jun 12 08:14:41 n41.pufty.ci.centos.org dockerd-current[300]: time="2017-06-12T08:14:41.758615859+01:00" level=warning msg="devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section." Jun 12 08:14:41 n41.pufty.ci.centos.org dockerd-current[300]: time="2017-06-12T08:14:41.759174439+01:00" level=error msg="[graphdriver] prior storage driver \"devicemapper\" failed: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Can't set cookie dm_task_set_cookie failed" Jun 12 08:14:41 n41.pufty.ci.centos.org dockerd-current[300]: time="2017-06-12T08:14:41.759576207+01:00" level=fatal msg="Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Can't set cookie dm_task_set_cookie failed" Jun 12 08:14:41 n41.pufty.ci.centos.org systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE Jun 12 08:14:41 n41.pufty.ci.centos.org systemd[1]: Failed to start Docker Application Container Engine. Jun 12 08:14:41 n41.pufty.ci.centos.org systemd[1]: Unit docker.service entered failed state. Jun 12 08:14:41 n41.pufty.ci.centos.org systemd[1]: docker.service failed. Thanks, rtnpro -- Ratnadeep Debnath, https://www.waartaa.com GPG Fingerprint: 033C 8041 A0E9 CDBA 2E02 B785 2119 5486 F245 DFD6

1 0

Jenkins Gerrit Trigger problem, please investigate
by Attila Darazs 09 Jun '17

09 Jun '17

Dear CentOS CI admins, We realized that some of our jobs are not triggering properly when we submit changes to review.openstack.org. None of our jobs that is using this gerrit server triggers any more. We did not change anything in the jobs configs recently, the jobs are set up to use the "rdo-ci-openstack.org" gerrit server and trigger on any change. What's weird is that if we go to https://ci.centos.org/gerrit_manual_trigger/ then we can manually trigger the jobs that were supposed to be running automatically. Example test change: https://review.openstack.org/472267 Submitted: 2017-06-08 13:44 UTC One of the gates jobs that were supposed to trigger, but nothing happened: https://ci.centos.org/view/rdo/view/tripleo-gate/job/tripleo-quickstart-gat… Manually triggered at (with rdo-ci-openstack.org) 2017-06-08 13:58 UTC Triggered Builds 472267,1 tripleo-quickstart-gate-ocata-delorean-quick-basic #530 tripleo-quickstart-gate-master-delorean-quick-basic #2290 tripleo-quickstart-gate-newton-delorean-quick-basic #1453 As far as I can tell this was the last of our jobs that was still properly triggered by gerrit, 11 days ago: https://ci.centos.org/job/tripleo-quickstart-extras-gate-newton-delorean-fu… Please look into this, it seems to me that somehow the connection that listens to Gerrit events is down and it's important for us to have these gates running. Thanks, Attila

1 1

Unplanned Saferestart 14:35 UTC
by Brian Stinson 08 Jun '17

08 Jun '17

Hi Folks, To handle the Gerrit Triggering issue, I have issued a saferestart starting at 14:35 today. I expect this to finish up in the next hour or so. In the meantime jobs will build up in the queue and be executed shortly after jenkins returns. Thanks for your patience. -- Brian Stinson

2 2

Ongoing delayed triggers, and Github PR updates in CICO
by Brian Stinson 08 Jun '17

08 Jun '17

Hi Folks, Over the past few days, we've heard reports of delayed Github job triggers and PR status updates. Quite a few jobs are polling PRs instead of relying on the webhooks such that we are exhausting our Github API rate limit. We'll be reaching out to some of you individually if your jobs are affecting this, but I thought I'd give a single update on the situation here. If you have questions, please let us know. Cheers! -- Brian Stinson

1 0

Weird issue in CentOS CI GUI and jobs are taking longer time
by Budh Gurung 07 Jun '17

07 Jun '17

Hi Folks, Thanks for the wonderful CI platform. Recently, we are noticing that few of our jobs [1] are taking way longer than the usual time. Also, it is not notifying back to the PR which triggered those jobs. I noticed that on the list of executors view [2], one/many of the jobs are in the state of progress (see screenshot 1) and on clicking on the progress bar to go to the respective job [3](screenshot 2), that job seems to have completed quite a long back. We are also experiencing jobs taking a longer time in recent time. Anyone experiencing same? Any idea what is happening here? Regards, Budh Ram Gurung Software Engineer - Devtools

1 1

irregular execution of NM jobs
by Vladimir Benes 02 Jun '17

02 Jun '17

Hi all, not sure if it's expected but I see some irregularity in my jobs execution https://ci.centos.org/view/NetworkManager/job/NetworkManager-nm-1-4/ and others since job 55 it was executed every day at the same time but since then (May 25th) it was executed just two times. Any hints? CI itself doesn't seem to be super busy. thanks, Vladimir

2 1

Splitting a job and making dependencies
by Václav Pavlín 02 Jun '17

02 Jun '17

Hi all, I am very new to Jenkins, so please bear with me:) We have this JJB config: https://github.com/almighty/almighty-jobs/blob/master/devtools-ci-index.yaml Where there is this job-template: https://github.com/almighty/almighty-jobs/blob/master/devtools-ci-index.yam… The script basically copies content of git cloned directory and runs build script. If that run is successful it deploys the resulting docker image to OpenShift. We want to deploy images by tag - which corresponds to git commit hash, so it gets templated in by oc process. The problem is that there are templates which have more parameters and we have taken care of this with a small script in this repostory: https://github.com/openshiftio/saas/ So my goal is to use that saas tool to do the deploy instead of plain oc process > oc apply. As we have multiple templates which have different scripts, I'd like to avoid code duplication - I don't want to copy&paste the oc process/oc apply or the script calls around in the file. So my thinking to solve this was to create some kind of hierarchy - I found something about dependencies and something called multijob where a job would trigger a build phase (build a docker image) and a deploy phase (oc apply). The key part is that if any of the phases fail, whole job fails. Also, deploy phase must not be triggered when build phase fails. What would be the correct approach with JJB to solve this? Is this wrong thinking? Am I missing something from the Jenkins point of view? I know this would be ideally solved by pipelines, but I don't think that is an option at the moment. Looking forward to comments and suggestions, Vašek -- Red Hat Developer Tools team Brno, Czech Republic Phone: +420 739 666 824

2 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users June 2017