October 2016 - CI-users - lists.centos.org

Fix for the Jenkins browser memory leak on cico?
by Laurentiu Pancescu 24 Jan '17

24 Jan '17

Hi there, My proposed fix for Jenkins causing memory leaks in all major browsers has now been accepted upstream. [1] Is there any chance of patching ci.centos.org to include it? Regards, Laurențiu [1] https://github.com/jenkinsci/jenkins/pull/2539

4 5

Jenkins plugin
by Marek Skalický 01 Nov '16

01 Nov '16

Hi, is it possible to install Rebuild Plugin [1] in CentOS CI? (it is useful to rebuild Github PR job runs without polluting Github conversation by trigger keywords) Thanks, M. [1] https://wiki.jenkins-ci.org/display/JENKINS/Rebuild+Plugin

2 1

Emergency Maintenance Window: jumphost -> Immediately, Other CI Hosts -> 1AM UTC
by Brian Stinson 27 Oct '16

27 Oct '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Folks, We are announcing an emergency maintenance window to apply some important security updates to the machines in CI. We plan to reboot the jumphost as soon as we can. If you are logged into any CI host from outside, your connection may drop. (This will not affect any jobs or connections internal to CI) At 27-Oct-2016 1AM UTC, we will be rebooting the jenkins agents. We will start a quiet period 2 hours beforehand so jobs can clear out. Thank you for your patience while we work through this window. If there are any questions, please find us here or in #centos-devel on freenode. Cheers! - -- Brian Stinson CentOS CI Infrastructure Team -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJYELgyAAoJEIMGvNKzCweMj7kP/2jAPFB4UTopkmo6yPhrJX77 dnhuqWMe3souh88wzI3EmnauFjH6bAum3BLg/preRMvov3Dv6NuMPdkRP5Zn1lIF +u30jUg2YP8Z/k+vYnedNnXGJFjda1JCp2sLsF9n5D+Nht08ReIID8d85lPwoKE8 WKkfr5+NVPkXjbvrw4VQ++3vJrOQHF8C/aO+1p1Eskg2nr753zUvTC4/h9fWgEQh RQ2G7T9xNJxYCPoJCtIV/RV+xVaMJfkmLznfZHJEjmsM9YPe9vqv+cOiIg/UZWF7 0wQUT/E6DAG5Y00cVJXXP1HkPgqttYmyOrcq5P23k1wDInu2HpEEWGO7IwD6MVlr rzGGhOxTxt8c8e1Mz1Ia6xSFVqid/kErc+4fHH+H8mp/U9LVNIlur1UCveIdAWla hJTLqIKHEEuapkxd2asNutHGIwC1tm4+ZzfuzOQWJlzLAZJzmChDNp/VHPIzTMQc OWd2qLQflWglhDJIOHyvzB1yfClBLr2tURfCWF9/qc7R90rm+qypSOYbLnvCu6r8 PFOBvq8xFvG4xHTjJo2QCec8oW1Za3TzzV6lmT4rw/VJeOCqcmZaoEmsVEWssayP DvzPhVyEmowL3jluLNStC2Tou5KmVd9j6l+HTNpNXJVosMjA0Xg9WAvuynOqjPhO 25Fpv54NC0a6mue8q/AT =EStS -----END PGP SIGNATURE-----

2 1

CI Maintenance Window 27-May-2016 01h00 - 02h00 UTC (26-May-2016 21h00 - 22h00 EDT)
by Brian Stinson 24 Oct '16

24 Oct '16

Hi Folks, We will be having a maintenance window starting at 1AM UTC on Friday, May 27th to do the following work: - Upgrade to the latest Jenkins LTS - Upgrade all plugins (including the Github Plugin) to their latest versions - Install the Pipeline Plugin (bug: 10825) - Update the ansible version on slave01.ci.centos.org 1.9.2 -> 1.9.6 - Reboot the storage node The following services will be affected: - ci.centos.org: Jenkins Frontend - artifacts.ci.centos.org: File availability As usual we will have a quiet period starting 1 hour before in order to let pending jobs clear out. If there are any questions please let us know. Cheers! -- Brian Stinson CentOS CI Infrastructure Team

5 10

networking down?
by Colin Walters 21 Oct '16

21 Oct '16

A lot of the https://ci.centos.org/view/Atomic/ jobs are failing to DNS resolve github.com.

2 1

Introducing Linch-Pin, Hybrid cloud provisioning using ansible
by Clint Savage 20 Oct '16

20 Oct '16

Hi all, Recently, as part of the CentOS PaaS SIG and Continuous Infra, We've been working on some basic tooling for CI. One of the tools that came out of this work, is Linch-Pin. It's a fairly simple tool for multi-cloud provisioning that's starting to get some traction. Currently, it's being used to provision Duffy instances inside for one Jenkins job in CentOS CI[1], but can do so much more. If you're interested in learning more about it, or trying it out, I'd love to hear your take. It's all open source and ready for you to try it out. I've written a blog post about Linch-Pin, and plan to do a series over the coming months. Check out the first installment in the series. http://sexysexypenguins.com/posts/introducing-linch-pin/ A link to the project is in the blog post, but also included below[2]. Additionally, there is some documentation[3], which is in the process of being improved. I'd love feedback on the blog post. And we're obviously interested in feedback on the project as well. Cheers, herlo 1- https://ci.centos.org/job/origin-from-source-0-test-matrix/ 2- https://github.com/CentOS-PaaS-SIG/linch-pin 3- http://linch-pin.readthedocs.io

1 0

Jenkins Job Builder is now available on the CI Agents
by Brian Stinson 20 Oct '16

20 Oct '16

Hi Folks, We finished up the packaging and have installed Jenkins Job Builder 1.6.1 on slave01.ci.c.o Cheers! -- Brian Stinson CentOS CI Infrastructure Team

3 2

Jenkins Gerrit comment trigger broken again
by Attila Darazs 18 Oct '16

18 Oct '16

Hi folks, I think the Gerrit Trigger plug-in got updated again (to 2.22.0) and broke the retriggering of jobs by Gerrit comments. If it got updated please downgrade to 2.21.1. If it's this version, something is still broken, but then I don't know what. :) This is our conversation in #centos about it last time this happened (on the 6th of September): (01:43:13 PM) adarazs: https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger -> Version 2.22.0 (released Aug 17 2016) -> JENKINS-30821 - Add comment-added comment as job parameter (pull #295) -> https://issues.jenkins-ci.org/browse/JENKINS-30821 (01:43:13 PM) adarazs: -> looks to me like they were messing with this exact part of the plug-in and this is a contender for having this specific error appear while trying to fetch the comment that triggered the job. (01:43:25 PM) arrfab: adarazs: version is 2.22.0 so it seems the last one (01:44:35 PM) adarazs: it might help to downgrade to Version 2.21.1 (released Jun 07 2016) -Attila

2 2

Setting up JJB service jobs in CICO
by Ratnadeep Debnath 15 Oct '16

15 Oct '16

Hi, I want to create a job in ci.centos.org, which creates other Jenkins jobs, using Jenkins Job Builder. However, JJB is not installed on the Jenkins slaves or master node. I'd like to know what would be the best way to create such service jobs in CICO. Thanks, rtnpro

4 3

Refactoring the CI test for Vagrant images
by Laurentiu Pancescu 14 Oct '16

14 Oct '16

Hi there, I heard that the Atomic team would be potentially interested in reusing the cico test for our Vagrant images centos/6 and centos/7, but their images are not built in CBS. I refactored cloudinstance-vagrant-test to accept a single parameter, BUILD_INFO, containing a JSON string (an array of hashes containing build information). This should allow other projects to use it if desired, since it doesn't parse the CBS logs anymore. This part was now moved to cloudinstance-vagrant-build and cloudinstance-vagrant-manual-build, which generate a JSON string like the following pretty-printed equivalent: [ { "major_release": "6", "provider": "virtualbox", "url": "https://cbs.centos.org/kojifiles/work/tasks/4846/114846/centos-6-1-1.x86_64…" }, { "major_release": "6", "provider": "libvirt", "url": "https://cbs.centos.org/kojifiles/work/tasks/4846/114846/centos-6-1-1.x86_64…" }, { "major_release": "7", "provider": "libvirt", "url": "https://cbs.centos.org/kojifiles/work/tasks/4848/114848/centos-7-1-1.x86_64…" }, { "major_release": "7", "provider": "virtualbox", "url": "https://cbs.centos.org/kojifiles/work/tasks/4848/114848/centos-7-1-1.x86_64…" } ] The scripts are in https://github.com/lpancescu/cloudinstance-vagrant-cico-util. Hope this helps, Laurențiu

3 2

Infinite loop with multiple repos?
by Laurentiu Pancescu 13 Oct '16

13 Oct '16

Hello, while adapting the cloudinstance-vagrant-manual-build job on ci.c.o, I thought to skip the manual cloning of a second repo via shell scripts and add a second repo via the Jenkins UI (I expected it would be cloned into a sibling directory to the directory hosting the clone for the first repo, but it seems to have just overwritten it). When I ran the job, it not only failed, but retried the job immediately, causing some sort of infinite loop. Cancelling the current job was useless, since a new one started immediately, and I ended with more than 250 failed jobs in just 1-2 minutes; I had to disable the project to stop it. I only had a trigger to start cloudinstance-vagrant-test on success - no idea why it retried again and again... Is that expected behavior? I switched to manual git cloning, this works without issues. Laurentiu

1 0

ci.centos.org is now running the latest Jenkins LTS (Jenkins 2)
by Brian Stinson 13 Oct '16

13 Oct '16

Hi Folks, During the migration this week we went ahead and migrated our Jenkins instance to the latest LTS (which means Jenkins 2!). The core functionality is completely backwards compatible with Jenkins 1, and we kept all of our plugins pinned to the versions we had running before. Just a reminder for the only known effect: - If you're using JJB to configure some of your jobs you _may_ need to set 'query_plugins_info = False' in the [jenkins] section of your config file. If you notice any trouble, please let us know here. Cheers! -- Brian Stinson CentOS CI Infrastructure Team

2 1

[infra] : Hardware migration status (full/correct version)
by Fabian Arrotin 11 Oct '16

11 Oct '16

As announced some time ago ( https://lists.centos.org/pipermail/centos-announce/2016-September/022065.ht… ) , there was a big hardware migration/DC relocation happening yesterday that impacted several of our public services. Here is a small report of that migration, that finally took less time that previously announced. * 2016-10-10 - 1pm UTC : we started to preconfigure all changes, and powered down all the services in correct order * 2016-10-10 - 2pm => 6pm UTC : hardware was moved/migrated * 2016-10-10 - 6pm => 8pm UTC : hardware was (in serial order) plugged back in and ipmi/oob interfaces reconfigured, and network started to be validated * 2016-10-10 - 8:30pm UTC : https://accounts.centos.org was back online and operational * 2016-10-10 - 9pm UTC : https://cbs.centos.org was back online but builders were still disabled for validation * 2016-10-10 - 9:45pm UTC : https://cbs.centos.org : builders were enabled and working again * 2016-10-11 - 3:30am UTC : https://ci.centos.org now fully operational (after some network reconfiguration needed post-migration, and validations Thanks a lot to Brian Stinson (on-site) and DC people who helped for that migration (Matt, Jon, Nathan, Nina, Phil) Should you encounter an issue , feel free to discuss with us in #centos-devel on irc.freenode.net PS : sorry for the previous and incomplete mail : sent by accident before finished and probably due to lack of coffee and short night ... -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

3 2

[infra] : Hardware migration status
by Fabian Arrotin 11 Oct '16

11 Oct '16

As announced some time ago ( https://lists.centos.org/pipermail/centos-announce/2016-September/022065.ht… ) , there was a big hardware migration/DC relocation happening yesterday that impacted several of our public services. Here is a small report of that migration, that finally took less time that previously announced. * 2016-10-10 - 3pm UTC : we started to preconfigure all changes, and powered down all the services in correct order * 2016-10-10 - 3pm UTC * 2016-10-10 - 3pm UTC * 2016-10-10 - 3pm UTC * 2016-10-10 - 3pm UTC -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

1 0

OpenStack cluster in ci.centos.org
by Karanbir Singh 06 Oct '16

06 Oct '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Folks, If anyone is using the openstack infra in ci.centos.org - please note that when the ci hardware is relocated this monday morning on the 10th Oct, the openstack cluster will NOT be coming back online. Our plan is to reinstall the entire openstack hypervisors and redeploy with neuton the following day and bringup an entirely clear cluster. If anyone has content there that they need to preserve, please back that up and shutdown the VMs before Monday morning UTC 10th October. Regards, - -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJX9lvpAAoJEI3Oi2Mx7xbtXosIAKXfGzj81JhKe6xbqOxp1vjE bTb3/2BK0EJxLzD2QGPuXmFMzgZf5him+cxJUu/jjh5pGoPSgto4rEu1UzBLhapy RzQvfCn9e877hhwUaPmBrJyaNlsfF8mIKKtoXCqwYKXRGvka3cJbXTp0WbsTjEmi +JEOWKt3cYcDio7Wj+cJMG8mqEXmgDb1pUxiiQPnzrx0BqbG/DUg0O42Oj+AMqPX pI4HI+jUZRtMooYLd9PoGdZsAZf8Cxy/dXFPJz1zy9C+iZcDFCt0vY8OvxDUuEfA N41dgEpxapYN81l9BSz9fXOW8kc+AzUIxkFI5BearcrrFVkSkt77QGRWzr3NnwA= =Z5Qz -----END PGP SIGNATURE-----

1 0

[Infra] - Major outage for several services (please read)
by Fabian Arrotin 05 Oct '16

05 Oct '16

As pre-announced earlier this year (https://lists.centos.org/pipermail/centos-devel/2016-May/014792.html) we'll have to move a part of our existing hardware to a new DC. That means that the following public services will be powered off and unreachable : - https://cbs.centos.org (Koji build farm front end and also builders/storage nodes behind) - https://accounts.centos.org (auth backend) - https://ci.centos.org (jenkins-driven CI environment and all nodes in that dedicated/isolated environment) What does that mean ? - Nobody from the SIGs (https://wiki.centos.org/SpecialInterestGroup) will be able to submit build jobs/tasks (so no update at all during that migration) - No CI job will be able to track CBS nor external repositories (and so that means also that http://artifacts.ci.centos.org will be unreachable) - Not possible for existing users to update their details on accounts.centos.org, and not possible to create new accounts either We're working on a plan to minimize the downtime/reconfiguration part, but at first sight, due to the hardware move of the racks/cabling parts/etc, the announced downtime will be probably ~48h. Migration is scheduled for """"Monday October 10th, 1:00 pm UTC time"""". You can convert to local time with $(date -d '2016-10-10 13:00 UTC') We'll start to restore services as soon as possible, while working in coordination with the DC people closely (and after validation about network ports, connection) During this period you can come talk to us in #centos-devel and we will be posting updates there. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

2 4

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users October 2016