CI-users December 2015

ci-users@lists.centos.org

6 participants
8 discussions

Strategy for scaling jenkins slaves ?
by David Moreau Simard 16 May '16

16 May '16

Hi, The RDO slave currently has 15 execution threads running on two cores and 4GB of RAM. As we use ci.centos more and more for gating and periodic jobs, I can see that the slave starts to struggle a little bit CPU-wise when the execution threads are busy. The bulk of the jobs are offloaded to an ephemeral bare metal node but ansible still runs from the slave. Some jobs do run locally, such as smaller tox jobs. What's the strategy if we want to scale this ? For example, do we want to scale in increments of "10 threads/2 cores/4 GB RAM" by adding more slaves as need be or do we want to grow the specifications of the existing slave and grow the amount of threads ? I have a personal preference to have more smaller slaves as this also has the added benefit of redundancy so we're not left entirely out in the cold if this one big slave happens to crash. David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter]

5 7

artifacts from inside and outside the ci infra
by Karanbir Singh 06 Jan '16

06 Jan '16

hi, what was the final decision around how best to export artifacts so they are visible on the same url as internally inside of ci ? regards, -- Karanbir Singh +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh GnuPG Key : http://www.karan.org/publickey.asc

4 4

Introducing python-cicoclient: A library and CLI to admin.ci.centos.org
by David Moreau Simard 19 Dec '15

19 Dec '15

Heya, I read the docs, the mailing lists and glanced over the various scripts out there and was convinced there had to be a better way than using curl to communicate with admin.ci.centos.org. Introducing python-cicoclient: - Github: https://github.com/dmsimard/python-cicoclient - PyPi: https://pypi.python.org/pypi/python-cicoclient/ - Docs: http://python-cicoclient.readthedocs.org/en/latest/ I broke down this package in three main components: - client.py: Abstracts the http stuff - wrapper.py: Uses client.py to wrap around the available API calls and provide a consistent user experience - shell.py/cli.py: Provide a CLI interface around wrapper.py It can be used as both a CLI client with the provided 'cico' binary but also as a library - you can easily import wrapper and roll your own thing. Feel free to use it, contribute and provide feedback ! Please note that as of right now, there is a problem with the "node get" command due to an upstream bug, documented here: https://bugs.centos.org/view.php?id=9748 Please do not use "node get" until this bug is fixed :) David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter]

4 10

Artifacts: Unplanned Outage: 2015-12-18 08:40 UTC
by Brian Stinson 18 Dec '15

18 Dec '15

Hi Folks, The filesystem hosting http://ci.centos.org/artifacts was very close to running out of inodes. We mirrored the contents to another filesystem over the course of the day. At approximately 08:40AM UTC we had a very short outage (30seconds or so) to swap in the new filesystem. I have one final diff sync to do, and selinux is busy setting labels on things currently. We expect to be back to 100% in the next few hours. If you have any trouble, please let us know. Cheers! -- Brian Stinson CentOS CI Infrastructure Team

1 0

ci.centos.org Maintenance Thursday, 10-December-2015 01:00 UTC (20:00 EST)
by Brian Stinson 11 Dec '15

11 Dec '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Folks, There is a Jenkins update (including security fixes) scheduled for next week. We will be rebooting Jenkins to apply this fix, and to clear the backlog of config updates (adding IRC channels, plugins, etc.). We do not expect a long outage, but we'll schedule 1hr to be safe. As usual we will pause jobs 1 hour beforehand to let things clear out, and will post updates here and to #centos-devel on freenode. Cheers! - -- Brian Stinson CentOS CI Infrastructure Team -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJWYJBpAAoJEIMGvNKzCweM7/cP/1H5MQYoXFkDbPX/Wiar6+PH lxXFVWeKk/DMV2m5FGHbpe4JFJP5rLNs1wRc/5eAJWa6iVnxaGJ/34jGqjBBhJDN p9CisRvPvFQP+0/0jUhc460MuAbtTCgO7bmRyuCWMxzO+UH5+62mD1M+tblcR7F7 S4gFOnkSwjOWRIFgCDsfAS6fh6KFt/9nX10BjdDeCxad5rbhwy2iYcAT71kzjtD2 yYZ4/GdXHqWaO12dCvB4q8ndb9J6+s+vORb0UlzrrRezbkWFY7s+v2Sah4ypTcbN FAkJe7rwDKzK6WDXmBAMYNsmRyi2DJ1PR4DlHkBS9ootMUlNYye1dio/5C0+681B Xldy4NHR+Gs7m80XSpXGn3LsUyGdk43Ks8XZGl8vwj0v532qBdL7oM96mg2gbxfe OPpDKsSNtdznjysjeYmK6emkQ/+cKb4OZ7NBF1+Y9kviqCRHyn7r/e+kQdn+/P9v AIk7VmNKp3VosPoNWUCUiNobilaW4hOajcKFpkjFBN4fu+99wqmNrdJk9KwgBy4s 9CEtEtivmhLT0v49D1aiNEqIHLv4qmn+W/SjSIo4oqJ6SWAT2tLe+nPZj+Wdwdmv SMxjCSbQoAGEDoHEKXgTteLl2xa6h+PJV75LvocwdKoLHhuRDxmCy8V4Gy7P8muS 7tY25FBBs1Tw57CSX9GP =o79f -----END PGP SIGNATURE-----

1 2

openstack-puppet orphaning machines
by Karanbir Singh 07 Dec '15

07 Dec '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi Guys, We've had a few sessions get orphaned in the openstack-puppet setup these last few days ( we reclaimed 36 hosts day before, and 7 more today ). These are machines that were blocked for 24hrs+ and not released back to the deployment pools. Its not a problem, we expect to be reaping machines; just wanted to make sure the relevant people were aware of it. regards - -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJWXXQRAAoJEI3Oi2Mx7xbtvRoH/2H87GSZYyYvuWtX8tXKWh+K vaFtmPvvnTbGvny1f2BlTM8Jc4wwN+JZ8iNyi7A/Dy2/UUdnyK5zvgX9TAceBkFm gE/ytYgy6qez+SDSsRP+Oh2n01qNDZAJoOg5nU7A1oLDqA+qxotjTMjgPEtIwYW0 O6q266Tx6wyAq+Pjlnh+b3eFrQgN2DR9A4SM6vDSzowoSRKT40e7G3vrqbDRopls 43nsSWsSLBsMPlX7GGeb1mHVxfossfTCleGcONQN/yAu4kANkoV4WVGxhnsVP4du DP8wPJTD5pnyTcO13tdIFXy7s3tEC0hz75SvObqKPJy3MrfVRNaRnxxSgqCXt2M= =aDzi -----END PGP SIGNATURE-----

3 2

upcoming CentOS Linux 7 release
by Karanbir Singh 02 Dec '15

02 Dec '15

hi guys, RHEL 7.2 was released a few days back and we've been busy sorting through the code and working on the downstream CentOS Linux release ( its going to likely be released in November and will be called 7.1511 ). I'd like to make that content available on ci.centos.org ahead of public release. And there are a couple of ways to do this : 1) either we update the provisioning service, so all requests for a centos linux 7 machine will give you a 7.1511 build ( its 7.1503 at the moment ), with all the updates etc posted in sync that way. This also means that mirror.centos.org inside the ci infra will look DIFFERENT from mirror.centos.org the real users see, since it will be setup to look like the next release. 2) we make the content available in a new set of repos, called CentOS.next or something; and the ci users are able to optionally tweak their build scripts etc to include or not include these new changes. 3) we dont make this content available before its publicly released. of all those bits, I like (1) best; (2) only delays the content from hitting the ci infra by a week or 10 days. We dont usually take longer than that from when the builds are done to when the distro update is released. (3) just means that if there are changes, failures or breakages in the ci tested code, we only find out at the same time as the users - which isnt nice. thoughts ? -- Karanbir Singh +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh GnuPG Key : http://www.karan.org/publickey.asc

3 4

Jenkins trigger/notification issues
by James 01 Dec '15

01 Dec '15

I've finally got jenkins building working when clicking "Build Now". w00t. Thanks to bstinson for finding one of the issues. I'll write this up, but before I do that, I've got three open issues: 1) IRC notification doesn't work on post-build -- It's not even clear where you set the server, only the channel name field is visible 2) Email notification didn't work. 3) Most importantly: triggering the builds: best I can come up with is putting a git post-update hook in my local repo, and `wget` the special: "Trigger builds remotely (e.g., from scripts)" url when I want to run it. This actually works, but it requires you to actually already be logged in to jenkins, and I wasn't sure how to dump in my username and password into a wget command. No I don't want to screenscrape, phantomjs, or wget with my firefox session cookies! I also don't want to "poll scm" because that's mean/ugly. Thanks, James @purpleidea

2 5

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users December 2015