CI-users April 2016

ci-users@lists.centos.org

12 participants
8 discussions

Jenkins SafeRestart to add extra CSRF Protection 19-Apr-2016 14h30 UTC (09h30 EDT)
by Brian Stinson 21 May '16

21 May '16

Hi Folks, In response to news of directed attacks against public Jenkins instances[0], we are enabling some of the CSRF protections in ci.centos.org To do this we will issue a SafeRestart at 14:30 UTC Today! Running jobs will be given a chance to clear and new jobs should be queued up and will execute as soon as the restart finishes. Potential Impact: - If you are using the Jenkins REST interface you may need to modify your scripts to send the appropriate headers[1] - Jenkins Job Builder is tracking an issue to enable CSRF support[2]. Some basic tests were performed on our side, and simple jobs were configured correctly, but you may notice strange behavior if you are using JJB. [0]: https://groups.google.com/d/topic/jenkinsci-advisories/lJfvDs5s6bk [1]: https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API#RemoteaccessA… [2]: https://storyboard.openstack.org/#!/story/2000556 If you have any questions or comments, let us know here or find one of us in #centos-devel on Freenode. Cheers! -- Brian Stinson CentOS CI Infrastructure Team

3 7

Strategy for scaling jenkins slaves ?
by David Moreau Simard 16 May '16

16 May '16

Hi, The RDO slave currently has 15 execution threads running on two cores and 4GB of RAM. As we use ci.centos more and more for gating and periodic jobs, I can see that the slave starts to struggle a little bit CPU-wise when the execution threads are busy. The bulk of the jobs are offloaded to an ephemeral bare metal node but ansible still runs from the slave. Some jobs do run locally, such as smaller tox jobs. What's the strategy if we want to scale this ? For example, do we want to scale in increments of "10 threads/2 cores/4 GB RAM" by adding more slaves as need be or do we want to grow the specifications of the existing slave and grow the amount of threads ? I have a personal preference to have more smaller slaves as this also has the added benefit of redundancy so we're not left entirely out in the cold if this one big slave happens to crash. David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter]

5 7

going beyond getting started
by Colin Walters 27 Apr '16

27 Apr '16

Hi, so I got some more time to run through the process of onboarding. I got a simple job created via JJB that runs directly on the slave. At this point though I'd like to load some custom software (rpmdistro-gitoverlay). Now my understanding is that we only have a non-root user on the slave VM, and so if we want to do anything that isn't already installed, we should call out to Duffy? My first question is - has anyone tried out writing an Ansible dynamic inventory script for Duffy? The demo https://github.com/kbsingh/centos-ci-scripts/blob/master/build_python_scrip… is kind of obviously a poor man's Ansible =) I have 3 other questions about Duffy. First, might be interesting to investigate a configuration like this: http://www.projectatomic.io/blog/2015/05/building-and-running-live-atomic/ which basically runs out of RAM directly. This model is particularly well suited to workloads like Duffy where you don't *actually* want the OS to be persistent on disk. Using the disks as swap space instead of xfs/ext4 can be a dramatic speed improvement. (A large part is ignoring fsync and journaling) Beyond that, has any thought been given to also supporting e.g. OpenStack as a provisioning API? Or for that matter allocating a Kubernetes namespace or OpenShift project? My workloads for Project Atomic are going to be pretty mixed across all of these actually.

4 10

Yum errors today in ci.centos.org
by Dusty Mabe 25 Apr '16

25 Apr '16

We've seen a bunch of yum errors today with our tests. basically a bunch of these types of errors: [Errno -1] repomd.xml does not match metalink for epel Full log at [1]. I'm thinking maybe we should modify our tests to run yum in a loop to account for intermittent errors like this? [1] - https://ci.centos.org/job/atomicapp-test-docker-pr/67/console

6 9

api key not really used in a secret way
by Colin Walters 13 Apr '16

13 Apr '16

Not that this really matters a lot since we can probably trust each other right now not to use other's resources, but I noticed many people end up leaking the API key publicly, e.g. https://ci.centos.org/job/bstinson-centpkg-unittests/configure and https://ci.centos.org/job/adb-openshift-vagrantfile-tests/12/console and several others. The two problems seem to be including the Python script raw as a builder (which Jenkins exposes as public data), or injecting it as an environment variable (which shows up in the Jenkins console logs). I created: https://github.com/kbsingh/centos-ci-scripts/pull/4 but since there are many forks of this now, multiple groups will need to change their copies too.

3 2

Unplanned Outage: Inbound connections to ci.centos.org Mon 11-Apr-2016 (14:10 UTC)
by Brian Stinson 11 Apr '16

11 Apr '16

Hi Folks, This morning we had an unplanned outage of the NGINX proxy inbound to ci.centos.org. We were able to reboot the machine, and add some resources to help us better handle the inbound traffic. Thanks to Fabian for doing that! We've seen this behavior once or twice before, where inbound http connections fail intermittently. During those periods and during our emergency maintenance, connections internally between the slaves and the jenkins master remained up and no jobs were interrupted. If you have any questions, please let us know here on-list or in #centos-devel on freenode. Cheers! -- Brian Stinson CentOS CI Infrastructure Team

1 0

Disk space consumption on artifacts.ci.centos.org
by Fabian Arrotin 07 Apr '16

07 Apr '16

So, while working on the LV for the exported dir for rdo-store, I noticed (and Zabbix complained a lot about this in the last weeks) that the artifacts LV is getting bigger and bigger. Actually, the rdo project is more and more using disk space : 760G rdo It would be good to do some clean-up there, and only keep what's need to be publicly available (behind http://artifacts.ci.centos.org/rdo/ | https://ci.centos.org/artifacts/rdo/ ) Happy to implement that at the server itself (through a cron job) as long as we agree on the criteria (so for example delete everything older than 30 days) OTOH, keep in mind that I see multiple .qcow2 images (for undercloud ?) so those would be also deleted. Or would you want to delete yourself the unneeded files on your side ? -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

4 3

new NFS export for RDO-store
by Fabian Arrotin 04 Apr '16

04 Apr '16

Just to let the RDO people be aware that a specific nfs export has been made available for them on the artifacts node : artifacts.ci.centos.org:/srv/rdo-store (and will be accessible from n30.pufty.ci.centos.org) The underlying LV is 250GB (as discussed) but can be grown later online if needed. -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users April 2016