CI-users

ci-users@lists.centos.org

423 discussions

Strategy for scaling jenkins slaves ?
by David Moreau Simard 16 May '16

16 May '16

Hi, The RDO slave currently has 15 execution threads running on two cores and 4GB of RAM. As we use ci.centos more and more for gating and periodic jobs, I can see that the slave starts to struggle a little bit CPU-wise when the execution threads are busy. The bulk of the jobs are offloaded to an ephemeral bare metal node but ansible still runs from the slave. Some jobs do run locally, such as smaller tox jobs. What's the strategy if we want to scale this ? For example, do we want to scale in increments of "10 threads/2 cores/4 GB RAM" by adding more slaves as need be or do we want to grow the specifications of the existing slave and grow the amount of threads ? I have a personal preference to have more smaller slaves as this also has the added benefit of redundancy so we're not left entirely out in the cold if this one big slave happens to crash. David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter]

5 7

Accessing cbs.centos.org from inside CI
by Colin Walters 11 May '16

11 May '16

``` [atomic-sig@slave01 ~]$ host cbs.centos.org cbs.centos.org has address 172.20.1.15 [atomic-sig@slave01 ~]$ curl https://cbs.centos.org/repos/virt7-docker-common-candidate/x86_64/os/ ^C ``` Just times out - I'm guessing there's some intermediate firewall, or perhaps IP conflict on the 172.19 subnet? Using the external IP works: ``` [atomic-sig@slave01 ~]$ host cbs.centos.org 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases: cbs.centos.org has address 66.187.224.194 [atomic-sig@slave01 ~]$ curl http://66.187.224.194/repos/virt7-docker-common-candidate/x86_64/os/ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> ```

4 9

going beyond getting started
by Colin Walters 27 Apr '16

27 Apr '16

Hi, so I got some more time to run through the process of onboarding. I got a simple job created via JJB that runs directly on the slave. At this point though I'd like to load some custom software (rpmdistro-gitoverlay). Now my understanding is that we only have a non-root user on the slave VM, and so if we want to do anything that isn't already installed, we should call out to Duffy? My first question is - has anyone tried out writing an Ansible dynamic inventory script for Duffy? The demo https://github.com/kbsingh/centos-ci-scripts/blob/master/build_python_scrip… is kind of obviously a poor man's Ansible =) I have 3 other questions about Duffy. First, might be interesting to investigate a configuration like this: http://www.projectatomic.io/blog/2015/05/building-and-running-live-atomic/ which basically runs out of RAM directly. This model is particularly well suited to workloads like Duffy where you don't *actually* want the OS to be persistent on disk. Using the disks as swap space instead of xfs/ext4 can be a dramatic speed improvement. (A large part is ignoring fsync and journaling) Beyond that, has any thought been given to also supporting e.g. OpenStack as a provisioning API? Or for that matter allocating a Kubernetes namespace or OpenShift project? My workloads for Project Atomic are going to be pretty mixed across all of these actually.

4 10

Yum errors today in ci.centos.org
by Dusty Mabe 25 Apr '16

25 Apr '16

We've seen a bunch of yum errors today with our tests. basically a bunch of these types of errors: [Errno -1] repomd.xml does not match metalink for epel Full log at [1]. I'm thinking maybe we should modify our tests to run yum in a loop to account for intermittent errors like this? [1] - https://ci.centos.org/job/atomicapp-test-docker-pr/67/console

6 9

api key not really used in a secret way
by Colin Walters 13 Apr '16

13 Apr '16

Not that this really matters a lot since we can probably trust each other right now not to use other's resources, but I noticed many people end up leaking the API key publicly, e.g. https://ci.centos.org/job/bstinson-centpkg-unittests/configure and https://ci.centos.org/job/adb-openshift-vagrantfile-tests/12/console and several others. The two problems seem to be including the Python script raw as a builder (which Jenkins exposes as public data), or injecting it as an environment variable (which shows up in the Jenkins console logs). I created: https://github.com/kbsingh/centos-ci-scripts/pull/4 but since there are many forks of this now, multiple groups will need to change their copies too.

3 2

Unplanned Outage: Inbound connections to ci.centos.org Mon 11-Apr-2016 (14:10 UTC)
by Brian Stinson 11 Apr '16

11 Apr '16

Hi Folks, This morning we had an unplanned outage of the NGINX proxy inbound to ci.centos.org. We were able to reboot the machine, and add some resources to help us better handle the inbound traffic. Thanks to Fabian for doing that! We've seen this behavior once or twice before, where inbound http connections fail intermittently. During those periods and during our emergency maintenance, connections internally between the slaves and the jenkins master remained up and no jobs were interrupted. If you have any questions, please let us know here on-list or in #centos-devel on freenode. Cheers! -- Brian Stinson CentOS CI Infrastructure Team

1 0

Disk space consumption on artifacts.ci.centos.org
by Fabian Arrotin 07 Apr '16

07 Apr '16

So, while working on the LV for the exported dir for rdo-store, I noticed (and Zabbix complained a lot about this in the last weeks) that the artifacts LV is getting bigger and bigger. Actually, the rdo project is more and more using disk space : 760G rdo It would be good to do some clean-up there, and only keep what's need to be publicly available (behind http://artifacts.ci.centos.org/rdo/ | https://ci.centos.org/artifacts/rdo/ ) Happy to implement that at the server itself (through a cron job) as long as we agree on the criteria (so for example delete everything older than 30 days) OTOH, keep in mind that I see multiple .qcow2 images (for undercloud ?) so those would be also deleted. Or would you want to delete yourself the unneeded files on your side ? -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

4 3

new NFS export for RDO-store
by Fabian Arrotin 04 Apr '16

04 Apr '16

Just to let the RDO people be aware that a specific nfs export has been made available for them on the artifacts node : artifacts.ci.centos.org:/srv/rdo-store (and will be accessible from n30.pufty.ci.centos.org) The underlying LV is 250GB (as discussed) but can be grown later online if needed. -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

2 1

Unplanned Maintenance: Jenkins Safe Restart 30-Mar-2016 11h25 Eastern
by Brian Stinson 30 Mar '16

30 Mar '16

Hi Folks, We're working on an issue with our Gerrit trigger plugins, and we need to perform a saferestart of Jenkins to restore connection to gerrithub. You may notice the banner on ci.centos.org coming up soon (We're expecting this to happen around 11:25 AM Eastern time, 15:25 UTC). We attempted this last night, but there were a few long-running jobs that we didn't want to interrupt. During a saferestart, Jenkins will queue up any incoming jobs and resubmits them after the restart is finished. If you have any questions, please let us know here or in #centos-devel Cheers! -- Brian Stinson CentOS CI Infrastructure Team

1 1

CI+GitHub integration?
by James 29 Mar '16

29 Mar '16

I heard GitHub plugins now exist. Has anyone prepared some docs or a quick terminal session of the correct way to get this running? Once this works, my git tpush [1] script should also automatically work! Thanks, James [1] https://ttboj.wordpress.com/2016/02/16/introducing-git-tpush/

2 2

← Newer
1
...
34
35
36
37
38
39
40
...
43
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users