CI-users July 2017

ci-users@lists.centos.org

8 participants
12 discussions

Jenkins Full Restart ASAP
by Brian Stinson 31 Jul '17

31 Jul '17

Hi Folks, We will be entering a quiet period and issuing a full restart to do plugin updates and performance tuning on the Jenkins master. This is needed to handle a recent stability issue. Thanks for your patience while we handle this maintenance. Cheers! --Brian

1 1

Emergency Saferestart Immediately
by Brian Stinson 27 Jul '17

27 Jul '17

Hi Folks, We need to do a saferestart as soon as possible to update some credentials on the Jenkins master. I'll also be applying some plugin updates that contain a few security fixes. Thanks for your patience. --Brian

1 0

AltArch support in CI (status update)
by Fabian Arrotin 26 Jul '17

26 Jul '17

Hi, Just starting a thread on the list so that various projects having a need to test their builds on AltArch (aarch64/ppc64/ppc64le) within CI are aware of the following : Thanks to Brian, who coordinated with several folks, we'll be able soon to provide access to aarch64/ppc64/ppc64le nodes within ci.centos.org. So far we have configured/deployed/tested ppc64/ppc64le in the environment, so that we can automate the whole thing happening behind the scene (basically ansible being called for this, the same way it's actually done for x86_64). The last thing to do is now to make Duffy (https://wiki.centos.org/QaWiki/CI/Duffy) aware of those "other arches" nodes to be able to - hand over those nodes on specific request - reinstall those automatically by calling correct playbooks Worth noting that due to the amount of physical nodes for those AltArches, we'll not be able to provide you access to bare metal nodes, but rather to VMs running on those (at least for ppc64/ppc64le). That shouldn't be a problem as we'll be able to distribute resources so that you'll have multiple vcpus per VM, and for Power8, our hypervisors are installed with CentOS 7 ppc64le, that uses the kvm_hv module. That then means that it's possible to use nested virt (if needed) as it will then use the kvm_pr for this, slower, but works (http://wiki.qemu.org/Documentation/Platforms/POWER) If you have remark/comments, feel free ! -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

3 4

Re: [Ci-users] Jenkinsfiles and pipeline libraries in Centos CI
by Václav Pavlín 22 Jul '17

22 Jul '17

Hi Brian, Ari, Thanks for the link and offer for help. Ari, can you setup some meeting for when you are back, please? Or let me know where to join if any of you think it would be better to do it in some Centos CI meeting. I'll explore the Jenkinsfile on Monday. Cheers, Vašek On 21 Jul 2017 23:37, "Ari LiVigni" <alivigni(a)redhat.com> wrote: On Fri, Jul 21, 2017 at 5:24 PM, Brian Stinson <brian(a)bstinson.com> wrote: > On Jul 20 11:51, Václav Pavlín wrote: > > Hi all, > > > > We would like to start using Jenkinsfiles for some builds (building > docker > > images) so I wanted to ask if there are any CentOS CI specific libraries > or > > example Jenkinsfiles you could share. > > > > The reason is that in Jenkinsfile, guys currently have, they do docker > > build directly on the slave, which is not gonna work in CentOS CI where > the > > cico node get magic is a way to go. > > > > I was thinking about putting together some library that would make it > > easier to build docker images in CentOS CI with Jenkinsfile, but first > > wanted to check if something like this already exists. > > > > Have a nice day, > > Vašek > > > > -- > > Red Hat Developer Tools team > > Brno, Czech Republic > > Phone: +420 739 666 824 > > > _______________________________________________ > > Ci-users mailing list > > Ci-users(a)centos.org > > https://lists.centos.org/mailman/listinfo/ci-users > > > I don't know off the top of my head of many folks doing Jenkinsfiles in > ci.centos.org yet, but the fedora-atomic CI pipeline guys have a few > things: > > https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/Jenkinsfile > > I'm happy to help you put something together that we can put in a > central location (so others can use as well). > > When I am back from PTO I should be able to help as well. We are actually taking the Jenkinsfile and going to split the stages up into shared libraries going forward but we wanted to get something working. > -- > Brian > _______________________________________________ > Ci-users mailing list > Ci-users(a)centos.org > https://lists.centos.org/mailman/listinfo/ci-users > -- -== @ri ==-

2 1

Jenkinsfiles and pipeline libraries in Centos CI
by Václav Pavlín 21 Jul '17

21 Jul '17

Hi all, We would like to start using Jenkinsfiles for some builds (building docker images) so I wanted to ask if there are any CentOS CI specific libraries or example Jenkinsfiles you could share. The reason is that in Jenkinsfile, guys currently have, they do docker build directly on the slave, which is not gonna work in CentOS CI where the cico node get magic is a way to go. I was thinking about putting together some library that would make it easier to build docker images in CentOS CI with Jenkinsfile, but first wanted to check if something like this already exists. Have a nice day, Vašek -- Red Hat Developer Tools team Brno, Czech Republic Phone: +420 739 666 824

3 2

data hub in centos ci
by Anton Sherkhonov 13 Jul '17

13 Jul '17

Hi, I'm with Systems Design & Engineering group and I'm working on the Data Hub in CentOS CI OpenShift. Major part of Data Hub infrastructure is Elasticsearch 5. Elasticsearch 5 is not officially supported in OpenShift 3.5 deployed in CentOS CI, we're working on getting proper Elasticsearch 5 support into OpenShift 3.7 and most likely we'll need at least OpenShift 3.6 features to properly support Elasticsearch 5. The reason for the requirement of OpenShift 3.6 is because we're moving to the new oauth-proxy to secure the routes for Kibana/Cerebro web ui, which requires OpenShift 3.6. It would be great to have CentOS CI OpenShift environment upgraded to 3.6 soon after 3.6 is out. Thanks, Anton.

3 4

Unplanned outage incident 06h37 - 08h12 UTC
by Brian Stinson 13 Jul '17

13 Jul '17

Issue Summary ============= A misconfigured ulimit on jenkins.ci.centos.org caused Jenkins to fail with too many open files. This also caused the root volume to fill up because of noisy messages in the logs. Access to the Jenkins HTTP interface was affected during this period. Root Cause ========== During the reconfigure and move to a new host, the 'nofile'ulimits to jenkins were reset to the default (4096). Jenkins reached this limit before the next scheduled garbage collection. Recovery ======== At 08h12 we cleared out the jenkins log to free up disk space and set the ulimits for the jenkins user to the appropriate value, and jenkins was restarted. Corrective Measures =================== The ulimit change was reflected in our ansible scripts and deployed. Impact ====== Jobs running during the window should have completed, but may not have reported back status to Jenkins. SCM/Github jobs that would have been triggered during that period were picked up when the Jenkins service came live again. Jobs with triggers through other means (messaging, HTTP POST, etc.) may not have been launched. We appreciate your patience during this outage, and apologize for any inconvenience. -- Brian Stinson CentOS CI Infrastructure Team

1 0

apps.ci
by Karanbir Singh 05 Jul '17

05 Jul '17

Hi Brian, Ari Can we get apps.ci documented on https://wiki.centos.org/QaWiki/CI Additionally, can we get the auth backend offloading to accounts.centos.org via id.centos.org ? I believe ipsilon already has whats needed, and it should just be a single change to the backing openshift master config. Regards -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc

3 4

tearing down os.ci.centos.org
by Karanbir Singh 05 Jul '17

05 Jul '17

The os.ci.centos.org openshift cluster has been running since Aug 2016 and has been a huge contributor to success for a multitude of projects. however, its being replaced by apps.ci.centos.org, which is better supported, better resourced bringup for openshift that engaged deeply within the PaaS SIG. I've set the decomission date for os.ci to be 20th July 2017. If anyone has apps running on there, please reach out and we can workout a migration plan. regards -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc

1 0

'git' is no longer available
by Karanbir Singh 03 Jul '17

03 Jul '17

https://ci.centos.org/view/Devtools/job/devtools-build-run-che-build-master… is this something from the work over the weekend ? -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users July 2017