CI-users

ci-users@lists.centos.org

2 participants
420 discussions

[infra] : Hardware migration status (full/correct version)
by Fabian Arrotin 11 Oct '16

11 Oct '16

As announced some time ago ( https://lists.centos.org/pipermail/centos-announce/2016-September/022065.ht… ) , there was a big hardware migration/DC relocation happening yesterday that impacted several of our public services. Here is a small report of that migration, that finally took less time that previously announced. * 2016-10-10 - 1pm UTC : we started to preconfigure all changes, and powered down all the services in correct order * 2016-10-10 - 2pm => 6pm UTC : hardware was moved/migrated * 2016-10-10 - 6pm => 8pm UTC : hardware was (in serial order) plugged back in and ipmi/oob interfaces reconfigured, and network started to be validated * 2016-10-10 - 8:30pm UTC : https://accounts.centos.org was back online and operational * 2016-10-10 - 9pm UTC : https://cbs.centos.org was back online but builders were still disabled for validation * 2016-10-10 - 9:45pm UTC : https://cbs.centos.org : builders were enabled and working again * 2016-10-11 - 3:30am UTC : https://ci.centos.org now fully operational (after some network reconfiguration needed post-migration, and validations Thanks a lot to Brian Stinson (on-site) and DC people who helped for that migration (Matt, Jon, Nathan, Nina, Phil) Should you encounter an issue , feel free to discuss with us in #centos-devel on irc.freenode.net PS : sorry for the previous and incomplete mail : sent by accident before finished and probably due to lack of coffee and short night ... -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

3 2

[infra] : Hardware migration status
by Fabian Arrotin 11 Oct '16

11 Oct '16

1 0

OpenStack cluster in ci.centos.org
by Karanbir Singh 06 Oct '16

06 Oct '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Folks, If anyone is using the openstack infra in ci.centos.org - please note that when the ci hardware is relocated this monday morning on the 10th Oct, the openstack cluster will NOT be coming back online. Our plan is to reinstall the entire openstack hypervisors and redeploy with neuton the following day and bringup an entirely clear cluster. If anyone has content there that they need to preserve, please back that up and shutdown the VMs before Monday morning UTC 10th October. Regards, - -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJX9lvpAAoJEI3Oi2Mx7xbtXosIAKXfGzj81JhKe6xbqOxp1vjE bTb3/2BK0EJxLzD2QGPuXmFMzgZf5him+cxJUu/jjh5pGoPSgto4rEu1UzBLhapy RzQvfCn9e877hhwUaPmBrJyaNlsfF8mIKKtoXCqwYKXRGvka3cJbXTp0WbsTjEmi +JEOWKt3cYcDio7Wj+cJMG8mqEXmgDb1pUxiiQPnzrx0BqbG/DUg0O42Oj+AMqPX pI4HI+jUZRtMooYLd9PoGdZsAZf8Cxy/dXFPJz1zy9C+iZcDFCt0vY8OvxDUuEfA N41dgEpxapYN81l9BSz9fXOW8kc+AzUIxkFI5BearcrrFVkSkt77QGRWzr3NnwA= =Z5Qz -----END PGP SIGNATURE-----

1 0

[Infra] - Major outage for several services (please read)
by Fabian Arrotin 05 Oct '16

05 Oct '16

As pre-announced earlier this year (https://lists.centos.org/pipermail/centos-devel/2016-May/014792.html) we'll have to move a part of our existing hardware to a new DC. That means that the following public services will be powered off and unreachable : - https://cbs.centos.org (Koji build farm front end and also builders/storage nodes behind) - https://accounts.centos.org (auth backend) - https://ci.centos.org (jenkins-driven CI environment and all nodes in that dedicated/isolated environment) What does that mean ? - Nobody from the SIGs (https://wiki.centos.org/SpecialInterestGroup) will be able to submit build jobs/tasks (so no update at all during that migration) - No CI job will be able to track CBS nor external repositories (and so that means also that http://artifacts.ci.centos.org will be unreachable) - Not possible for existing users to update their details on accounts.centos.org, and not possible to create new accounts either We're working on a plan to minimize the downtime/reconfiguration part, but at first sight, due to the hardware move of the racks/cabling parts/etc, the announced downtime will be probably ~48h. Migration is scheduled for """"Monday October 10th, 1:00 pm UTC time"""". You can convert to local time with $(date -d '2016-10-10 13:00 UTC') We'll start to restore services as soon as possible, while working in coordination with the DC people closely (and after validation about network ports, connection) During this period you can come talk to us in #centos-devel and we will be posting updates there. on behalf of the Infra team, -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

2 4

Slave "atomicapp-ci-slave01" CI machine is offline
by Budh Gurung 28 Sep '16

28 Sep '16

Hi, It seems that slave "atomicapp-ci-slave01" CI machine is offline in https://ci.centos.org which handle the CI for vagrant-service-manager project [1]. Could we make it online? What could the reason for its being offline? Attached few screenshots for the same. [1] https://github.com/projectatomic/vagrant-service-manager Regards, Budh Ram Gurung

2 1

Rollback of the Multi Job Plugin
by Brian Stinson 28 Sep '16

28 Sep '16

Hi Folks, We had a report earlier today that presentation of some Multi Jobs was causing stack traces in the Jenkins interface. We've tracked this down to an interaction between the presentation layer and jobs that parameterize downstream jobs with custom attributes. The best fix for now was to downgrade to Multi Job Plugin version 1.21 which should now be active. We'll track the appropriate bugs[0] upstream and make sure this is resolved for our upcoming move to Jenkins 2. [0]: https://issues.jenkins-ci.org/browse/JENKINS-38053 Cheers! -- Brian Stinson CentOS CI Infrastructure Team

1 0

Pattern for using JJB and python-cicoclient
by Nigel Babu 23 Sep '16

23 Sep '16

Hello, With David's help, I've built a better pattern to consume python-cicoclient alongside JJB. A major goal of this work was to avoid using curl calls to Github. In summary, it uses a builder to get nodes and a publisher to release nodes without leaking. I've created a Github repo[1] with a real-world job (actually a copy of one of our jobs). Please let me know if you find this useful and if you have suggestions for improvements. As I convert more jobs to JJB, it will be more flexible for requesting more than one node and more than one OS version. [1]: https://github.com/nigelbabu/centos-ci-sample -- nigelb

1 0

reaping duffy nodes at 6hrs
by Karanbir Singh 21 Sep '16

21 Sep '16

hi, We seem to still be reaping quite a few machines at the 6hr lease limit, if anyone needs the machines for longer, let one of us know. Regards -- Karanbir Singh +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh GnuPG Key : http://www.karan.org/publickey.asc

4 4

Another jenkins plugin needed
by Clint Savage 19 Sep '16

19 Sep '16

Looks like I could use the following plugin. https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin Thanks, herlo

3 4

FYI : temporary secondary rdo-jenkins slave node
by Fabian Arrotin 17 Sep '16

17 Sep '16

Hi, As discussed with Attila this morning, there were quite some issues on the rdo-jenkins.ci.centos.org VM in the last days (leading to jenkins slave not running and all jobs not running either). Lot of xfs issues on the /home partition, and no xfs_repair didn't seem to have helped. As agreed with Attila, I deployed a rdo-jenkins-2.ci.centos.org VM (on the same hypervisor) with a reduced vcpu/memory so that he can then configure it (through the RDO ansible playbooks) and migrate what would need to be migrated (like probably the jenkins workspace on that node) After we'll have confirmation that it seems to work fine, we'll shutdown the previous VM, and we'll change the vcpu/memory settings on the newly configured VM. -- Fabian Arrotin The CentOS Project | http://www.centos.org gpg key: 56BEC54E | twitter: @arrfab

4 4

← Newer
1
...
29
30
31
32
33
34
35
...
42
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

CI-users