[Ci-users] FYI : temporary secondary rdo-jenkins slave node

Fri Sep 16 17:38:50 UTC 2016
David Moreau Simard <dms at redhat.com>

I had to install some other things that were expected in the slave
(i.e, pip, tox, cicoclient) but it was otherwise okay.

I added the slave to our monitoring.
When are we planning to bump the resources on the new slave ?
I brought up the three cloud slaves (4 threads each) in order to clear
our growing job queue.

Also, can we keep the old slave around for a while longer, in a "just
in case" basis ?
Barebones 1 core/2GB RAM should be plenty, just to make sure we don't
forget anything.

David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]

On Fri, Sep 16, 2016 at 9:05 AM, Attila Darazs <adarazs at redhat.com> wrote:
> On 09/16/2016 02:18 PM, Fabian Arrotin wrote:
>> Hi,
>> As discussed with Attila this morning, there were quite some issues on
>> the rdo-jenkins.ci.centos.org VM in the last days (leading to jenkins
>> slave not running and all jobs not running either).
>> Lot of xfs issues on the /home partition, and no xfs_repair didn't seem
>> to have helped.
>> As agreed with Attila, I deployed a rdo-jenkins-2.ci.centos.org VM (on
>> the same hypervisor) with a reduced vcpu/memory so that he can then
>> configure it (through the RDO ansible playbooks) and migrate what would
>> need to be migrated (like probably the jenkins workspace on that node)
>> After we'll have confirmation that it seems to work fine, we'll shutdown
>> the previous VM, and we'll change the vcpu/memory settings on the newly
>> configured VM.
> First, thanks to Fabian for the awesome and quick help.
> What happened from my side so far:
> - copied the keys and .ssh dir from /home/rhos-ci to the new slave in
> /home/rdo-ci (setting this slave up with that user to be less confusing)
> - added the crontab entries from the old slave to start the java slave
> client automatically
> - installed java jdk and did quickstart.sh --install-deps to add the
> necessary packages to the system
> - reduced the executors on the slave from 13 to 8 to not overload the slave
> with the smaller VCPU/memory count
> I started a periodic job on it and it seems to be running fine, so I added
> back all the regular labels to it and now it's running jobs.
> David should probably install a couple of more stuff to make the weirdo jobs
> work as well.
> The user ssh key stayed the same, but now it's for rdo-ci instead of
> rhos-ci, and the host is rdo-jenkins-2.ci.centos.org
> Attila