Hi,
The RDO slave currently has 15 execution threads running on two cores and 4GB of RAM. As we use ci.centos more and more for gating and periodic jobs, I can see that the slave starts to struggle a little bit CPU-wise when the execution threads are busy.
The bulk of the jobs are offloaded to an ephemeral bare metal node but ansible still runs from the slave. Some jobs do run locally, such as smaller tox jobs.
What's the strategy if we want to scale this ? For example, do we want to scale in increments of "10 threads/2 cores/4 GB RAM" by adding more slaves as need be or do we want to grow the specifications of the existing slave and grow the amount of threads ?
I have a personal preference to have more smaller slaves as this also has the added benefit of redundancy so we're not left entirely out in the cold if this one big slave happens to crash.
David Moreau Simard Senior Software Engineer | Openstack RDO
dmsimard = [irc, github, twitter]
On 19/12/15 18:02, David Moreau Simard wrote:
The bulk of the jobs are offloaded to an ephemeral bare metal node but ansible still runs from the slave. Some jobs do run locally, such as smaller tox jobs.
What's the strategy if we want to scale this ?
At the moment the slaves all run from VM's hosted near the admin and jenkins instances - and are setup manually, and managed manually - this was very much a stop gap arrangement till we can get a better virtualised setup in place. We've been looking at and trying to scope up getting an RDO cloud in place, that could then be used for 3 things :
1) making an openstack api available for people who want to just consume VM's for their workloads
2) migrating the slaves into openstack managed ( ie, self/user managed ) virtual machines
3) offering up image backed resources for people looking at doing testing with other OSs, eg what the libvirt and libguestfs folks do at the moment.
We have a dedicated hardware slab ( ~ 24 phy machines worth ) dedicated to this task ( so as to not cut into the ci baremetal pools ), but are waiting on the RH facility folks to get it wired up and dial-toned.
Given the nature and impact of this setup, I am going to try and see if we can speed up delivery of that infra from the present timeline of end Feb '16.
regards,
On 13/01/16 12:52, Karanbir Singh wrote:
On 19/12/15 18:02, David Moreau Simard wrote:
The bulk of the jobs are offloaded to an ephemeral bare metal node but ansible still runs from the slave. Some jobs do run locally, such as smaller tox jobs.
What's the strategy if we want to scale this ?
At the moment the slaves all run from VM's hosted near the admin and jenkins instances - and are setup manually, and managed manually - this was very much a stop gap arrangement till we can get a better virtualised setup in place. We've been looking at and trying to scope up getting an RDO cloud in place, that could then be used for 3 things :
- making an openstack api available for people who want to just consume
VM's for their workloads
[snip]
- offering up image backed resources for people looking at doing
testing with other OSs, eg what the libvirt and libguestfs folks do at the moment.
We have a dedicated hardware slab ( ~ 24 phy machines worth ) dedicated to this task ( so as to not cut into the ci baremetal pools ), but are waiting on the RH facility folks to get it wired up and dial-toned.
Given the nature and impact of this setup, I am going to try and see if we can speed up delivery of that infra from the present timeline of end Feb '16.
I'd just like to chime in and say that the ability to provision VMs on OpenStack (instead of C6/7 physical hosts) would be a big help for me on migrating the Foreman tests to CentOS CI. We currently run VMs on Rackspace for CentOS, Debian, Fedora and Ubuntu so need a broad selection of images which Duffy + bare metal hosts doesn't provide.
I've been trying to spin up VMs on the physical hosts for each OS, but it's fiddly to set up compared to spinning up a pre-built VM image on OpenStack. (Little networking setup issues, virt-install workarounds, bugs in our tests that assume certain package lists, and the complexity of another provisioning layer.)
Looking forward to this, thanks for the update.
On Mon, Jan 25, 2016 at 9:35 AM, Dominic Cleal dominic@cleal.org wrote:
I'd just like to chime in and say that the ability to provision VMs on OpenStack (instead of C6/7 physical hosts) would be a big help for me on migrating the Foreman tests to CentOS CI. We currently run VMs on Rackspace for CentOS, Debian, Fedora and Ubuntu so need a broad selection of images which Duffy + bare metal hosts doesn't provide.
I've been trying to spin up VMs on the physical hosts for each OS, but it's fiddly to set up compared to spinning up a pre-built VM image on OpenStack. (Little networking setup issues, virt-install workarounds, bugs in our tests that assume certain package lists, and the complexity of another provisioning layer.)
Looking forward to this, thanks for the update.
I have scripts which make it easy to spin up vagrant + oh-my-vagrant vms. Would this help?
On 25/01/16 14:40, James wrote:
On Mon, Jan 25, 2016 at 9:35 AM, Dominic Cleal dominic@cleal.org wrote:
I'd just like to chime in and say that the ability to provision VMs on OpenStack (instead of C6/7 physical hosts) would be a big help for me on migrating the Foreman tests to CentOS CI. We currently run VMs on Rackspace for CentOS, Debian, Fedora and Ubuntu so need a broad selection of images which Duffy + bare metal hosts doesn't provide.
I've been trying to spin up VMs on the physical hosts for each OS, but it's fiddly to set up compared to spinning up a pre-built VM image on OpenStack. (Little networking setup issues, virt-install workarounds, bugs in our tests that assume certain package lists, and the complexity of another provisioning layer.)
Looking forward to this, thanks for the update.
I have scripts which make it easy to spin up vagrant + oh-my-vagrant vms. Would this help?
Probably not, we already have scripts in the project that I don't wish to replace.
On 13/01/16 12:52, Karanbir Singh wrote:
We've been looking at and trying to scope up getting an RDO cloud in place, that could then be used for 3 things :
- making an openstack api available for people who want to just consume
VM's for their workloads
- migrating the slaves into openstack managed ( ie, self/user managed )
virtual machines
- offering up image backed resources for people looking at doing
testing with other OSs, eg what the libvirt and libguestfs folks do at the moment.
We have a dedicated hardware slab ( ~ 24 phy machines worth ) dedicated to this task ( so as to not cut into the ci baremetal pools ), but are waiting on the RH facility folks to get it wired up and dial-toned.
Given the nature and impact of this setup, I am going to try and see if we can speed up delivery of that infra from the present timeline of end Feb '16.
Do you expect this service to be available soon?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 16/05/16 10:46, Dominic Cleal wrote:
On 13/01/16 12:52, Karanbir Singh wrote:
We've been looking at and trying to scope up getting an RDO cloud in place, that could then be used for 3 things :
- making an openstack api available for people who want to just
consume VM's for their workloads
- migrating the slaves into openstack managed ( ie, self/user
managed ) virtual machines
- offering up image backed resources for people looking at
doing testing with other OSs, eg what the libvirt and libguestfs folks do at the moment.
We have a dedicated hardware slab ( ~ 24 phy machines worth ) dedicated to this task ( so as to not cut into the ci baremetal pools ), but are waiting on the RH facility folks to get it wired up and dial-toned.
Given the nature and impact of this setup, I am going to try and see if we can speed up delivery of that infra from the present timeline of end Feb '16.
Do you expect this service to be available soon?
I've got the basic stuff in place, and we should be able to open for wider testing in the next day or so.
regards
- -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc
On 16/05/16 12:26, Karanbir Singh wrote:
On 16/05/16 10:46, Dominic Cleal wrote:
On 13/01/16 12:52, Karanbir Singh wrote:
We've been looking at and trying to scope up getting an RDO cloud in place, that could then be used for 3 things :
- making an openstack api available for people who want to just
consume VM's for their workloads
- migrating the slaves into openstack managed ( ie, self/user
managed ) virtual machines
- offering up image backed resources for people looking at
doing testing with other OSs, eg what the libvirt and libguestfs folks do at the moment.
We have a dedicated hardware slab ( ~ 24 phy machines worth ) dedicated to this task ( so as to not cut into the ci baremetal pools ), but are waiting on the RH facility folks to get it wired up and dial-toned.
Given the nature and impact of this setup, I am going to try and see if we can speed up delivery of that infra from the present timeline of end Feb '16.
Do you expect this service to be available soon?
I've got the basic stuff in place, and we should be able to open for wider testing in the next day or so.
That's great news, thanks. I'd be happy to help test.
-
David Moreau Simard -
Dominic Cleal -
James -
Karanbir Singh -
Karanbir Singh