[Ci-users] Changes to CentOS CI: reminder of Phase 1 and 2

Tue Aug 23 14:55:09 UTC 2022
Fabian Arrotin <arrfab at centos.org>

On 23/08/2022 16:41, Camila Granella wrote:
> Hi all,
> 
> Earlier today the infra team attempted to bump the amount of metal machines
> available for provisioning on Duffy.
> However, the AWS API returned that currently there is no capacity to 
> provision
> metal machines in the Availability Zone we are currently in (us-east-1a).
> For this reason, we will need to default to the use of EC2.
> 

I had a look at the number of deployed c5n.metal instances for c8s and 
it reached 11 nodes ... so that also means that now Duffy is trying to 
have 5 nodes in Ready state (it was bumped from 1 to 5 through git 
commit/push earlier today)

It seems we're reaching a limit of c5n.metal available physical machines 
in us-east-1 (we use 3 availability zones there, through three subnets 
in dedicated duffy VPC)

Worth knowing that Duffy is catching ansible error and so knows that it 
was failing, so just retries every 60 seconds to provision such instance 
type machines , but by looking at the logs, we clearly ask much more 
than what AWS can offer. And that's also normal : AWS is about EC2 
Virtual Machines, not (costly) bare-metal options.
Also worth knowing that we added that option to let people transition 
their workflow but clearly metal option will be limited (by AWS 
availability, not even by us in this case) ....

For the time being, you can just put all your jobs in a queue, and retry 
to get one node through duffy api, if duffy itself was able to have some 
in ready state . At each point, one can see the pool status :

duffy client show-pool metal-ec2-c5n-centos-8s-x86_64
{
   "action": "get",
   "pool": {
     "name": "metal-ec2-c5n-centos-8s-x86_64",
     "fill_level": 5,
     "levels": {
       "provisioning": 0,
       "ready": 0,
       "contextualizing": 0,
       "deployed": 10,
       "deprovisioning": 0
     }
   }
}

In this case, it's showing 10 metal nodes deployed to tenants, and duffy 
not able to provision more (provisioning will show number and back to 
zero if it fails)


-- 
Fabian Arrotin
The CentOS Project | https://www.centos.org
gpg key: 17F3B7A1 | twitter: @arrfab
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0xA25DBAFB17F3B7A1.asc
Type: application/pgp-keys
Size: 12767 bytes
Desc: OpenPGP public key
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20220823/1f6e8481/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20220823/1f6e8481/attachment-0002.sig>