[Ci-users] Infra : scheduled hardware maintenance (Openshift/NFS)

Tue Sep 29 20:20:30 UTC 2020
Fabian Arrotin <arrfab at centos.org>

On 17/09/2020 16:30, Fabian Arrotin wrote:
> Due to a hardware maintenance that needs to take place on the NFS
> storage node used by openshift ("legacy" and current one - ocp ), we'll
> have to shutdown the openshift cluster, and then proceed with hardware
> maintenance on the NFS server (that itself needs to be powered down, no
> way to actually do that "online")
> 
> Migration is scheduled for """"Wednesday September 30th, 12:00 pm UTC
> time"""".
> You can convert to local time with $(date -d '2020-09-30 12:00 UTC')
> 
> The expected "downtime" is estimated to ~60 minutes , time needed to
> shutdown the machine, install new disks, restart the machine and also do
> some updates and tuning on the setup.
> 
> For more informations about this, here are some relevant tickets that
> were created for the perf issue in openshift and nfs :
> 
> https://pagure.io/centos-infra/issue/53
> https://pagure.io/centos-infra/issue/105
> https://pagure.io/centos-infra/issue/85
> https://pagure.io/centos-infra/issue/26
> 
> <subliminal message>
> PS : worth noting that while we'll investigate reports on new ocp
> cluster, we'll probably  not spend time investigating in the old/legacy
> one, that projects are supposed to migrate away from soon, as the legacy
> openshift setup will disappear soon (see
> https://pagure.io/centos-infra/issue/16)
> </subliminal message>
> 


Reminder !  :-)

Also, due to the needed time to also properly/cleanly power down all
nodes, we decided to start at 11:00 am UTC, to be ready when on-site
engineer will start un-racking storage server for hardware maintenance
and put it back online after (we have a fixed appointment for when to do it)

I'd like to remind all projects still on the old openshift cluster that
despite our calls to have projects migrated, only a very few did.
So we'll have discussion (centos ci infra team) about how to deal with
this but at first sight, we'll just announce a date/deadline for
decommissioning the old infra

-- 
Fabian Arrotin
The CentOS Project | https://www.centos.org
gpg key: 17F3B7A1 | twitter: @arrfab

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/ci-users/attachments/20200929/0a5aeded/attachment-0002.sig>