[Ci-users] [infra] : Hardware migration status (full/correct version)

Tue Oct 11 09:38:18 UTC 2016
Karanbir Singh <kbsingh at centos.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/10/16 08:46, Fabian Arrotin wrote:
> As announced some time ago ( 
> https://lists.centos.org/pipermail/centos-announce/2016-September/0220
65.html
>
> 
) , there was a big hardware migration/DC relocation happening yesterday
> that impacted several of our public services. Here is a small
> report of that migration, that finally took less time that
> previously announced.
> 
> * 2016-10-10 - 1pm UTC : we started to preconfigure all changes,
> and powered down all the services in correct order * 2016-10-10 -
> 2pm => 6pm UTC : hardware was moved/migrated * 2016-10-10 - 6pm =>
> 8pm UTC : hardware was (in serial order) plugged back in and
> ipmi/oob interfaces reconfigured, and network started to be 
> validated * 2016-10-10 - 8:30pm UTC : https://accounts.centos.org
> was back online and operational * 2016-10-10 - 9pm UTC :
> https://cbs.centos.org was back online but builders were still
> disabled for validation * 2016-10-10 - 9:45pm UTC :
> https://cbs.centos.org : builders were enabled and working again *
> 2016-10-11 - 3:30am UTC : https://ci.centos.org now fully
> operational (after some network reconfiguration needed
> post-migration, and validations
> 
> 
> Thanks a lot to Brian Stinson (on-site) and DC people who helped
> for that migration (Matt, Jon, Nathan, Nina, Phil)
> 
> Should you encounter an issue , feel free to discuss with us in 
> #centos-devel on irc.freenode.net
> 
> PS : sorry for the previous and incomplete mail : sent by accident 
> before finished and probably due to lack of coffee and short night
> ...
> 

Well done Fabian, Brian and everyone involved in this process, its
been a long way getting here, but its fantastic to see the move
executed so well.

Moving hundreds of machines across DC's is never fun, and to have done
this, brought all the equipment back up on an entirely new network, in
under 15 hrs, is commendable.

regards,

- -- 
Karanbir Singh, Project Lead, The CentOS Project
+44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS
GnuPG Key : http://www.karan.org/publickey.asc
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJX/LMJAAoJEI3Oi2Mx7xbtjKYH/38E1cKChoNxa5O/uXgvNSNC
Zykjw17y/09vzq5JVj+ucO5iVtXeFgsyp2hoOjmcOauubOAW3/wQdzmnEc3EyfBu
eIwHHQStl00s4cqRa/Krx6ITLPlYDR/Zf4/dKHc7qbhXzyfvOLeuQyGUhRyT5J0u
JtCm9WVPKaHvgBipGVLu7BJjnQV9FPol9D0qPUEJac2PsTepNvoPDJLBXT2GErKB
YtqaE8fnQ1PO0XFmcgCaFVoTWV+oom7XRZThoeK7qQ2b2dkPpdr3bKYVKvN5C4rO
pbmYp/60ITjXd/kkDFT6gJtJHF6aTYz192Ukq16e/Rxg0VKFx1P2PRoPLLTwOjs=
=9Xof
-----END PGP SIGNATURE-----