As announced some time ago ( https://lists.centos.org/pipermail/centos-announce/2016-September/022065.htm... ) , there was a big hardware migration/DC relocation happening yesterday that impacted several of our public services. Here is a small report of that migration, that finally took less time that previously announced.
* 2016-10-10 - 1pm UTC : we started to preconfigure all changes, and powered down all the services in correct order * 2016-10-10 - 2pm => 6pm UTC : hardware was moved/migrated * 2016-10-10 - 6pm => 8pm UTC : hardware was (in serial order) plugged back in and ipmi/oob interfaces reconfigured, and network started to be validated * 2016-10-10 - 8:30pm UTC : https://accounts.centos.org was back online and operational * 2016-10-10 - 9pm UTC : https://cbs.centos.org was back online but builders were still disabled for validation * 2016-10-10 - 9:45pm UTC : https://cbs.centos.org : builders were enabled and working again * 2016-10-11 - 3:30am UTC : https://ci.centos.org now fully operational (after some network reconfiguration needed post-migration, and validations
Thanks a lot to Brian Stinson (on-site) and DC people who helped for that migration (Matt, Jon, Nathan, Nina, Phil)
Should you encounter an issue , feel free to discuss with us in #centos-devel on irc.freenode.net
PS : sorry for the previous and incomplete mail : sent by accident before finished and probably due to lack of coffee and short night ...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/10/16 08:46, Fabian Arrotin wrote:
As announced some time ago ( https://lists.centos.org/pipermail/centos-announce/2016-September/0220
65.html
) , there was a big hardware migration/DC relocation happening yesterday
that impacted several of our public services. Here is a small report of that migration, that finally took less time that previously announced.
- 2016-10-10 - 1pm UTC : we started to preconfigure all changes,
and powered down all the services in correct order * 2016-10-10 - 2pm => 6pm UTC : hardware was moved/migrated * 2016-10-10 - 6pm => 8pm UTC : hardware was (in serial order) plugged back in and ipmi/oob interfaces reconfigured, and network started to be validated * 2016-10-10 - 8:30pm UTC : https://accounts.centos.org was back online and operational * 2016-10-10 - 9pm UTC : https://cbs.centos.org was back online but builders were still disabled for validation * 2016-10-10 - 9:45pm UTC : https://cbs.centos.org : builders were enabled and working again * 2016-10-11 - 3:30am UTC : https://ci.centos.org now fully operational (after some network reconfiguration needed post-migration, and validations
Thanks a lot to Brian Stinson (on-site) and DC people who helped for that migration (Matt, Jon, Nathan, Nina, Phil)
Should you encounter an issue , feel free to discuss with us in #centos-devel on irc.freenode.net
PS : sorry for the previous and incomplete mail : sent by accident before finished and probably due to lack of coffee and short night ...
Well done Fabian, Brian and everyone involved in this process, its been a long way getting here, but its fantastic to see the move executed so well.
Moving hundreds of machines across DC's is never fun, and to have done this, brought all the equipment back up on an entirely new network, in under 15 hrs, is commendable.
regards,
- -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc
On Tue, Oct 11, 2016 at 5:38 AM, Karanbir Singh kbsingh@centos.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/10/16 08:46, Fabian Arrotin wrote:
As announced some time ago ( https://lists.centos.org/pipermail/centos-announce/2016-September/0220
65.html
) , there was a big hardware migration/DC relocation happening yesterday
that impacted several of our public services. Here is a small report of that migration, that finally took less time that previously announced.
- 2016-10-10 - 1pm UTC : we started to preconfigure all changes,
and powered down all the services in correct order * 2016-10-10 - 2pm => 6pm UTC : hardware was moved/migrated * 2016-10-10 - 6pm => 8pm UTC : hardware was (in serial order) plugged back in and ipmi/oob interfaces reconfigured, and network started to be validated * 2016-10-10 - 8:30pm UTC : https://accounts.centos.org was back online and operational * 2016-10-10 - 9pm UTC : https://cbs.centos.org was back online but builders were still disabled for validation * 2016-10-10 - 9:45pm UTC : https://cbs.centos.org : builders were enabled and working again * 2016-10-11 - 3:30am UTC : https://ci.centos.org now fully operational (after some network reconfiguration needed post-migration, and validations
Thanks a lot to Brian Stinson (on-site) and DC people who helped for that migration (Matt, Jon, Nathan, Nina, Phil)
Should you encounter an issue , feel free to discuss with us in #centos-devel on irc.freenode.net
PS : sorry for the previous and incomplete mail : sent by accident before finished and probably due to lack of coffee and short night ...
Well done Fabian, Brian and everyone involved in this process, its been a long way getting here, but its fantastic to see the move executed so well.
Moving hundreds of machines across DC's is never fun, and to have done this, brought all the equipment back up on an entirely new network, in under 15 hrs, is commendable.
+1 -- I was super pumped to see everything back online first thing this morning. Good job, all!
regards,
Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux)
iQEcBAEBAgAGBQJX/LMJAAoJEI3Oi2Mx7xbtjKYH/38E1cKChoNxa5O/uXgvNSNC Zykjw17y/09vzq5JVj+ucO5iVtXeFgsyp2hoOjmcOauubOAW3/wQdzmnEc3EyfBu eIwHHQStl00s4cqRa/Krx6ITLPlYDR/Zf4/dKHc7qbhXzyfvOLeuQyGUhRyT5J0u JtCm9WVPKaHvgBipGVLu7BJjnQV9FPol9D0qPUEJac2PsTepNvoPDJLBXT2GErKB YtqaE8fnQ1PO0XFmcgCaFVoTWV+oom7XRZThoeK7qQ2b2dkPpdr3bKYVKvN5C4rO pbmYp/60ITjXd/kkDFT6gJtJHF6aTYz192Ukq16e/Rxg0VKFx1P2PRoPLLTwOjs= =9Xof -----END PGP SIGNATURE----- _______________________________________________ Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
-
Fabian Arrotin -
Harry Rybacki -
Karanbir Singh