-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Apologies for this, the correct date should be : 2016-04-16 Sat. The timestamps remain accurate. On 16/04/16 10:09, Karanbir Singh wrote: > Hi, > > At 05:00 UTC 2016-05-16 an automated component update caused the > mirrorlist service for CentOS to go down for all IPv4 based > services. The IPv6 service for mirrorlist.centos.org was > unaffected. > > By 08:11 UTC 2016-05-16 I rolled back the impacted components, > disabled the update mechanics, and restarted services. > > By 08:15 UTC 2016-05-16 Services were returning to normal. > > By 08:23 UTC 2016-05-16 we had multiple confirmations from around > the world that services were restored. > > --------------- system wide followup: > > - We will work to extend and add where needed, better testing > around each component involved in such roles. > > - I will work with Fabian and make sure that all our automated > component and system changes that impact a production service are > only run during regular working hours for the team. > > --------------- root cause: > > all the components involved in the downtime are backed up and we > will start looking at the root cause of why services went down, for > now the immediate focus was to restore services which was done by a > rollback. > > --------------- reporting issues: > > Note that for real time, time sensitive issues always drop into > #centos-devel on irc.freenode.net and let us know, along with > filing a bug report at bugs.centos.org - For any non time critical > issues, please report them at bugs.centos.org against the > 'Infrastructure' project, and we will aim to address them as soon > as possible. > - -- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJXErmsAAoJEI3Oi2Mx7xbtJdgIAI1+4KTYCdI4O+kFlrJ5IlMy TwFGYt7O1WyGDfXC27VLehDxNP0lFiZOd1TMMSWy0/P64TYcBr5GLa4Vc7WLpLef ywgjvgNE9VY69z4D1Lkgdxlgnhg31tPtW8kA1Py24nKrEFMYYi7/2tUGZn43kE8D SRn0CywFizCFIScB0xI7uSJRE2PzFFwixIG4LaEdqppxcId6hsKb80hzpIP35pPU eXlagl5CMad2XD0aVqKrqoU5W9vS9swKv9rc6ASsUHB28W7KcT0mq0YsIojhS81L w8hiSsTpW3b62uBb55MIvkg1Y11H0qlkW306ZaZAXjTpRsvNaz82+qRmgcVVJvc= =Uvpc -----END PGP SIGNATURE-----