[CentOS-announce] *correction* Notification of outage : mirrorlist.centos.org (ipv4) 2016-04-16 08:58 UTC

Sat Apr 16 22:16:12 UTC 2016
Karanbir Singh <kbsingh at centos.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Apologies for this, the correct date should be : 2016-04-16 Sat. The
timestamps remain accurate.


On 16/04/16 10:09, Karanbir Singh wrote:
> Hi,
> 
> At 05:00 UTC 2016-05-16 an automated component update caused the 
> mirrorlist service for CentOS to go down for all IPv4 based
> services. The IPv6 service for mirrorlist.centos.org was
> unaffected.
> 
> By 08:11 UTC 2016-05-16 I rolled back the impacted components, 
> disabled the update mechanics, and restarted services.
> 
> By 08:15 UTC 2016-05-16 Services were returning to normal.
> 
> By 08:23 UTC 2016-05-16 we had multiple confirmations from around
> the world that services were restored.
> 
> --------------- system wide followup:
> 
> - We will work to extend and add where needed, better testing
> around each component involved in such roles.
> 
> - I will work with Fabian and make sure that all our automated 
> component and system changes that impact a production service are
> only run during regular working hours for the team.
> 
> --------------- root cause:
> 
> all the components involved in the downtime are backed up and we
> will start looking at the root cause of why services went down, for
> now the immediate focus was to restore services which was done by a
> rollback.
> 
> --------------- reporting issues:
> 
> Note that for real time, time sensitive issues always drop into 
> #centos-devel on irc.freenode.net and let us know, along with
> filing a bug report at bugs.centos.org - For any non time critical
> issues, please report them at bugs.centos.org against the
> 'Infrastructure' project, and we will aim to address them as soon
> as possible.
> 


- -- 
Karanbir Singh, Project Lead, The CentOS Project
+44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS
GnuPG Key : http://www.karan.org/publickey.asc
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJXErmsAAoJEI3Oi2Mx7xbtJdgIAI1+4KTYCdI4O+kFlrJ5IlMy
TwFGYt7O1WyGDfXC27VLehDxNP0lFiZOd1TMMSWy0/P64TYcBr5GLa4Vc7WLpLef
ywgjvgNE9VY69z4D1Lkgdxlgnhg31tPtW8kA1Py24nKrEFMYYi7/2tUGZn43kE8D
SRn0CywFizCFIScB0xI7uSJRE2PzFFwixIG4LaEdqppxcId6hsKb80hzpIP35pPU
eXlagl5CMad2XD0aVqKrqoU5W9vS9swKv9rc6ASsUHB28W7KcT0mq0YsIojhS81L
w8hiSsTpW3b62uBb55MIvkg1Y11H0qlkW306ZaZAXjTpRsvNaz82+qRmgcVVJvc=
=Uvpc
-----END PGP SIGNATURE-----