[Ci-users] Major infrastructure outage: 2017-09-19

Tue Sep 19 21:10:28 UTC 2017
Brian Stinson <bstinson at redhat.com>

Incident Date:

Incident Time:
20:20 UTC

Approximate outage Time
35 minutes

Services Affected:
- Local mirrors hosted at mirror*.ci.centos.org 
- NFS mounted directories to the CI storage node
- CI Monitoring

The server that services mirror requests in ci.centos.org was hard
locked. We got into the console and rebooted the server and service is
now restored. There is a short period left when we are pausing jobs in
CI to let Duffy catch back up. 

Thank you for your patience.

Brian Stinson
CentOS CI Infrastructure Team