-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Sep 14 16:30, Karanbir Singh wrote:

hi

we've had a 4 hour outage on ci.centos.org for jobs running with the githubprb plugin to validate PR's etc on github.com

everything looks fixed now, please report if you continue to see issues. Brian will have details on what broke etc later in the day

regards

-- Karanbir Singh, Project Lead, The CentOS Project +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS GnuPG Key : http://www.karan.org/publickey.asc

---

Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users

TL;DR First:

If you are using the github pull request builder plugin, please test if you can check the 'Use Use github hooks for build triggering' (`github-hooks: true` in your JJB trigger).

Some of you have this unchecked to be sure that changes get picked up by polling in case a webhook gets missed. Please be sure to test your jobs to see if you can check the github hooks checkbox

There was no single root cause for our trouble today but we were bitten by a few things at once:

1. There was a brief network outage between us and Github that would not normally cause an issue for us except for...

2. A bug in the Github-api plugin that doesn't handle timeouts well: https://issues.jenkins-ci.org/browse/JENKINS-45142

So we attempted to mitigate by deploying a new version of the plugin, but...

3. There was a regression that caused the Github Pull Request Builder to regress to an issue where we exhaust our entire Github API rate limit in about 15 minutes

So we reverted to an older version of the ghprb plugin, but...

4. We're still using quite a bit of our rate limit, I'm waiting on a functional combination of github, github-api, and ghprb plugin updates that will make this more sane

Thanks very much for your patience today all.

- -- Brian Stinson CentOS CI Infrastructure Team