[Ci-users] githubprb outage in ci today

Fri Sep 15 03:56:27 UTC 2017
Brian Stinson <brian at bstinson.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 14 16:30, Karanbir Singh wrote:
> hi
> 
> we've had a 4 hour outage on ci.centos.org for jobs running with the
> githubprb plugin to validate PR's etc on github.com
> 
> everything looks fixed now, please report if you continue to see issues.
> Brian will have details on what broke etc later in the day
> 
> regards
> 
> -- 
> Karanbir Singh, Project Lead, The CentOS Project
> +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS
> GnuPG Key : http://www.karan.org/publickey.asc
> 




> _______________________________________________
> Ci-users mailing list
> Ci-users at centos.org
> https://lists.centos.org/mailman/listinfo/ci-users

TL;DR First:

If you are using the github pull request builder plugin, please test if
you can check the 'Use Use github hooks for build triggering'
(`github-hooks: true` in your JJB trigger). 

Some of you have this unchecked to be sure that changes get picked up by
polling in case a webhook gets missed. Please be sure to test your jobs
to see if you can check the github hooks checkbox


There was no single root cause for our trouble today but we were bitten
by a few things at once: 

1. There was a brief network outage between us and Github that would not
   normally cause an issue for us except for...

2. A bug in the Github-api plugin that doesn't handle timeouts well:
   https://issues.jenkins-ci.org/browse/JENKINS-45142

   So we attempted to mitigate by deploying a new version of the plugin,
   but...

3. There was a regression that caused the Github Pull Request Builder to
   regress to an issue where we exhaust our entire Github API rate limit
   in about 15 minutes

   So we reverted to an older version of the ghprb plugin, but...

4. We're still using quite a bit of our rate limit, I'm waiting on a
   functional combination of github, github-api, and ghprb plugin updates
   that will make this more sane

Thanks very much for your patience today all.

- --
Brian Stinson 
CentOS CI Infrastructure Team
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJZu09rAAoJEIMGvNKzCweMUz4QAL7eJ7P2yHeAOHxMABBj0gnb
80EyOQGsrqVaTQdPsR9y2xwUSsXt4PAL6QmkiVtuviCaKcYQ/QCztAkjEwVZsDea
stI3N3ITfQFsdsNwnvum1pXeyzeX8y+qNH5pIl3eeC99psy0QKBULGW4oxz+nwjn
5iuW51uABIsTZNOjcua6GLMCxEVQbIPOpVvR6bW3r+GMs6rcW7s6hdnDBPRkz84k
6LMC+1dNMSao51h2ZbWUXlnRnREATrLF+PVMkdnSZABqm8koI99AgSAOAWT8h70f
ZFa8kFGTRDXBfVPfiaTWZZ4hH6SiSQHWsVesW5JKdzQ6CA+zPMDiv78+Sy8f0vP6
13/msZb3MyOdS8AdG4+H4rll//Y2KH+RyJOROaEJ1bFaKgy4MtyV6NjARkVyCFex
s0P6CjoeVhTsqrnQMpGsMya3zGQmP2UJVAYfjhB7ULR0dnTa2MpCsMZVP1ehJ0Qr
reO8Jpt5DMiFIz7guCqdJ/d+lLCdYVXHhPuUkngyowUQVAFapkJZ5jleMnX42Yt5
WLEXZeFYbzErcMkKq8gl/HU1hTTFYA/oWDpPMRDxAmYB8jvMFE7O123ejAAcl60n
N9HCgJN2JHRMS9odgA3XhOFoT/ucKeW+aC98N36pAI5gB20tzv8/sNVu7se2q1Dd
lXCca/dI9YVOYChRpeL0
=Rjyk
-----END PGP SIGNATURE-----