-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sep 14 16:30, Karanbir Singh wrote: > hi > > we've had a 4 hour outage on ci.centos.org for jobs running with the > githubprb plugin to validate PR's etc on github.com > > everything looks fixed now, please report if you continue to see issues. > Brian will have details on what broke etc later in the day > > regards > > -- > Karanbir Singh, Project Lead, The CentOS Project > +44-207-0999389 | http://www.centos.org/ | twitter.com/CentOS > GnuPG Key : http://www.karan.org/publickey.asc > > _______________________________________________ > Ci-users mailing list > Ci-users at centos.org > https://lists.centos.org/mailman/listinfo/ci-users TL;DR First: If you are using the github pull request builder plugin, please test if you can check the 'Use Use github hooks for build triggering' (`github-hooks: true` in your JJB trigger). Some of you have this unchecked to be sure that changes get picked up by polling in case a webhook gets missed. Please be sure to test your jobs to see if you can check the github hooks checkbox There was no single root cause for our trouble today but we were bitten by a few things at once: 1. There was a brief network outage between us and Github that would not normally cause an issue for us except for... 2. A bug in the Github-api plugin that doesn't handle timeouts well: https://issues.jenkins-ci.org/browse/JENKINS-45142 So we attempted to mitigate by deploying a new version of the plugin, but... 3. There was a regression that caused the Github Pull Request Builder to regress to an issue where we exhaust our entire Github API rate limit in about 15 minutes So we reverted to an older version of the ghprb plugin, but... 4. We're still using quite a bit of our rate limit, I'm waiting on a functional combination of github, github-api, and ghprb plugin updates that will make this more sane Thanks very much for your patience today all. - -- Brian Stinson CentOS CI Infrastructure Team -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJZu09rAAoJEIMGvNKzCweMUz4QAL7eJ7P2yHeAOHxMABBj0gnb 80EyOQGsrqVaTQdPsR9y2xwUSsXt4PAL6QmkiVtuviCaKcYQ/QCztAkjEwVZsDea stI3N3ITfQFsdsNwnvum1pXeyzeX8y+qNH5pIl3eeC99psy0QKBULGW4oxz+nwjn 5iuW51uABIsTZNOjcua6GLMCxEVQbIPOpVvR6bW3r+GMs6rcW7s6hdnDBPRkz84k 6LMC+1dNMSao51h2ZbWUXlnRnREATrLF+PVMkdnSZABqm8koI99AgSAOAWT8h70f ZFa8kFGTRDXBfVPfiaTWZZ4hH6SiSQHWsVesW5JKdzQ6CA+zPMDiv78+Sy8f0vP6 13/msZb3MyOdS8AdG4+H4rll//Y2KH+RyJOROaEJ1bFaKgy4MtyV6NjARkVyCFex s0P6CjoeVhTsqrnQMpGsMya3zGQmP2UJVAYfjhB7ULR0dnTa2MpCsMZVP1ehJ0Qr reO8Jpt5DMiFIz7guCqdJ/d+lLCdYVXHhPuUkngyowUQVAFapkJZ5jleMnX42Yt5 WLEXZeFYbzErcMkKq8gl/HU1hTTFYA/oWDpPMRDxAmYB8jvMFE7O123ejAAcl60n N9HCgJN2JHRMS9odgA3XhOFoT/ucKeW+aC98N36pAI5gB20tzv8/sNVu7se2q1Dd lXCca/dI9YVOYChRpeL0 =Rjyk -----END PGP SIGNATURE-----