Hi Folks,
In response to news of directed attacks against public Jenkins instances[0], we are enabling some of the CSRF protections in ci.centos.org
To do this we will issue a SafeRestart at 14:30 UTC Today! Running jobs will be given a chance to clear and new jobs should be queued up and will execute as soon as the restart finishes.
Potential Impact: - If you are using the Jenkins REST interface you may need to modify your scripts to send the appropriate headers[1]
- Jenkins Job Builder is tracking an issue to enable CSRF support[2]. Some basic tests were performed on our side, and simple jobs were configured correctly, but you may notice strange behavior if you are using JJB.
[0]: https://groups.google.com/d/topic/jenkinsci-advisories/lJfvDs5s6bk [1]: https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API#RemoteaccessAP... [2]: https://storyboard.openstack.org/#!/story/2000556
If you have any questions or comments, let us know here or find one of us in #centos-devel on Freenode.
Cheers! -- Brian Stinson CentOS CI Infrastructure Team
The first part of this maintenance has been done. We will need to schedule a full restart for tonight (00h UTC). We'll be monitoring running jobs throughout the day.
Cheers --Brian
On Apr 19 08:54, Brian Stinson wrote:
On Tue, Apr 19, 2016, at 11:03 AM, Brian Stinson wrote:
Trying to access: https://ci.centos.org/job/atomic-rdgo-centos7/40/console I get:
HTTP ERROR 403
Problem accessing /job/atomic-rdgo-centos7/40/logText/progressiveHtml. Reason:
No valid crumb was included in the request
I get this regardless of whether or not I'm logged in. My guess is that the master needs the full restart?
On Apr 19 11:18, Colin Walters wrote:
We were able to temporarily work around this, you should be able to get in to the consoles until tonight's full restart which will fix the issue you mentioned and re-enable the rest of the CSRF protections.
--Brian
Hi All,
This is finally finished up. We had a long quiet period while a few jobs finished up and it looks like everything got queued up for re-execution once we restarted.
We'll be checking in with the JJB folks and others using the Jenkins REST API to see how they're affected by the new CSRF settings.
Cheers! --Brian
On Apr 19 10:03, Brian Stinson wrote:
On 04/19/2016 10:56 PM, Brian Stinson wrote:
I tested a JJB push for RDO, and it worked fine. However, I have a very odd issue that correlates timing wise with this restart.
The image building jobs in the RDO promotion pipelines[1] are all failing the first time they try to get an image via a 'file://' URL. The first occurrence of this was in the middle of the night last night[2], and there have been no code or CI changes in that time frame. I dont have a good explanation of how this could be related to the jenkins restart, as that image building is happening on a duffy node. On the other hand, it seems suspicious timing given that no code or CI changes happened.
[1] https://ci.centos.org/view/rdo/view/promotion-pipeline/ [2] https://ci.centos.org/job/tripleo-quickstart-promote-master-delorean-build-i...
On 04/20/2016 07:39 AM, John Trowbridge wrote:
Sorry for the noise. correlation!=causation. Looks like ansible released 2.0.2 last night as well, which makes much more sense as a root cause.
On Tue, Apr 19, 2016, at 09:54 AM, Brian Stinson wrote:
It looks like this also caused:
https://github.com/janinko/ghprb/issues/84
However I'm a bit confused - it seems like a lot more people should be hitting this. Perhaps people just aren't turning on CSRF?
Then I also found https://github.com/jenkinsci/ghprb-plugin/commit/cb8447f991aebe3de688d3548c4... which: $ git describe --contains cb8447f991aebe3de688d3548c451dd128e16900 ghprb-1.28~3^2
So it *should* be in the 1.30.4 we're running according to https://ci.centos.org/pluginManager/api/json?tree=plugins%5BshortName,versio...]
Did anyone else manage to get the ghprb hooks working?
(Aside, I was trying to work around this by using the raw `github` plugin's webhook which does work, but I couldn't quite figure out how to make a single job that builds multiple PRs be "stable", i.e. avoid retriggering for previously built PRs, plus in the end we do need a way to retrigger as ghprb handles)