Hi Folks,
In response to news of directed attacks against public Jenkins instances[0], we are enabling some of the CSRF protections in ci.centos.org
To do this we will issue a SafeRestart at 14:30 UTC Today! Running jobs will be given a chance to clear and new jobs should be queued up and will execute as soon as the restart finishes.
Potential Impact: - If you are using the Jenkins REST interface you may need to modify your scripts to send the appropriate headers[1]
- Jenkins Job Builder is tracking an issue to enable CSRF support[2]. Some basic tests were performed on our side, and simple jobs were configured correctly, but you may notice strange behavior if you are using JJB.
[0]: https://groups.google.com/d/topic/jenkinsci-advisories/lJfvDs5s6bk [1]: https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API#RemoteaccessAP... [2]: https://storyboard.openstack.org/#!/story/2000556
If you have any questions or comments, let us know here or find one of us in #centos-devel on Freenode.
Cheers! -- Brian Stinson CentOS CI Infrastructure Team
The first part of this maintenance has been done. We will need to schedule a full restart for tonight (00h UTC). We'll be monitoring running jobs throughout the day.
Cheers --Brian
On Apr 19 08:54, Brian Stinson wrote:
Hi Folks,
In response to news of directed attacks against public Jenkins instances[0], we are enabling some of the CSRF protections in ci.centos.org
To do this we will issue a SafeRestart at 14:30 UTC Today! Running jobs will be given a chance to clear and new jobs should be queued up and will execute as soon as the restart finishes.
Potential Impact:
If you are using the Jenkins REST interface you may need to modify your scripts to send the appropriate headers[1]
Jenkins Job Builder is tracking an issue to enable CSRF support[2]. Some basic tests were performed on our side, and simple jobs were configured correctly, but you may notice strange behavior if you are using JJB.
If you have any questions or comments, let us know here or find one of us in #centos-devel on Freenode.
Cheers!
Brian Stinson CentOS CI Infrastructure Team _______________________________________________ Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
On Tue, Apr 19, 2016, at 11:03 AM, Brian Stinson wrote:
The first part of this maintenance has been done. We will need to schedule a full restart for tonight (00h UTC). We'll be monitoring running jobs throughout the day.
Trying to access: https://ci.centos.org/job/atomic-rdgo-centos7/40/console I get:
HTTP ERROR 403
Problem accessing /job/atomic-rdgo-centos7/40/logText/progressiveHtml. Reason:
No valid crumb was included in the request
I get this regardless of whether or not I'm logged in. My guess is that the master needs the full restart?
On Apr 19 11:18, Colin Walters wrote:
On Tue, Apr 19, 2016, at 11:03 AM, Brian Stinson wrote:
The first part of this maintenance has been done. We will need to schedule a full restart for tonight (00h UTC). We'll be monitoring running jobs throughout the day.
Trying to access: https://ci.centos.org/job/atomic-rdgo-centos7/40/console I get:
HTTP ERROR 403
Problem accessing /job/atomic-rdgo-centos7/40/logText/progressiveHtml. Reason:
No valid crumb was included in the requestI get this regardless of whether or not I'm logged in. My guess is that the master needs the full restart? _______________________________________________ Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
We were able to temporarily work around this, you should be able to get in to the consoles until tonight's full restart which will fix the issue you mentioned and re-enable the rest of the CSRF protections.
--Brian
Hi All,
This is finally finished up. We had a long quiet period while a few jobs finished up and it looks like everything got queued up for re-execution once we restarted.
We'll be checking in with the JJB folks and others using the Jenkins REST API to see how they're affected by the new CSRF settings.
Cheers! --Brian
On Apr 19 10:03, Brian Stinson wrote:
The first part of this maintenance has been done. We will need to schedule a full restart for tonight (00h UTC). We'll be monitoring running jobs throughout the day.
Cheers --Brian
On Apr 19 08:54, Brian Stinson wrote:
Hi Folks,
In response to news of directed attacks against public Jenkins instances[0], we are enabling some of the CSRF protections in ci.centos.org
To do this we will issue a SafeRestart at 14:30 UTC Today! Running jobs will be given a chance to clear and new jobs should be queued up and will execute as soon as the restart finishes.
Potential Impact:
If you are using the Jenkins REST interface you may need to modify your scripts to send the appropriate headers[1]
Jenkins Job Builder is tracking an issue to enable CSRF support[2]. Some basic tests were performed on our side, and simple jobs were configured correctly, but you may notice strange behavior if you are using JJB.
If you have any questions or comments, let us know here or find one of us in #centos-devel on Freenode.
Cheers!
Brian Stinson CentOS CI Infrastructure Team _______________________________________________ Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
On 04/19/2016 10:56 PM, Brian Stinson wrote:
Hi All,
This is finally finished up. We had a long quiet period while a few jobs finished up and it looks like everything got queued up for re-execution once we restarted.
We'll be checking in with the JJB folks and others using the Jenkins REST API to see how they're affected by the new CSRF settings.
Cheers! --Brian
I tested a JJB push for RDO, and it worked fine. However, I have a very odd issue that correlates timing wise with this restart.
The image building jobs in the RDO promotion pipelines[1] are all failing the first time they try to get an image via a 'file://' URL. The first occurrence of this was in the middle of the night last night[2], and there have been no code or CI changes in that time frame. I dont have a good explanation of how this could be related to the jenkins restart, as that image building is happening on a duffy node. On the other hand, it seems suspicious timing given that no code or CI changes happened.
[1] https://ci.centos.org/view/rdo/view/promotion-pipeline/ [2] https://ci.centos.org/job/tripleo-quickstart-promote-master-delorean-build-i...
On Apr 19 10:03, Brian Stinson wrote:
The first part of this maintenance has been done. We will need to schedule a full restart for tonight (00h UTC). We'll be monitoring running jobs throughout the day.
Cheers --Brian
On Apr 19 08:54, Brian Stinson wrote:
Hi Folks,
In response to news of directed attacks against public Jenkins instances[0], we are enabling some of the CSRF protections in ci.centos.org
To do this we will issue a SafeRestart at 14:30 UTC Today! Running jobs will be given a chance to clear and new jobs should be queued up and will execute as soon as the restart finishes.
Potential Impact:
If you are using the Jenkins REST interface you may need to modify your scripts to send the appropriate headers[1]
Jenkins Job Builder is tracking an issue to enable CSRF support[2]. Some basic tests were performed on our side, and simple jobs were configured correctly, but you may notice strange behavior if you are using JJB.
If you have any questions or comments, let us know here or find one of us in #centos-devel on Freenode.
Cheers!
Brian Stinson CentOS CI Infrastructure Team _______________________________________________ Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
On 04/20/2016 07:39 AM, John Trowbridge wrote:
On 04/19/2016 10:56 PM, Brian Stinson wrote:
Hi All,
This is finally finished up. We had a long quiet period while a few jobs finished up and it looks like everything got queued up for re-execution once we restarted.
We'll be checking in with the JJB folks and others using the Jenkins REST API to see how they're affected by the new CSRF settings.
Cheers! --Brian
I tested a JJB push for RDO, and it worked fine. However, I have a very odd issue that correlates timing wise with this restart.
The image building jobs in the RDO promotion pipelines[1] are all failing the first time they try to get an image via a 'file://' URL. The first occurrence of this was in the middle of the night last night[2], and there have been no code or CI changes in that time frame. I dont have a good explanation of how this could be related to the jenkins restart, as that image building is happening on a duffy node. On the other hand, it seems suspicious timing given that no code or CI changes happened.
Sorry for the noise. correlation!=causation. Looks like ansible released 2.0.2 last night as well, which makes much more sense as a root cause.
[1] https://ci.centos.org/view/rdo/view/promotion-pipeline/ [2] https://ci.centos.org/job/tripleo-quickstart-promote-master-delorean-build-i...
On Apr 19 10:03, Brian Stinson wrote:
The first part of this maintenance has been done. We will need to schedule a full restart for tonight (00h UTC). We'll be monitoring running jobs throughout the day.
Cheers --Brian
On Apr 19 08:54, Brian Stinson wrote:
Hi Folks,
In response to news of directed attacks against public Jenkins instances[0], we are enabling some of the CSRF protections in ci.centos.org
To do this we will issue a SafeRestart at 14:30 UTC Today! Running jobs will be given a chance to clear and new jobs should be queued up and will execute as soon as the restart finishes.
Potential Impact:
If you are using the Jenkins REST interface you may need to modify your scripts to send the appropriate headers[1]
Jenkins Job Builder is tracking an issue to enable CSRF support[2]. Some basic tests were performed on our side, and simple jobs were configured correctly, but you may notice strange behavior if you are using JJB.
If you have any questions or comments, let us know here or find one of us in #centos-devel on Freenode.
Cheers!
Brian Stinson CentOS CI Infrastructure Team _______________________________________________ Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
On Tue, Apr 19, 2016, at 09:54 AM, Brian Stinson wrote:
Hi Folks,
In response to news of directed attacks against public Jenkins instances[0], we are enabling some of the CSRF protections in ci.centos.org
It looks like this also caused:
https://github.com/janinko/ghprb/issues/84
However I'm a bit confused - it seems like a lot more people should be hitting this. Perhaps people just aren't turning on CSRF?
Then I also found https://github.com/jenkinsci/ghprb-plugin/commit/cb8447f991aebe3de688d3548c4... which: $ git describe --contains cb8447f991aebe3de688d3548c451dd128e16900 ghprb-1.28~3^2
So it *should* be in the 1.30.4 we're running according to https://ci.centos.org/pluginManager/api/json?tree=plugins%5BshortName,versio...]
Did anyone else manage to get the ghprb hooks working?
(Aside, I was trying to work around this by using the raw `github` plugin's webhook which does work, but I couldn't quite figure out how to make a single job that builds multiple PRs be "stable", i.e. avoid retriggering for previously built PRs, plus in the end we do need a way to retrigger as ghprb handles)
-
Brian Stinson -
Colin Walters -
John Trowbridge