I'm currently trying to reboot a CentOS 7.5 workstation (to complete an upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop job is running for ...' - the counter initially gave a limit of '1min 30s' - but each time it reaches that limit, it just adds on ~90 seconds to the limit ...
Currently the limit is '25min 33s'
I'm in no hurry to have this workstation operational, but I guess at some point I will have to power cycle it ...
Does anyone know how to bypass this? - or at least stop it increasing the limit each time it is reached?
It does seems rather pointless to keep increasing the limit like this ...
Thanks
James Pearson
James Pearson wrote:
I'm currently trying to reboot a CentOS 7.5 workstation (to complete an upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop job is running for ...' - the counter initially gave a limit of '1min 30s' - but each time it reaches that limit, it just adds on ~90 seconds to the limit ...
Currently the limit is '25min 33s'
I'm in no hurry to have this workstation operational, but I guess at some point I will have to power cycle it ...
Does anyone know how to bypass this? - or at least stop it increasing the limit each time it is reached?
It does seems rather pointless to keep increasing the limit like this ...
It _finally_ gave up at 30 mins and rebooted
James Pearson
James Pearson wrote:
James Pearson wrote:
I'm currently trying to reboot a CentOS 7.5 workstation (to complete an upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop job is running for ...' - the counter initially gave a limit of '1min 30s' - but each time it reaches that limit, it just adds on ~90 seconds to the limit ...
Currently the limit is '25min 33s'
I'm in no hurry to have this workstation operational, but I guess at some point I will have to power cycle it ...
Does anyone know how to bypass this? - or at least stop it increasing the limit each time it is reached?
It does seems rather pointless to keep increasing the limit like this ...
It _finally_ gave up at 30 mins and rebooted
One question: did it have a mounted nfs filesystem?
The joys of systemd....
mark
mark
James Pearson wrote:
James Pearson wrote:
I'm currently trying to reboot a CentOS 7.5 workstation (to complete an upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop job is running for ...' - the counter initially gave a limit of '1min 30s' - but each time it reaches that limit, it just adds on ~90 seconds to the limit ...
Currently the limit is '25min 33s'
I'm in no hurry to have this workstation operational, but I guess at some point I will have to power cycle it ...
Does anyone know how to bypass this? - or at least stop it increasing the limit each time it is reached?
It does seems rather pointless to keep increasing the limit like this ...
It _finally_ gave up at 30 mins and rebooted
One question: did it have a mounted nfs filesystem?
The joys of systemd....
Yes, NFS integration with systemd is broken by default, at least it was still the case when I last checked. If you want it to work correctly, you have to add 'x-systemd.requires=network-online.target' as NFS mount option.
Clearly, how should systemd know that NFS won't work without network? I knew you agree :-)
Regards, Simon
On 22/05/2019 14:43, mark wrote:
James Pearson wrote:
James Pearson wrote:
I'm currently trying to reboot a CentOS 7.5 workstation (to complete an upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop job is running for ...' - the counter initially gave a limit of '1min 30s' - but each time it reaches that limit, it just adds on ~90 seconds to the limit ...
Currently the limit is '25min 33s'
I'm in no hurry to have this workstation operational, but I guess at some point I will have to power cycle it ...
Does anyone know how to bypass this? - or at least stop it increasing the limit each time it is reached?
It does seems rather pointless to keep increasing the limit like this ...
It _finally_ gave up at 30 mins and rebooted
One question: did it have a mounted nfs filesystem?
The joys of systemd....
mark mark
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
"Anything Windows can do, systemd can do better" (with apologies to Irving Berlin).
mark wrote:
James Pearson wrote:
James Pearson wrote:
I'm currently trying to reboot a CentOS 7.5 workstation (to complete an upgrade to 7.6), but it is 'stuck' while shutting down with 'A stop job is running for ...' - the counter initially gave a limit of '1min 30s' - but each time it reaches that limit, it just adds on ~90 seconds to the limit ...
Currently the limit is '25min 33s'
I'm in no hurry to have this workstation operational, but I guess at some point I will have to power cycle it ...
Does anyone know how to bypass this? - or at least stop it increasing the limit each time it is reached?
It does seems rather pointless to keep increasing the limit like this ...
It _finally_ gave up at 30 mins and rebooted
One question: did it have a mounted nfs filesystem?
All our boxes have NFS mounted files systems - and usually this isn't a problem - reboots work without an issue
In this case, it appeared to be 'stuck' on a local file bind mounted over a file on an NFS mounted file system
But that isn't really the point - I don't really want to have to wait a maximum of 30 minutes for the reboot to give up waiting for 'whatever'
Poking about a bit, I see that /usr/lib/systemd/system/reboot.target has the line:
JobTimeoutSec=30min
(there is a similar JobTimeoutSec=30min in poweroff.target)
I'm guessing I could create something like /etc/systemd/system/reboot.target.d/override.conf containing something like:
[Unit] JobTimeoutSec=3min
Now I need to see if I can reproduce the issue and see if this setting works ...
James Pearson
On Wed, May 22, 2019 at 7:44 AM mark m.roth@5-cent.us wrote:
The joys of systemd....
I'm not sure it's right to blame systemd. Systemd asked nicely for the service to shutdown. The service didn't, probably because the update change something and pulled the rug out from beneath it. Systemd then waited a bit to make sure the service wasn't just being slow, and finally gave up and forcibly killed it. I think this is a reasonable approach to killing a misbehaving service while trying to minimize data-loss, and the timeout can be configured.
This hasn't happened to me recently, but I think I've tried Ctl-C and Ctl-Alt-Del without much success. That leaves the Big Red Switch (which is mostly small and black these days).
Jim
Once upon a time, James Szinger jszinger@gmail.com said:
On Wed, May 22, 2019 at 7:44 AM mark m.roth@5-cent.us wrote:
The joys of systemd....
I'm not sure it's right to blame systemd. Systemd asked nicely for the service to shutdown. The service didn't, probably because the update change something and pulled the rug out from beneath it.
Right - before systemd, any old init script could also block shutdown.
This hasn't happened to me recently, but I think I've tried Ctl-C and Ctl-Alt-Del without much success. That leaves the Big Red Switch (which is mostly small and black these days).
There's a "magic" thing systemd does now - hit C-A-D seven times in two seconds and it'll stop what it is waiting for and just go ahead and reboot. Will kill anything not shut down, but at least it'll still try to cleanly unmount filesystems and such I believe.
On Wed, May 22, 2019 at 09:07:32AM -0600, James Szinger wrote:
On Wed, May 22, 2019 at 7:44 AM mark m.roth@5-cent.us wrote:
The joys of systemd....
I'm not sure it's right to blame systemd. Systemd asked nicely for the service to shutdown.
But we can blame systemd for the cryptic message
A stop job is running
Surely systemd knows what service it is waiting for, why doesn't it tell us?
The stop job XYZ is running
jon
On Wed, May 22, 2019 at 03:41:24PM -0400, Jon LaBadie wrote:
On Wed, May 22, 2019 at 09:07:32AM -0600, James Szinger wrote:
On Wed, May 22, 2019 at 7:44 AM mark m.roth@5-cent.us wrote:
The joys of systemd....
I'm not sure it's right to blame systemd. Systemd asked nicely for the service to shutdown.
But we can blame systemd for the cryptic message
A stop job is running
I didn't read this thread all that carefully, but has anyone mentioned editing /etc/systemd/system.conf and changing DefaultTimeoutStartSec and DefaultTimeoutStopSec to a lower value?
-- Scott Robbins PGP keyID EB3467D6 ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 ) gpg --keyserver pgp.mit.edu --recv-keys EB3467D6
Scott Robbins wrote:
** WARNING: This mail is from an external source **
On Wed, May 22, 2019 at 03:41:24PM -0400, Jon LaBadie wrote:
On Wed, May 22, 2019 at 09:07:32AM -0600, James Szinger wrote:
On Wed, May 22, 2019 at 7:44 AM mark m.roth@5-cent.us wrote:
The joys of systemd....
I'm not sure it's right to blame systemd. Systemd asked nicely for the service to shutdown.
But we can blame systemd for the cryptic message
A stop job is running
I didn't read this thread all that carefully, but has anyone mentioned editing /etc/systemd/system.conf and changing DefaultTimeoutStartSec and DefaultTimeoutStopSec to a lower value?
All the entries in /etc/systemd/system.conf are commented out - and as the systemd-system.conf man page states:
By default the configuration file in /etc/systemd/ contains commented out entries showing the defaults as a guide to the administrator
The DefaultTimeoutStopSec line in /etc/systemd/system.conf is:
#DefaultTimeoutStartSec=90s
In my case, the 'timeout' on whatever it was trying to do was 90 seconds - but it kept being increased by ~90 secs until it finally gave up at 30 minutes ...
So I don't think changing DefaultTimeoutStartSec will help here ?
There is no mention of 30 minutes in /etc/systemd/system.conf, that is why I'm guessing it has something to do with the JobTimeoutSec in /usr/lib/systemd/system/reboot.target
A bit of (more) googling seems to suggest that this is the setting that needs to be changed. I believe this means something like 'after 30 minutes from starting the reboot process, force a reboot regardless'
Personally, I think 30 minutes is far too long to wait, especially in the case of servers where no console access is available ...
James Pearson
On Wed, May 22, 2019 at 1:41 PM Jon LaBadie jcu@labadie.us wrote:
But we can blame systemd for the cryptic message
A stop job is running
Surely systemd knows what service it is waiting for, why doesn't it tell us?
The stop job XYZ is running
The message reported by the OP and the message I see is 'A stop job is running for ...' where the ellipses stand in for the unit that systemd is waiting for. It seems pretty clear IMHO. Actually debugging it is harder since the system is not available during shutdown, but that's a generic problem and the systemd docs do provide debugging tips.
Jim
-
Chris Adams -
J Martin Rushton -
James Pearson -
James Szinger -
Jon LaBadie -
mark -
Scott Robbins -
Simon Matter