[CentOS] Syncronize systemctl status with reality?

Tue Aug 29 20:52:04 UTC 2017

----- Original Message -----
From: "James Hogarth" <james.hogarth at gmail.com>
To: "centos" <centos at centos.org>
Sent: Tuesday, August 29, 2017 2:03:44 PM
Subject: Re: [CentOS] Syncronize systemctl status with reality?

On 29 Aug 2017 17:58, "Leroy Tennison" <leroy at datavoiceint.com> wrote:

The particular issue is with puppetmaster (which admittedly takes 4 minutes
to actually start, setting TimeoutStartSec=300 in it's unit file stopped
the false timeout report) but I have seen it one other time (don't remember
the details).

systemctl status puppetmaster
● puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor
preset: enabled)
Active: failed (Result: resources) since Tue 2017-08-29 11:24:36 CDT; 22min
ago
Process: 897 ExecStart=/usr/bin/puppet master (code=exited,
status=0/SUCCESS)

Aug 29 11:22:39 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Reopening log files
Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Starting Puppet master
version 3.8.5
Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Could not run: Address
already in use - listen(2)
Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: PID 1233
read from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 11:24:36 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: Unit
entered failed state.
Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: Failed
with result 'resources'.

However, ps -ef | grep puppet (run just after the above) returns
puppet 1380 1 0 11:26 ? 00:00:08 Passenger RubyApp: /usr/share/puppet/rack/
puppetmasterd
root 2015 1341 0 11:48 pts/0 00:00:00 grep --color=auto puppet

Earlier ps .. also reported
puppet 1355 1166 3 11:26 ? 00:00:01 Passenger AppPreloader:
/usr/share/puppet/rack/puppetmasterd

And, the "bottom line", puppet agent -t on a client works. It reports
finishing the catalog run and the client's yaml files on puppetmaster are
up to date.

Is there a command to tell systemd to re-scan running state and update its
understanding on what it finds? I tried systemctl daemon-reload just to be
sure that didn't solve the problem before posting this.
_______________________________________________
CentOS mailing list
CentOS at centos.org
https://lists.centos.org/mailman/listinfo/centos

First glance ity looks like someone has started that puppetmaster manually
at some point. As such it's not in a cgroup systemd is tracking so it isn't
aware of it.

Your attempts to start the service are failing because that manually
started instance already has the port open.

Kill it with pkill -f puppet and then use ss -tnp to check for the port
being freed (wait for any time_wait states to go... which is why I'm not
filtering by listen).

Once it's clear then try starting with systemctl
_______________________________________________
CentOS mailing list
CentOS at centos.org
https://lists.centos.org/mailman/listinfo/centos

OK, something weird is definitely going on here, I have the luxury of rebooting this system so it did.  Here's what I got, note the time stamps.

ps -ef | grep puppet
root       932     1  0 15:23 ?        00:00:00 /usr/bin/ruby /usr/bin/puppet master
root      1343  1327  0 15:24 pts/0    00:00:00 grep --color=auto puppet
(immediately afterward as fast as I could type:) uptime
 15:24:56 up 1 min,  1 user,  load average: 0.16, 0.07, 0.02
systemctl status puppetmaster
● puppetmaster.service - Puppet master
   Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor preset: enabled)
   Active: activating (start) since Tue 2017-08-29 15:23:44 CDT; 1min 24s ago
  Control: 932 (puppet)
    Tasks: 1
   Memory: 2.4M
      CPU: 4ms
   CGroup: /system.slice/puppetmaster.service
           └─932 /usr/bin/ruby /usr/bin/puppet master

Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...

After a short delay:
systemctl status puppetmaster
● puppetmaster.service - Puppet master
   Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor preset: enabled)
   Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; 11s ago
  Process: 932 ExecStart=/usr/bin/puppet master (code=exited, status=0/SUCCESS)

Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master version 3.8.5
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address already in use - listen(2)
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 read from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit entered failed state.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed with result 'resources'.

ps -ef | grep puppet 
root      1373  1327  0 15:25 pts/0    00:00:00 grep --color=auto puppet

date
Tue Aug 29 15:26:04 CDT 2017

uptime
 15:26:08 up 2 min,  1 user,  load average: 0.08, 0.06, 0.02

(Getting a little smarter...)

date;uptime;ps -ef | grep puppet        note the AppPreloader
Tue Aug 29 15:27:41 CDT 2017
 15:27:41 up 4 min,  1 user,  load average: 0.04, 0.05, 0.02
puppet    1377  1167  1 15:26 ?        00:00:01 Passenger AppPreloader: /usr/share/puppet/rack/puppetmasterd
puppet    1402  1377  4 15:26 ?        00:00:03 Passenger RubyApp: /usr/share/puppet/rack/puppetmasterd
root      1457  1327  0 15:27 pts/0    00:00:00 grep --color=auto puppet

date;uptime;systemctl status puppetmaster
Tue Aug 29 15:28:48 CDT 2017
 15:28:48 up 5 min,  1 user,  load average: 0.15, 0.10, 0.03
● puppetmaster.service - Puppet master
   Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor preset: enabled)
   Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; 3min 37s ago
  Process: 932 ExecStart=/usr/bin/puppet master (code=exited, status=0/SUCCESS)

Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master version 3.8.5
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address already in use - listen(2)
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 read from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit entered failed state.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed with result 'resources'.

(And finally)
date;uptime;ps -ef | grep puppet
Tue Aug 29 15:36:48 CDT 2017
 15:36:48 up 13 min,  1 user,  load average: 0.00, 0.01, 0.00
puppet    1402     1  0 15:26 ?        00:00:03 Passenger RubyApp: /usr/share/puppet/rack/puppetmasterd
root      1704  1327  0 15:36 pts/0    00:00:00 grep --color=auto puppet

date;uptime;systemctl status puppetmaster
Tue Aug 29 15:36:57 CDT 2017
 15:36:57 up 13 min,  1 user,  load average: 0.00, 0.01, 0.00
● puppetmaster.service - Puppet master
   Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor preset: enabled)
   Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; 11min ago
  Process: 932 ExecStart=/usr/bin/puppet master (code=exited, status=0/SUCCESS)

Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master version 3.8.5
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address already in use - listen(2)
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 read from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit entered failed state.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed with result 'resources'.

The AppPreloader is doing things (and probably confusing systemd in the process) but I didn't start that, it was a part of the reboot.  I looked at /etc/init.d/puppetmaster to see if something strange was being done and it was one of the simpler init scripts I've seen, uses start-stop-daemon.

I don't have any control over this, 'systemctl list-unit-files | grep puppet' shows the puppetmaster.service is enabled, I just need a "cleanup" solution, any ideas?  And, BTW, thanks for any feedback.