[CentOS] Syncronize systemctl status with reality?
James Hogarth
james.hogarth at gmail.com
Wed Aug 30 18:23:39 UTC 2017
On 29 August 2017 at 21:52, Leroy Tennison <leroy at datavoiceint.com> wrote:
> ----- Original Message -----
> From: "James Hogarth" <james.hogarth at gmail.com>
> To: "centos" <centos at centos.org>
> Sent: Tuesday, August 29, 2017 2:03:44 PM
> Subject: Re: [CentOS] Syncronize systemctl status with reality?
>
> On 29 Aug 2017 17:58, "Leroy Tennison" <leroy at datavoiceint.com> wrote:
>
> The particular issue is with puppetmaster (which admittedly takes 4 minutes
> to actually start, setting TimeoutStartSec=300 in it's unit file stopped
> the false timeout report) but I have seen it one other time (don't remember
> the details).
>
> systemctl status puppetmaster
> ● puppetmaster.service - Puppet master
> Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor
> preset: enabled)
> Active: failed (Result: resources) since Tue 2017-08-29 11:24:36 CDT; 22min
> ago
> Process: 897 ExecStart=/usr/bin/puppet master (code=exited,
> status=0/SUCCESS)
>
> Aug 29 11:22:39 puppetmaster02 systemd[1]: Starting Puppet master...
> Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Reopening log files
> Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Starting Puppet master
> version 3.8.5
> Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Could not run: Address
> already in use - listen(2)
> Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: PID 1233
> read from file /run/puppet/master.pid does not exist or is a zombie.
> Aug 29 11:24:36 puppetmaster02 systemd[1]: Failed to start Puppet master.
> Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: Unit
> entered failed state.
> Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: Failed
> with result 'resources'.
>
> However, ps -ef | grep puppet (run just after the above) returns
> puppet 1380 1 0 11:26 ? 00:00:08 Passenger RubyApp: /usr/share/puppet/rack/
> puppetmasterd
> root 2015 1341 0 11:48 pts/0 00:00:00 grep --color=auto puppet
>
> Earlier ps .. also reported
> puppet 1355 1166 3 11:26 ? 00:00:01 Passenger AppPreloader:
> /usr/share/puppet/rack/puppetmasterd
>
> And, the "bottom line", puppet agent -t on a client works. It reports
> finishing the catalog run and the client's yaml files on puppetmaster are
> up to date.
>
> Is there a command to tell systemd to re-scan running state and update its
> understanding on what it finds? I tried systemctl daemon-reload just to be
> sure that didn't solve the problem before posting this.
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>
>
> First glance ity looks like someone has started that puppetmaster manually
> at some point. As such it's not in a cgroup systemd is tracking so it isn't
> aware of it.
>
> Your attempts to start the service are failing because that manually
> started instance already has the port open.
>
> Kill it with pkill -f puppet and then use ss -tnp to check for the port
> being freed (wait for any time_wait states to go... which is why I'm not
> filtering by listen).
>
> Once it's clear then try starting with systemctl
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>
> OK, something weird is definitely going on here, I have the luxury of
> rebooting this system so it did. Here's what I got, note the time stamps.
>
> ps -ef | grep puppet
> root 932 1 0 15:23 ? 00:00:00 /usr/bin/ruby
> /usr/bin/puppet master
> root 1343 1327 0 15:24 pts/0 00:00:00 grep --color=auto puppet
> (immediately afterward as fast as I could type:) uptime
> 15:24:56 up 1 min, 1 user, load average: 0.16, 0.07, 0.02
> systemctl status puppetmaster
> ● puppetmaster.service - Puppet master
> Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled;
> vendor preset: enabled)
> Active: activating (start) since Tue 2017-08-29 15:23:44 CDT; 1min 24s
> ago
> Control: 932 (puppet)
> Tasks: 1
> Memory: 2.4M
> CPU: 4ms
> CGroup: /system.slice/puppetmaster.service
> └─932 /usr/bin/ruby /usr/bin/puppet master
>
> Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
>
>
>
> After a short delay:
> systemctl status puppetmaster
> ● puppetmaster.service - Puppet master
> Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled;
> vendor preset: enabled)
> Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT;
> 11s ago
> Process: 932 ExecStart=/usr/bin/puppet master (code=exited,
> status=0/SUCCESS)
>
> Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
> Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
> Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master
> version 3.8.5
> Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address
> already in use - listen(2)
> Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360
> read from file /run/puppet/master.pid does not exist or is a zombie.
> Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
> Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit
> entered failed state.
> Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed
> with result 'resources'.
>
> ps -ef | grep puppet
> root 1373 1327 0 15:25 pts/0 00:00:00 grep --color=auto puppet
>
> date
> Tue Aug 29 15:26:04 CDT 2017
>
> uptime
> 15:26:08 up 2 min, 1 user, load average: 0.08, 0.06, 0.02
>
>
>
> (Getting a little smarter...)
>
> date;uptime;ps -ef | grep puppet note the AppPreloader
> Tue Aug 29 15:27:41 CDT 2017
> 15:27:41 up 4 min, 1 user, load average: 0.04, 0.05, 0.02
> puppet 1377 1167 1 15:26 ? 00:00:01 Passenger AppPreloader:
> /usr/share/puppet/rack/puppetmasterd
> puppet 1402 1377 4 15:26 ? 00:00:03 Passenger RubyApp:
> /usr/share/puppet/rack/puppetmasterd
> root 1457 1327 0 15:27 pts/0 00:00:00 grep --color=auto puppet
>
> date;uptime;systemctl status puppetmaster
> Tue Aug 29 15:28:48 CDT 2017
> 15:28:48 up 5 min, 1 user, load average: 0.15, 0.10, 0.03
> ● puppetmaster.service - Puppet master
> Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled;
> vendor preset: enabled)
> Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT;
> 3min 37s ago
> Process: 932 ExecStart=/usr/bin/puppet master (code=exited,
> status=0/SUCCESS)
>
> Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
> Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
> Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master
> version 3.8.5
> Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address
> already in use - listen(2)
> Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360
> read from file /run/puppet/master.pid does not exist or is a zombie.
> Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
> Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit
> entered failed state.
> Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed
> with result 'resources'.
>
>
>
> (And finally)
> date;uptime;ps -ef | grep puppet
> Tue Aug 29 15:36:48 CDT 2017
> 15:36:48 up 13 min, 1 user, load average: 0.00, 0.01, 0.00
> puppet 1402 1 0 15:26 ? 00:00:03 Passenger RubyApp:
> /usr/share/puppet/rack/puppetmasterd
> root 1704 1327 0 15:36 pts/0 00:00:00 grep --color=auto puppet
>
> date;uptime;systemctl status puppetmaster
> Tue Aug 29 15:36:57 CDT 2017
> 15:36:57 up 13 min, 1 user, load average: 0.00, 0.01, 0.00
> ● puppetmaster.service - Puppet master
> Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled;
> vendor preset: enabled)
> Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT;
> 11min ago
> Process: 932 ExecStart=/usr/bin/puppet master (code=exited,
> status=0/SUCCESS)
>
> Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
> Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
> Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master
> version 3.8.5
> Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address
> already in use - listen(2)
> Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360
> read from file /run/puppet/master.pid does not exist or is a zombie.
> Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
> Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit
> entered failed state.
> Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed
> with result 'resources'.
>
>
>
> The AppPreloader is doing things (and probably confusing systemd in the
> process) but I didn't start that, it was a part of the reboot. I looked at
> /etc/init.d/puppetmaster to see if something strange was being done and it
> was one of the simpler init scripts I've seen, uses start-stop-daemon.
>
> I don't have any control over this, 'systemctl list-unit-files | grep
> puppet' shows the puppetmaster.service is enabled, I just need a "cleanup"
> solution, any ideas? And, BTW, thanks for any feedback.
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>
Note that since systemctl shows puppetmaster.service as a unit file the
init script wont' be used at all so you can ignore the file in /etc/init.d
... that's just there because puppetlabs are being lazy and using the same
RPM on EL6 as EL7 rather than having the right files for the right
subsystems on the right distribution versions ... ./sigh
Otherwise you're down to standard troubleshooting process - use ss -tlnp to
identify the process that is holding the port open, systemctl cat
puppetmaster.service and see what it's trying to do etc etc
More information about the CentOS
mailing list