Hi everyone,
We seem to be having issues on multiple CentOS 7.3 machines. The problem seems to revolve around polkitd. At some random time, polkitd seems to stop responding on my systems. Along with this, there might be hundreds of defunct pkla-check-authorization processes. If I reboot, then things are fine for a while.
I don't see any activity in the unabridged journal to suggest anything that might be triggering polkitd. The puppet run finished 5 minutes before polkitd lost it's head.
polkit version is polkit-0.112-11.el7_3.x86_64
Any help is appreciated.
Thanks, Jason
Here is some condensed output from the "journalctl -u polkit" command: Mar 09 04:02:14 myhost systemd[1]: Starting Authorization Manager... Mar 09 04:02:14 myhost polkitd[1018]: Started polkitd version 0.112 Mar 09 04:02:14 myhost polkitd[1018]: Loading rules from directory /etc/polkit-1/rules.d Mar 09 04:02:14 myhost polkitd[1018]: Loading rules from directory /usr/share/polkit-1/rules.d Mar 09 04:02:14 myhost polkitd[1018]: Finished loading, compiling and executing 7 rules Mar 09 04:02:14 myhost systemd[1]: Started Authorization Manager. Mar 09 04:02:14 myhost polkitd[1018]: Acquired the name org.freedesktop.PolicyKit1 on the system bus Mar 09 04:02:53 myhost polkitd[1018]: Registered Authentication Agent for unix-session:c1 (system bus name :1.41 [gnome-shell --mode=gdm], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) Mar 09 04:08:25 myhost polkitd[1018]: Reloading rules Mar 09 04:08:25 myhost polkitd[1018]: Collecting garbage unconditionally... Mar 09 04:08:25 myhost polkitd[1018]: Loading rules from directory /etc/polkit-1/rules.d Mar 09 04:08:25 myhost polkitd[1018]: Loading rules from directory /usr/share/polkit-1/rules.d Mar 09 04:08:25 myhost polkitd[1018]: Finished loading, compiling and executing 8 rules Mar 09 04:08:25 myhost polkitd[1018]: Reloading rules Mar 09 04:08:25 myhost polkitd[1018]: Collecting garbage unconditionally... Mar 09 04:08:25 myhost polkitd[1018]: Loading rules from directory /etc/polkit-1/rules.d Mar 09 04:08:25 myhost polkitd[1018]: Loading rules from directory /usr/share/polkit-1/rules.d Mar 09 04:08:25 myhost polkitd[1018]: Finished loading, compiling and executing 8 rules Mar 09 04:08:53 myhost polkitd[1018]: Reloading rules ... (snipped more rules loading) Mar 09 04:10:39 myhost polkitd[1018]: Collecting garbage unconditionally... Mar 09 04:10:39 myhost polkitd[1018]: Loading rules from directory /etc/polkit-1/rules.d Mar 09 04:10:39 myhost polkitd[1018]: Loading rules from directory /usr/share/polkit-1/rules.d Mar 09 04:10:39 myhost polkitd[1018]: Finished loading, compiling and executing 7 rules Mar 09 16:59:42 myhost polkitd[1018]: /etc/polkit-1/rules.d/49-polkit-pkla-compat.rules:21: Error: Error spawning helper: Timed out after 10 seconds (g-io-error-quark, 24) Mar 09 16:59:42 myhost polkitd[1018]: Error evaluating authorization rules Mar 10 04:13:34 myhost polkitd[1018]: /etc/polkit-1/rules.d/49-polkit-pkla-compat.rules:21: Error: Error spawning helper: Timed out after 10 seconds (g-io-error-quark, 24) Mar 10 04:13:34 myhost polkitd[1018]: Error evaluating authorization rules Mar 10 04:14:32 myhost polkitd[1018]: /etc/polkit-1/rules.d/49-polkit-pkla-compat.rules:21: Error: Error spawning helper: Timed out after 10 seconds (g-io-error-quark, 24) ... (snipped more lines about error evaluating rules)...
--------------------------------------------------------------------------- Jason Edgecombe | Linux Administrator UNC Charlotte | The William States Lee College of Engineering 9201 University City Blvd. | Charlotte, NC 28223-0001 Phone: 704-687-1943 jwedgeco@uncc.edu | http://engr.uncc.edu | Facebook --------------------------------------------------------------------------- If you are not the intended recipient of this transmission or a person responsible for delivering it to the intended recipient, any disclosure, copying, distribution, or other use of any of the information in this transmission is strictly prohibited. If you have received this transmission in error, please notify me immediately by reply e-mail or by telephone at 704-687-1943. Thank you.
Hi everyone,
I'm replying to myself to help anyone else who happens to get the polkit timeouts. Our CentOS7 machines are joined to our Active Directory domain and use AD for authentication and account lookups (Using the SSSD AD provider). We're NOT using FreeIPA. The polkit timeouts were caused by sssd taking too long to respond to user information lookups for users that were in Active Directory.
The solution was to set "enumerate = False" in /etc/sssd/sssd.conf and restart the sssd service or reboot the machine. If "enumerate" is not present in sssd.conf, then it defaults to False.
In addition to the polkit hangs, we were also experiencing the following problems, which went away or improved after the sssd.conf change was made:
- Running "id $USERNAME" was taking many seconds when looking up users in Active Directory - Logins were taking a while (5+ seconds) or would just hang - Unlocking a machine from the screensaver would sometimes fail. - General system sluggishness. - High system CPU/load with no obvious culprits according to "top" - The sssd_be process was often taking 5% or more of a CPU.
The problems were more prevalent on our big time-sharing systems (64 cores/512GB RAM), that have multiple (15+) simultaneous users running large memory or CPU interactive jobs. The problems also hit some of our single-user workstations, but the most-affected users were still our compute-heavy research users.
For others' reference, I'm also running the sssd cache on a tmpfs filesystem and here is my sanitized sssd.conf file.:
[sssd] config_file_version = 2 services = nss, pam domains = subdomain.example.com
[domain/subdomain.example.com] ad_domain = subdomain.example.com krb5_realm = subdomain.example.com realmd_tags = manages-system joined-with-samba cache_credentials = True id_provider = ad auth_provider = ad chpass_provider = ad
enumerate = False access_provider = ad krb5_store_password_if_offline = True ldap_id_mapping = False use_fully_qualified_names = False krb5_renewable_lifetime = 60d krb5_lifetime = 60d krb5_renew_interval = 600s
Sincerely, Jason
--------------------------------------------------------------------------- Jason Edgecombe | Linux Administrator UNC Charlotte | The William States Lee College of Engineering 9201 University City Blvd. | Charlotte, NC 28223-0001 Phone: 704-687-1943 jwedgeco@uncc.edu | http://engr.uncc.edu | Facebook --------------------------------------------------------------------------- If you are not the intended recipient of this transmission or a person responsible for delivering it to the intended recipient, any disclosure, copying, distribution, or other use of any of the information in this transmission is strictly prohibited. If you have received this transmission in error, please notify me immediately by reply e-mail or by telephone at 704-687-1943. Thank you.
On Fri, Mar 10, 2017 at 1:01 PM, Edgecombe, Jason jwedgeco@uncc.edu wrote:
Hi everyone,
We seem to be having issues on multiple CentOS 7.3 machines. The problem seems to revolve around polkitd. At some random time, polkitd seems to stop responding on my systems. Along with this, there might be hundreds of defunct pkla-check-authorization processes. If I reboot, then things are fine for a while.
I don't see any activity in the unabridged journal to suggest anything that might be triggering polkitd. The puppet run finished 5 minutes before polkitd lost it's head.
polkit version is polkit-0.112-11.el7_3.x86_64
Any help is appreciated.
Thanks, Jason
Here is some condensed output from the "journalctl -u polkit" command: Mar 09 04:02:14 myhost systemd[1]: Starting Authorization Manager... Mar 09 04:02:14 myhost polkitd[1018]: Started polkitd version 0.112 Mar 09 04:02:14 myhost polkitd[1018]: Loading rules from directory /etc/polkit-1/rules.d Mar 09 04:02:14 myhost polkitd[1018]: Loading rules from directory /usr/share/polkit-1/rules.d Mar 09 04:02:14 myhost polkitd[1018]: Finished loading, compiling and executing 7 rules Mar 09 04:02:14 myhost systemd[1]: Started Authorization Manager. Mar 09 04:02:14 myhost polkitd[1018]: Acquired the name org.freedesktop.PolicyKit1 on the system bus Mar 09 04:02:53 myhost polkitd[1018]: Registered Authentication Agent for unix-session:c1 (system bus name :1.41 [gnome-shell --mode=gdm], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) Mar 09 04:08:25 myhost polkitd[1018]: Reloading rules Mar 09 04:08:25 myhost polkitd[1018]: Collecting garbage unconditionally... Mar 09 04:08:25 myhost polkitd[1018]: Loading rules from directory /etc/polkit-1/rules.d Mar 09 04:08:25 myhost polkitd[1018]: Loading rules from directory /usr/share/polkit-1/rules.d Mar 09 04:08:25 myhost polkitd[1018]: Finished loading, compiling and executing 8 rules Mar 09 04:08:25 myhost polkitd[1018]: Reloading rules Mar 09 04:08:25 myhost polkitd[1018]: Collecting garbage unconditionally... Mar 09 04:08:25 myhost polkitd[1018]: Loading rules from directory /etc/polkit-1/rules.d Mar 09 04:08:25 myhost polkitd[1018]: Loading rules from directory /usr/share/polkit-1/rules.d Mar 09 04:08:25 myhost polkitd[1018]: Finished loading, compiling and executing 8 rules Mar 09 04:08:53 myhost polkitd[1018]: Reloading rules ... (snipped more rules loading) Mar 09 04:10:39 myhost polkitd[1018]: Collecting garbage unconditionally... Mar 09 04:10:39 myhost polkitd[1018]: Loading rules from directory /etc/polkit-1/rules.d Mar 09 04:10:39 myhost polkitd[1018]: Loading rules from directory /usr/share/polkit-1/rules.d Mar 09 04:10:39 myhost polkitd[1018]: Finished loading, compiling and executing 7 rules Mar 09 16:59:42 myhost polkitd[1018]: /etc/polkit-1/rules.d/49-polkit-pkla-compat.rules:21: Error: Error spawning helper: Timed out after 10 seconds (g-io-error-quark, 24) Mar 09 16:59:42 myhost polkitd[1018]: Error evaluating authorization rules Mar 10 04:13:34 myhost polkitd[1018]: /etc/polkit-1/rules.d/49-polkit-pkla-compat.rules:21: Error: Error spawning helper: Timed out after 10 seconds (g-io-error-quark, 24) Mar 10 04:13:34 myhost polkitd[1018]: Error evaluating authorization rules Mar 10 04:14:32 myhost polkitd[1018]: /etc/polkit-1/rules.d/49-polkit-pkla-compat.rules:21: Error: Error spawning helper: Timed out after 10 seconds (g-io-error-quark, 24) ... (snipped more lines about error evaluating rules)...
Jason Edgecombe | Linux Administrator UNC Charlotte | The William States Lee College of Engineering 9201 University City Blvd. | Charlotte, NC 28223-0001 Phone: 704-687-1943 jwedgeco@uncc.edu | http://engr.uncc.edu | Facebook
If you are not the intended recipient of this transmission or a person responsible for delivering it to the intended recipient, any disclosure, copying, distribution, or other use of any of the information in this transmission is strictly prohibited. If you have received this transmission in error, please notify me immediately by reply e-mail or by telephone at 704-687-1943. Thank you.
-
Edgecombe, Jason