Recently i've migrated our SVN server (virtual machine) from C6 to C7 (more precisely - migrated data to freshly installed virtual machine). And we have problem with very slow authentication. Server is configured with SSSD, user data are fetching from our LDAP server. SVN is configured with apache (pwauth for authentication + LDAP search for Require ldap-group).
It takes pwauth even 10 seconds to authenticate. Whet it comes to svn's externals it could take as long as 9 minutes to _svn up_ project (when there are no commits to fetch). Every external may take even 15 seconds (and sometimes even more).
SSSD was configured at first with authconfig / authconfig-tui. I was struggling with SSSD configuration but with no success. I'm not sure where to look (SSSD, apache?). How can i debug this issue?
sssd.conf: [domain/default] autofs_provider = ldap cache_credentials = True krb5_realm = # ldap_search_base = ou=Main,o=company id_provider = ldap auth_provider = ldap chpass_provider = ldap ldap_uri = ldap://ldap.our.domain/ ldap_group_search_base = ou=Group,ou=Main,o=company ldap_user_search_base = ou=People,ou=Main,o=company ldap_id_use_start_tls = False ldap_tls_cacertdir = /etc/openldap/cacerts ldap_tls_reqcert = allow #debug_level = 4 refresh_expired_interval = 120 enumerate = True ldap_referrals = False [sssd] services = nss, pam, autofs config_file_version = 2
domains = default [nss] homedir_substring = /home entry_cache_timeout = 5400
[pam] pam_id_timeout=20
apache: LDAPCacheTTL 30 <VirtualHost 10.0.32.19:80> ErrorLog logs/svn_http_error_log CustomLog logs/svn_http_access_log "%t %u %{SVN-ACTION}e" env=SVN-ACTION ServerName svn.our.domain DirectoryIndex none DefineExternalAuth pwauth pipe /usr/bin/pwauth #AddExternalGroup unixgroup /usr/sbin/unixgroup #SetExternalGroupMethod unixgroup environment
<Location /> SVNPathAuthz off DAV svn SVNPath /home/repos/subversion_free_avr
AuthBasicAuthoritative off AuthBasicProvider socache external AuthExternal pwauth AuthnCacheProvideFor external AuthType Basic AuthName "Subversion repository" AuthLDAPURL ldap://ldap.our.domain/ou=Main,o=company AuthLDAPGroupAttribute memberUid AuthLDAPGroupAttributeIsDN off Require ldap-group cn=programmers,ou=group,ou=main,o=company #GroupExternal unixgroup #Require group programmers #Require valid-user #AuthzSVNAccessFile /home/repos/svn.access </Location> </VirtualHost>
On same server we have redmine (with database on separate server and LDAP auth) and git repositories (with gitbucket as frontend, also LDAP auth) but those repos aren't extensively used right now. Redmine works not-so-bad, so i guess it is not overall server performance issue.
Disks performance (measured under normal workload): [root@luah pam.d]# hdparm -tT /dev/vda
/dev/vda: (system) Timing cached reads: 11412 MB in 2.00 seconds = 5710.28 MB/sec Timing buffered disk reads: 522 MB in 3.63 seconds = 143.79 MB/sec [root@luah pam.d]# hdparm -tT /dev/vdd
/dev/vdd: (/home where all data resides) Timing cached reads: 10020 MB in 2.00 seconds = 5013.17 MB/sec Timing buffered disk reads: 172 MB in 3.20 seconds = 53.73 MB/sec
It's comparable with other our VMs.
Any ideas?
On Mon, Apr 11, 2016 at 05:22:43PM +0200, Leon Fauster wrote:
Am 11.04.2016 um 15:44 schrieb Marcin Trendota moonwolf.rh@gmail.com:
Any ideas?
DNS?
Is LDAP listed in the /etc/nsswitch.conf?
W dniu 11.04.2016 o 20:07, Scott Robbins pisze:
Any ideas?
DNS?
Is LDAP listed in the /etc/nsswitch.conf?
In nsswitch.conf i have: passwd: files sss shadow: files sss group: files sss
DNS works fine. I think that sssd communicates with LDAP server with every authentication - i have tons of following entries in log:
And it repeats for same user over and over again. Is this correct behavior?
W dniu 12.04.2016 o 09:45, Marcin Trendota pisze:
DNS works fine. I think that sssd communicates with LDAP server with every authentication - i have tons of following entries in log: http://pastebin.com/rZVjk0gW And it repeats for same user over and over again. Is this correct behavior?
authconfig --enableforcelegacy --disablesssd --disablesssdauth --updateall
Did the trick - so it looks like it's problem with SSSD. I would like not to remove it, but it's production server and must works flawlessly.
On Tue, Apr 12, 2016 at 09:45:17AM +0200, Marcin Trendota wrote:
W dniu 11.04.2016 o 20:07, Scott Robbins pisze:
Any ideas?
DNS?
Is LDAP listed in the /etc/nsswitch.conf?
In nsswitch.conf i have: passwd: files sss shadow: files sss group: files sss
DNS works fine. I think that sssd communicates with LDAP server with every authentication - i have tons of following entries in log:
And it repeats for same user over and over again. Is this correct behavior?
RedHat never really mastered LDAP, unfortunately. I have a by now ancient article, that mentions it.
What I found back then is that if it EVER uses LDAP for anything, upon boot, it will look for an LDAP server. It doesn't even have to be one that the workstation uses--it just wants to know there's one on the network.
In other words, I have server1 which workstation1 can use for authentication, and server2, which does, say, address book duty.
The workstation also does authenticate for some things against the server1. If server1 is off, although there's a local account on workstation, it will look and hang for awhile, till it gives up and finally logs on the local account.
However, if server2 is on--that is, as long as there's an LDAP server that it can see, even though that server isn't used for authentication, workstation1 will, IIRC, try it, see it can't authenticate, and then use local authentication.
Again, this is something that happened a long time ago and I haven't tried in a long while. However, that's why I asked if LDAP was involved, because it might be the issue.
On 12 Apr 2016 16:29, "Scott Robbins" scottro11@gmail.com wrote:
On Tue, Apr 12, 2016 at 09:45:17AM +0200, Marcin Trendota wrote:
W dniu 11.04.2016 o 20:07, Scott Robbins pisze:
Any ideas?
DNS?
Is LDAP listed in the /etc/nsswitch.conf?
In nsswitch.conf i have: passwd: files sss shadow: files sss group: files sss
DNS works fine. I think that sssd communicates with LDAP server with every authentication - i have tons of following entries in log:
http:// http://pastebin.com/rZVjk0gWpastebin.com
http://pastebin.com/rZVjk0gW/ http://pastebin.com/rZVjk0gWrZVjk0gW http://pastebin.com/rZVjk0gW
And it repeats for same user over and over again. Is this correct
behavior?
RedHat never really mastered LDAP, unfortunately. I have a by now ancient article, that mentions it.
<snip>
What utter nonsense. Just because you poorly configured your system does not mean that Red Hat never really mastered it... And translating very old experiences to CentOS 7 is even more ridiculous and counter productive.
To the OP enumerate is always painful, I'd remove that for a start.
My experience with the DAV SVN though is that clients are horrible in their requests... So many it hits it so hard...
After various testing I ended up going with the Apache LDAP cache module and doing the auth at the Apache level, not system.
Was far better in performance with the SVN server being hit fairly hard. I can try and dig out an example configuration if you would like.
The bonus here as well is that svn users are separated cleanly from system users... No reason for a dev to have a shell account on there ;)
On 04/12/2016 09:51 AM, James Hogarth wrote:
To the OP enumerate is always painful, I'd remove that for a start.
This was my experience too, for what it's worth. When I first set up a new system pointed at LDAP it was absurdly slow to authenticate. Setting Enumerate to False in /etc/sssd/sssd.conf made all the difference.
On 4/12/2016 7:56 PM, David Nelson wrote:
On 04/12/2016 09:51 AM, James Hogarth wrote:
To the OP enumerate is always painful, I'd remove that for a start.
This was my experience too, for what it's worth. When I first set up a new system pointed at LDAP it was absurdly slow to authenticate. Setting Enumerate to False in /etc/sssd/sssd.conf made all the difference.
Hello,
I had similar problem recently with Centos6 machine, which was in another country and had ~100ms latency to the LDAP server. When I did "id user", it took around 20 seconds. I did some debugging, and when the user was not a member of additional groups, it was much faster (5 seconds), but still slow. It seems that for each member of a group, the client did a query to the LDAP server. I put "ignore_group_members = true" in sssd.conf and now it's much faster. Can you try this?
Regards,
On 4/12/16 12:15 PM, Todor Petkov wrote:
On 4/12/2016 7:56 PM, David Nelson wrote:
On 04/12/2016 09:51 AM, James Hogarth wrote:
To the OP enumerate is always painful, I'd remove that for a start.
This was my experience too, for what it's worth. When I first set up a new system pointed at LDAP it was absurdly slow to authenticate. Setting Enumerate to False in /etc/sssd/sssd.conf made all the difference.
Hello,
I had similar problem recently with Centos6 machine, which was in another country and had ~100ms latency to the LDAP server. When I did "id user", it took around 20 seconds. I did some debugging, and when the user was not a member of additional groups, it was much faster (5 seconds), but still slow. It seems that for each member of a group, the client did a query to the LDAP server. I put "ignore_group_members = true" in sssd.conf and now it's much faster. Can you try this?
Regards,
In my particular case the server is already widely used so I'm not in a good position to test it. But next time I have to set up a new system that authenticates against LDAP, I'll be sure to do that!
W dniu 12.04.2016 o 18:56, David Nelson pisze:
On 04/12/2016 09:51 AM, James Hogarth wrote:
To the OP enumerate is always painful, I'd remove that for a start.
This was my experience too, for what it's worth. When I first set up a new system pointed at LDAP it was absurdly slow to authenticate. Setting Enumerate to False in /etc/sssd/sssd.conf made all the difference.
At the beginning it was off. I've turned it on hoping it will eventually speed up authentication. With no success.
James Hogarth wrote:
On 12 Apr 2016 16:29, "Scott Robbins" scottro11@gmail.com wrote:
On Tue, Apr 12, 2016 at 09:45:17AM +0200, Marcin Trendota wrote:
W dniu 11.04.2016 o 20:07, Scott Robbins pisze:
<SNIP>
After various testing I ended up going with the Apache LDAP cache module and doing the auth at the Apache level, not system.
Was far better in performance with the SVN server being hit fairly hard. I can try and dig out an example configuration if you would like.
The bonus here as well is that svn users are separated cleanly from system users... No reason for a dev to have a shell account on there ;)
I'd be *very* interested in that configuration, if you post it here, or offlist, to me.
mark
On Tue, April 12, 2016 11:57 am, m.roth@5-cent.us wrote:
James Hogarth wrote:
On 12 Apr 2016 16:29, "Scott Robbins" scottro11@gmail.com wrote:
On Tue, Apr 12, 2016 at 09:45:17AM +0200, Marcin Trendota wrote:
W dniu 11.04.2016 o 20:07, Scott Robbins pisze:
<SNIP> > After various testing I ended up going with the Apache LDAP cache module > and doing the auth at the Apache level, not system. > > Was far better in performance with the SVN server being hit > fairly hard. I can try and dig out an example configuration if > you would like. > > The bonus here as well is that svn users are separated cleanly > from system users... No reason for a dev to have a shell account > on there ;)
I'd be *very* interested in that configuration, if you post it here, or offlist, to me.
Me too. Please, post for everyone, or add me to off-list message.
Valeri
mark
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On 12 April 2016 at 18:03, Valeri Galtsev galtsev@kicp.uchicago.edu wrote:
On Tue, April 12, 2016 11:57 am, m.roth@5-cent.us wrote:
James Hogarth wrote:
On 12 Apr 2016 16:29, "Scott Robbins" scottro11@gmail.com wrote:
On Tue, Apr 12, 2016 at 09:45:17AM +0200, Marcin Trendota wrote:
W dniu 11.04.2016 o 20:07, Scott Robbins pisze:
<SNIP> > After various testing I ended up going with the Apache LDAP cache module > and doing the auth at the Apache level, not system. > > Was far better in performance with the SVN server being hit > fairly hard. I can try and dig out an example configuration if > you would like. > > The bonus here as well is that svn users are separated cleanly > from system users... No reason for a dev to have a shell account > on there ;)
I'd be *very* interested in that configuration, if you post it here, or offlist, to me.
Me too. Please, post for everyone, or add me to off-list message.
Valeri
The CA.crt assumes that is used to sign the LDAPS certs ... replace as required ;) This assumes multiple SVN repos under /srv/svn/repos This includes a local userfile for any quick hacks or system things that you don't want to hit LDAP for - can be removed. This also allows fallback from one server to another if need be, note that it will need to timeout on the first though. This took a fair chunk of load off of our LDAP server and made checkouts a far more pleasant experience.
Bonus points if you get your CM to change ordering of LDAP servers between repos (or other web auth) ;) _____________________________________________________________
LDAPTrustedGlobalCert CA_BASE64 /etc/pki/tls/certs/CA.crt
# Enable caching by mod_ldap LDAPSharedCacheSize 500000 LDAPCacheEntries 1024 LDAPCacheTTL 600 LDAPOpCacheEntries 1024 LDAPOpCacheTTL 600
<Location /ldap-status> SSLRequire true SetHandler ldap-status </Location>
<Location /repos> DAV svn SVNParentPath /srv/svn/repos </Location>
<Location /repos/repo1> SSLRequireSSL AuthName "SVN Repo 1" AuthType Basic AuthLDAPBindDN cn=svnbind,cn=systemusers,dc=example,dc=com AuthLDAPBindPassword plaintextpassword AuthUserFile /etc/httpd/svnpasswd
AuthLDAPURL "ldaps://ldapserver1.example.com/dc=example,dc=com?uid ldaps://ldapserver2.example.com/dc=example,dc=com?uid "
AuthBasicProvider file ldap AuthzLDAPAuthoritative off AuthLDAPGroupAttribute member AuthLDAPGroupAttributeIsDN On
# READ <Limit OPTIONS PROPFIND GET REPORT> Require ldap-group cn=dev,cn=groups,dc=example,dc=com Require ldap-group cn=qa,cn=groups,dc=example,dc=com </Limit> # WRITE <LimitExcept OPTIONS PROPFIND GET REPORT> Require ldap-group cn=dev,cn=groups,dc=example,dc=com </LimitExcept> </Location>
On Tue, April 12, 2016 2:33 pm, James Hogarth wrote:
On 12 April 2016 at 18:03, Valeri Galtsev galtsev@kicp.uchicago.edu wrote:
On Tue, April 12, 2016 11:57 am, m.roth@5-cent.us wrote:
James Hogarth wrote:
On 12 Apr 2016 16:29, "Scott Robbins" scottro11@gmail.com wrote:
On Tue, Apr 12, 2016 at 09:45:17AM +0200, Marcin Trendota wrote:
W dniu 11.04.2016 o 20:07, Scott Robbins pisze:
<SNIP>
<snip>
Thanks a lot!
The moment my svn servers start getting hit big time I will know what to change, thanks to you!
Valeri
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
-
David Nelson -
James Hogarth -
Leon Fauster -
m.roth＠5-cent.us -
Marcin Trendota -
Scott Robbins -
Todor Petkov -
Valeri Galtsev