I think that the most secure setup is to use both LDAPI (ldap connections over Unix sockets) for connections inside the ldap server and TLS for connections from everywhere else on the network. Plus, ldapi connections are much faster than TCP connections.
Am I wrong?