Skip to content

Instantly share code, notes, and snippets.

@aderixon

aderixon/openldap-tlsmc-php-bug.md

Last active October 8, 2019 10:40
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save aderixon/01ee459155a5f51264cb0f029c4b6f87 to your computer and use it in GitHub Desktop.
Save aderixon/01ee459155a5f51264cb0f029c4b6f87 to your computer and use it in GitHub Desktop.
Download ZIP
CentOS/RHEL 7.5+ OpenLDAP TLS bug with self-signed certificate & PHP LDAP
Raw
openldap-tlsmc-php-bug.md

We're seeing an intermittent issue with PHP LDAP against a TLS connection using a self-signed server certificate on CentOS 7.6, in which for some connections the certificate fails to verify (despite the CA cert being present on the client). This only affects a web application using PHP-FPM and the standard PHP LDAP module - LDAP authentication through sssd works fine.

Components:

  • CentOS Linux release 7.6.1810
  • sssd-1.16.2-13.el7_6.8.x86_64
  • openldap-2.4.44-21.el7_6.x86_64
  • nss-3.36.0-7.1.el7_6.x86_64
  • php72u-fpm-7.2.22-1.el7.ius.x86_64
  • php72u-ldap-7.2.22-1.el7.ius.x86_64

(Same behaviour observed on CentOS 7.7.)

LDAP CA certificate is installed correctly and working in /etc/openldap/cacerts:

lrwxrwxrwx 1 root root   25 Aug 31  2018 47785293.0 -> authconfig_downloaded.pem
-rw-r--r-- 1 root root 1862 Aug 31  2018 authconfig_downloaded.pem

/etc/openldap/ldap.conf contains (only TLS-related setting):

TLS_CACERTDIR /etc/openldap/cacerts

It looks like in some circumstances (but crucially, not every time), the TLSMC layer is creating a temporary OpenLDAP directory with a PEM certificate from ca-bundle.crt(?), and then our self-signed cert fails to validate. In the PHP log with full LDAP debugging enabled, we see:

[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLSMC: MozNSS compatibility interception begins."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_intercept_initialization: INFO: entry options follow:"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_intercept_initialization: INFO: cacertdir = `/etc/openldap/cacerts'"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_intercept_initialization: INFO: certfile = `(null)'"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_intercept_initialization: INFO: keyfile = `(null)'"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_convert: INFO: trying to open NSS DB with CACertDir = `/etc/openldap/cacerts'."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_open_nssdb: INFO: trying to initialize moznss using security dir `/etc/openldap/cacerts` prefix ``."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_open_nssdb: INFO: initialized MozNSS context."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_convert: INFO: trying with PEM dir = `/tmp/openldap-tlsmc-cacerts--C492C3C1B21AD11A26B8D9EEB15B4935251A3B6C1184C6665014ACE8DA360E40'."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_convert: WARN: will try to create PEM dir."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_prepare_dir: INFO: preparing PEM directory `/tmp/openldap-tlsmc-cacerts--C492C3C1B21AD11A26B8D9EEB15B4935251A3B6C1184C6665014ACE8DA360E40'."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_prepare_dir: INFO: creating a subdirectory `cacerts'."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_prepare_dir: INFO: successfully created PEM directory structure."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_extract_cacerts: INFO: found cert nick=`PEM Token #0:ca-bundle.crt - 14', a trusted CA."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_extract_cacerts: INFO: extracting cert nick=`PEM Token #0:ca-bundle.crt - 14' to file `/tmp/openldap-tlsmc-cacerts--C492C3C1B21AD11A26B8D9EEB15B4935251A3B6C1184C6665014ACE8DA360E40/cacerts/cert0.pem'."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_cert_create_hash_symlink: INFO: the cert is now symlinked to /tmp/openldap-tlsmc-cacerts--C492C3C1B21AD11A26B8D9EEB15B4935251A3B6C1184C6665014ACE8DA360E40/cacerts/653b494a.0."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_extract_cert_key_pair: WARN: supplied nickname is empty (NULL)."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_convert: WARN: extracted cert file is not present."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_convert: WARN: extracted key file is not present."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_intercept_initialization: INFO: altered options follow:"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_intercept_initialization: INFO: cacertdir = `/tmp/openldap-tlsmc-cacerts--C492C3C1B21AD11A26B8D9EEB15B4935251A3B6C1184C6665014ACE8DA360E40/cacerts'"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_intercept_initialization: INFO: certfile = `(null)'"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_intercept_initialization: INFO: keyfile = `(null)'"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "tlsmc_intercept_initialization: INFO: successfully intercepted TLS initialization. Continuing with OpenSSL only."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLSMC: MozNSS compatibility interception ends."
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLS trace: SSL_connect:before/connect initialization"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLS trace: SSL_connect:SSLv2/v3 write client hello A"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLS trace: SSL_connect:error in SSLv2/v3 read server hello A"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "ldap_int_tls_start: ldap_int_tls_connect needs read"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "ldap_int_poll: fd: 14 tm: 5"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "ldap_is_sock_ready: 14"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "ldap_ndelay_off: 14"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLS trace: SSL_connect:SSLv3 read server hello A"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLS certificate verification: depth: 1, err: 19, subject: /O=uk-ac-jccs/OU=Organizational CA, issuer: /O=uk-ac-jccs/OU=Organizational CA"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLS certificate verification: Error, self signed certificate in certificate chain"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLS trace: SSL3 alert write:fatal:unknown CA"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLS trace: SSL_connect:error in error"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLS trace: SSL_connect:error in error"
[07-Oct-2019 14:24:57] WARNING: [pool www] child 28815 said into stderr: "TLS: can't connect: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (self signed certificate in certificate chain)."

The CA certificate extracted into the temporary openldap dir is:

    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root
        Validity
            Not Before: May 12 18:46:00 2000 GMT
            Not After : May 12 23:59:00 2025 GMT
        Subject: C=IE, O=Baltimore, OU=CyberTrust, CN=Baltimore CyberTrust Root

README from that folder (as you can see, we have nothing configured in NSS DB format):

PARAMETERS:
nssdb_dir: /etc/openldap/cacerts
nssdb_prefix: 
ld_cacertdir: /etc/openldap/cacerts
ld_cert: (null)
ld_key: (null)
euid: 997

FILES:
cert8.db: (null)
cert9.db: (null)
key3.db: (null)
key4.db: (null)
secmod.db: (null)

Once this occurs, it appears that further PHP connections to that worker also fail due to validation errors, so eventually no LDAP connections succeed.

For comparison, a working connection looks like:

[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "TLSMC: MozNSS compatibility interception begins."
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_intercept_initialization: INFO: entry options follow:"
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_intercept_initialization: INFO: cacertdir = `/etc/openldap/cacerts'"
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_intercept_initialization: INFO: certfile = `(null)'"
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_intercept_initialization: INFO: keyfile = `(null)'"
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_convert: INFO: trying to open NSS DB with CACertDir = `/etc/openldap/cacerts'."
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_open_nssdb: INFO: trying to initialize moznss using security dir `/etc/openldap/cacerts` prefix ``."
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_open_nssdb: WARN: could not initialize MozNSS context - error -8015."
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_convert: INFO: cannot open the NSS DB, expecting PEM configuration is present."
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_intercept_initialization: INFO: altered options follow:"
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_intercept_initialization: INFO: cacertdir = `/etc/openldap/cacerts'"
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_intercept_initialization: INFO: certfile = `(null)'"
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_intercept_initialization: INFO: keyfile = `(null)'"
[07-Oct-2019 14:24:48] WARNING: [pool www] child 28813 said into stderr: "tlsmc_intercept_initialization: INFO: successfully intercepted TLS initialization. Continuing with OpenSSL only."

Looks like a bug in TLSMC, but the intermittent nature of it (we don't see it on all systems, at all times or with other LDAP-enabled components such as sssd) makes it difficult to track down. It looks like it must occur during the call to NSS_InitContext() at line 808 in the openldap-tlsmc.patch code - should fail but for some reason succeeds in a particular context.

Workaround: override the CACERTDIR setting in your application and explicitly set it to /etc/openldap/cacerts/ or add TLS_MOZNSS_COMPATIBILITY off to /etc/openldap/ldap.conf (providing you haven't got any required certificates configured in NSS - convert them to OpenSSL PEM format if so).

Reference: https://fedoraproject.org/wiki/OpenLDAP-and-MozNSS-Compatibility-Layer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment