Skip to content

Instantly share code, notes, and snippets.

@carlossg

carlossg/readme.md

Created September 17, 2015 10:36
Show Gist options
  • Save carlossg/64c9f8050d637f51c77c to your computer and use it in GitHub Desktop.
Save carlossg/64c9f8050d637f51c77c to your computer and use it in GitHub Desktop.
Download ZIP
Mesos SSL problems
Raw
readme.md

Configuration

In /etc/default/mesos

export SSL_ENABLED=true
export SSL_SUPPORT_DOWNGRADE=false
export SSL_REQUIRE_CERT=true
export SSL_CERT_FILE=/etc/mesos/tiger-jdoe-controller-1.tiger.acme.net.crt
export SSL_KEY_FILE=/etc/mesos/tiger-jdoe-controller-1.tiger.acme.net.key
export SSL_CA_FILE=/etc/mesos/rootCA.pem

Problem #1 slaves can't connect to master over SSL

In master logs:

W0917 10:10:55.881906 23468 openssl.cpp:237] Error with certificate at depth: 0
Issuer: /C=US/ST=CA/O=acme/OU=Tiger
Subject: /C=US/O=acme/OU=Tiger/CN=tiger-jdoe-worker-2.tiger.acme.net
Error (26): unsupported certificate purpose

but slave cert has what I believe are correct certificate purposes

# openssl x509 -in /etc/mesos/tiger.crt -noout -purpose
Certificate purposes:
SSL client : Yes
SSL client CA : No
SSL server : Yes
SSL server CA : No
Netscape SSL server : Yes
Netscape SSL server CA : No
S/MIME signing : No
S/MIME signing CA : No
S/MIME encryption : No
S/MIME encryption CA : No
CRL signing : No
CRL signing CA : No
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : No
Time Stamp signing : No
Time Stamp signing CA : No

Problem #2 can't connect to master if cert validation is enabled

SSL_REQUIRE_CERT implies SSL_VALIDATE_CERT, and failed to connect with curl even enabling all protocols

With this config in the master

SSL_CA_FILE=/etc/mesos/rootCA.pem
SSL_CERT_FILE=/etc/mesos/tiger.crt
SSL_ENABLED=true
SSL_ENABLE_SSL_V3=true
SSL_ENABLE_TLS_V1_0=true
SSL_ENABLE_TLS_V1_1=true
SSL_KEY_FILE=/etc/mesos/tiger.key
SSL_REQUIRE_CERT=true
SSL_SUPPORT_DOWNGRADE=false

hostname manually set to tiger-jdoe-controller-1.tiger.acme.net to ensure SSL hostname validation works

Master running with

/usr/sbin/mesos-master --zk=zk://zk:xxxxxx@10.16.2.1:2181/mesos --port=5050 --log_dir=/var/log/mesos --authenticate=true --authenticate_slaves=true --credentials=/etc/mesos/credentials --hostname=tiger-jdoe-controller-1.tiger.acme.net --quorum=1 --work_dir=/var/lib/mesos

Different curl versions fail to connect to master when providing a client certificate

# openssl x509 -in /home/ubuntu/tiger-client.cert.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4101 (0x1005)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=CA, O=acme, OU=Tiger
        Validity
            Not Before: Sep 17 08:31:25 2015 GMT
            Not After : Sep 14 08:31:25 2025 GMT
        Subject: C=US, O=acme, OU=Tiger, CN=tiger-client.tiger.acme.net
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus: ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Client, S/MIME
            Netscape Comment:
                OpenSSL Generated Client Certificate
            X509v3 Subject Key Identifier:
                13:B2:3B:BE:31:0B:77:83:F0:5E:E4:A0:5A:43:1A:BA:17:FB:11:06
            X509v3 Authority Key Identifier:
                keyid:56:75:92:EA:2F:89:8A:06:A7:C7:68:AB:7D:D4:C6:EB:10:AC:F5:7E

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, E-mail Protection
    Signature Algorithm: sha256WithRSAEncryption
    ...

Using ubuntu 14.04.3 curl 7.35.0

# curl --version
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

# curl -v --cacert /etc/mesos/rootCA.pem --key /home/ubuntu/tiger-client.key.pem --cert /home/ubuntu/tiger-client.cert.pem https://tiger-jdoe-controller-1.tiger.acme.net:5050/master/state.json
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to tiger-jdoe-controller-1.tiger.acme.net (127.0.0.1) port 5050 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/mesos/rootCA.pem
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS handshake, CERT verify (15):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES128-SHA
* Server certificate:
*    subject: C=US; O=acme; OU=Tiger; CN=tiger-jdoe-controller-1.tiger.acme.net
*    start date: 2015-09-16 18:28:30 GMT
*    expire date: 2025-09-13 18:28:30 GMT
*    common name: tiger-jdoe-controller-1.tiger.acme.net (matched)
*    issuer: C=US; ST=CA; O=acme; OU=Tiger
*    SSL certificate verify ok.
> GET /master/state.json HTTP/1.1
> User-Agent: curl/7.35.0
> Host: tiger-jdoe-controller-1.tiger.acme.net:5050
> Accept: */*
>
* Empty reply from server
* Connection #0 to host tiger-jdoe-controller-1.tiger.acme.net left intact
curl: (52) Empty reply from server

Using ubuntu:wily with curl installed

root@tiger-jdoe-controller-1:~# docker run --net=host --rm test curl --version
curl 7.43.0 (x86_64-pc-linux-gnu) libcurl/7.43.0 GnuTLS/3.3.15 zlib/1.2.8 libidn/1.28 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

root@tiger-jdoe-controller-1:~# docker run --net=host -v /etc/mesos/rootCA.pem:/etc/mesos/rootCA.pem -v /home/ubuntu/tiger-client.key.pem:/home/ubuntu/tiger-client.key.pem -v /home/ubuntu/tiger-client.cert.pem:/home/ubuntu/tiger-client.cert.pem --rm test curl -v --cacert /etc/mesos/rootCA.pem --key /home/ubuntu/tiger-client.key.pem --cert /home/ubuntu/tiger-client.cert.pem https://tiger-jdoe-controller-1.tiger.acme.net:5050/master/state.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1...
* Connected to tiger-jdoe-controller-1.tiger.acme.net (127.0.0.1) port 5050 (#0)
* found 1 certificates in /etc/mesos/rootCA.pem
* found 748 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / RSA_AES_128_CBC_SHA1
*    server certificate verification OK
*    server certificate status verification SKIPPED
*    common name: tiger-jdoe-controller-1.tiger.acme.net (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: C=US,O=acme,OU=Tiger,CN=tiger-jdoe-controller-1.tiger.acme.net
*    start date: Wed, 16 Sep 2015 18:28:30 GMT
*    expire date: Sat, 13 Sep 2025 18:28:30 GMT
*    issuer: C=US,ST=CA,O=acme,OU=Tiger
*    compression: NULL
* ALPN, server did not agree to a protocol
> GET /master/state.json HTTP/1.1
> Host: tiger-jdoe-controller-1.tiger.acme.net:5050
> User-Agent: curl/7.43.0
> Accept: */*
>
* GnuTLS recv error (-110): The TLS connection was non-properly terminated.
* Closing connection 0
curl: (56) GnuTLS recv error (-110): The TLS connection was non-properly terminated.

Using curl 7.42.1 from appropriate/curl

root@tiger-jdoe-controller-1:~# docker run --net=host --rm  appropriate/curl curl --version
curl 7.42.1 (x86_64-alpine-linux-musl) libcurl/7.42.1 OpenSSL/1.0.2d zlib/1.2.8 libssh2/1.5.0
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

root@tiger-jdoe-controller-1:~# docker run --net=host -v /etc/mesos/rootCA.pem:/etc/mesos/rootCA.pem -v /home/ubuntu/tiger-client.key.pem:/home/ubuntu/tiger-client.key.pem -v /home/ubuntu/tiger-client.cert.pem:/home/ubuntu/tiger-client.cert.pem --rm  appropriate/curl curl -v --cacert /etc/mesos/rootCA.pem --key /home/ubuntu/tiger-client.key.pem --cert /home/ubuntu/tiger-client.cert.pem https://tiger-jdoe-controller-1.tiger.acme.net:5050/master/state.json
*   Trying 127.0.0.1...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to tiger-jdoe-controller-1.tiger.acme.net (127.0.0.1) port 5050 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/mesos/rootCA.pem
  CApath: none
* TLSv1.2, TLS Unknown, Unknown (22):
} [5 bytes data]
* TLSv1.2, TLS handshake, Client hello (1):
} [512 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* SSLv2, Unknown (22):
{ [5 bytes data]
* TLSv1.2, TLS handshake, Server hello (2):
{ [54 bytes data]
* SSLv2, Unknown (22):
{ [5 bytes data]
* TLSv1.2, TLS handshake, CERT (11):
{ [2729 bytes data]
* SSLv2, Unknown (22):
{ [5 bytes data]
* TLSv1.2, TLS handshake, Request CERT (13):
{ [42 bytes data]
* TLSv1.2, TLS handshake, Server finished (14):
{ [4 bytes data]
* SSLv2, Unknown (22):
} [5 bytes data]
* TLSv1.2, TLS handshake, CERT (11):
} [2643 bytes data]
* SSLv2, Unknown (22):
} [5 bytes data]
* TLSv1.2, TLS handshake, Client key exchange (16):
} [262 bytes data]
* SSLv2, Unknown (22):
} [5 bytes data]
* TLSv1.2, TLS handshake, CERT verify (15):
} [264 bytes data]
* SSLv2, Unknown (20):
} [5 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
} [1 bytes data]
* SSLv2, Unknown (22):
} [5 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
} [16 bytes data]
* SSLv2, Unknown (20):
{ [5 bytes data]
* TLSv1.2, TLS change cipher, Client hello (1):
{ [1 bytes data]
* SSLv2, Unknown (22):
{ [5 bytes data]
* TLSv1.2, TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / AES128-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
*    subject: C=US; O=acme; OU=Tiger; CN=tiger-jdoe-controller-1.tiger.acme.net
*    start date: 2015-09-16 18:28:30 GMT
*    expire date: 2025-09-13 18:28:30 GMT
*    common name: tiger-jdoe-controller-1.tiger.acme.net (matched)
*    issuer: C=US; ST=CA; O=acme; OU=Tiger
*    SSL certificate verify ok.
* SSLv2, Unknown (23):
} [5 bytes data]
> GET /master/state.json HTTP/1.1
> Host: tiger-jdoe-controller-1.tiger.acme.net:5050
> User-Agent: curl/7.42.1
> Accept: */*
>
* Empty reply from server
* Connection #0 to host tiger-jdoe-controller-1.tiger.acme.net left intact
curl: (52) Empty reply from server
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment