Leo SSL hacking

leo_ssl_hacking.md

Notes on trying to get Leo -> Apache -> Jupyter working on localhost with 2-way SSL authentication. For this test:

• Jupyter is listening on port 8000 over HTTP
• Apache is listening on port 443 over HTTPS
• Leo is listening on port 8080 over HTTP

Start Jupyter server

cd /path/to/leonardo
./jupyter-docker/run-jupyter-local.sh start

openssl commands

All ran in /Users/rtitle/ssl/. Note: these are copied from various places, I'm not sure they're all correct.

• Generate the root CA. This creates a rootCA.key and a rootCA.pem.

openssl genrsa -out rootCA.key 2048 -des3
openssl req -x509 -new -nodes -key rootCA.key -days 1024 -out rootCA.pem -sha256 -subj "/C=US/ST=Massachusetts/L=Cambridge/O=Broad Institute/OU=DSP/CN=leonardo"

• Generate the jupyter server cert, signed by the CA. This generates a jupyter-server.key and a jupyter.server.crt.
• Note the common name is 'jupyter.firecloud.org'.

openssl genrsa -out jupyter-server.key 2048
openssl req -new -key jupyter-server.key -out jupyter-server.csr -subj "/C=US/ST=Massachusetts/L=Cambridge/O=Broad Institute/OU=DSP/CN=*.jupyter-dev.firecloud.org" -sha256
openssl x509 -req -in jupyter-server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out jupyter-server.crt -days 1500

• Generate a Leo client cert, signed by the CA. This generates a leo-client.key and a leo-client.crt.
• Note the common name is 'leonardo'. This doesn't matter so much, but should be different than the server cert.

openssl genrsa -out leo-client.key 2048
openssl req -new -key leo-client.key -out leo-client.csr -subj "/C=US/ST=Massachusetts/L=Cambridge/O=Broad Institute/OU=DSP/CN=leonardo-client" -sha256
openssl x509 -req -in leo-client.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out leo-client.crt -days 1500

• Generate a PKCS12 file. This is the file that we need to install on the client to communicate with the server.

openssl pkcs12 -export -inkey leo-client.key -in leo-client.crt -out leo-client.p12

• Base64-encode the PKCS12 file so it can be stored in vault.

openssl base64 -in leo-client.p12 -out leo-client.p12.b64

Configure Apache

Using this Apache config:

Listen 443

<VirtualHost *:443>
        ServerName "jupyter.firecloud.org"

        SSLEngine on
        SSLCertificateFile /Users/rtitle/ssl/jupyter-server.crt
        SSLCertificateKeyFile /Users/rtitle/ssl/jupyter-server.key

        SSLVerifyClient require
        SSLVerifyDepth 10
        SSLCACertificateFile /Users/rtitle/ssl/rootCA.pem

        ProxyPreserveHost On
        ProxyRequests off

        ProxyPass /api/notebooks/dsp-leo-test/test/api/kernels/ ws://127.0.0.1:8000/api/notebooks/dsp-leo-test/test/api/kernels/
        ProxyPassReverse /api/notebooks/dsp-leo-test/test/api/kernels/ http://127.0.0.1:8000/api/notebooks/dsp-leo-test/test/api/kernels/

        ProxyPass / http://127.0.0.1:8000/
        ProxyPassReverse / http://127.0.0.1:8000/
</VirtualHost>

Testing

It works!

Curl:

• curl -v -k "https://localhost/api/notebooks/dsp-leo-test/test"
  • Fails as expected with:
    • curl: (35) SSL peer handshake failed, the server most likely requires a client certificate to connect
• curl -v --cert ./leo-client.p12:pass --cacert ./rootCA.pem "https://localhost/api/notebooks/dsp-leo-test/test"
  • Fails as expected with:
    • SSL: certificate verification failed (result: 5)
  • This is because the hostname in the request (localhost) does not match the common name of the server cert (jupyter.firecloud.org).
  • Note this is technically just a warning; the check can be disabled by adding the --insecure flag.
• curl -v --cert ./leo-client.p12:pass --cacert ./rootCA.pem --resolve "jupyter.firecloud.org:443:127.0.0.1" "https://jupyter.firecloud.org/api/notebooks/dsp-leo-test/test"
  • Returns a 302 as expected

Browser:

• Import leo-client.p12 using the Keychain Access program (on Mac) and browse to https://localhost/api/notebooks/dsp-leo-test/test. Select the 'leonardo' client cert and the Apache proxy works correctly. I can browse Jupyter, create notebooks, and run python code!

Configure Leo

TODO! Will probably need this akka-http reference: http://doc.akka.io/docs/akka-http/10.0.9/scala/http/client-side/client-https-support.html

You need "--cacert" with the pub CA cert in your curl (not just the --cert) to do the validation. Because the CA Cert that curl (or your machine) comes with doesn't trust self-signed certs.

Yes! This works:

curl -v --cert ./leo-client.p12:pass --cacert ./rootCA.pem  --resolve "jupyter.firecloud.org:443:127.0.0.1" "https://jupyter.firecloud.org/api/notebooks/dsp-leo-test/test"

Updated the gist.

Vault write commands:

vault write secret/dsde/firecloud/qa/leonardo/jupyter jupyter-server.crt=@jupyter-server.crt
vault write secret/dsde/firecloud/qa/leonardo/jupyter jupyter-server.key=@jupyter-server.key
vault write secret/dsde/firecloud/qa/leonardo/jupyter leo-client.p12.b64=@leo-client.p12.b64
vault write secret/dsde/firecloud/qa/leonardo/jupyter rootCA.key=@rootCA.key
vault write secret/dsde/firecloud/qa/leonardo/jupyter rootCA.pem=@rootCA.pem

Don't use for edits - this overwrites the key each time.