Custom-JVM-Truststore.md

Testing/enabling custom JVM trustore

Set up a use case for testing before installing any bundles

Try reindexing from another cluster in the same env (updating the cluster id/password etc):

POST _reindex
{
  "source": {
    "remote": {
      "host": "https://4794ff5877d647abaf1f435a7090bbcc.192.168.44.10.ip.es.io:9243",
      "username": "elastic",
      "password": "aembsVCUrMAoXv2WRdqXEpZW"
    },
    "index": "logstash-0"
  },
  "dest": {
    "index": "logstash-0"
  }
}

This should return an error: "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"

Because the proxy uses self-signed certificate, and it gets rejected.

Prepare custom bundle:

• Pull server cert from the cluster (note, this is the proxy cert, not the one cluster uses for internal TLS):

   openssl s_client -connect https://4794ff5877d647abaf1f435a7090bbcc.192.168.44.10.ip.es.io:9243 -showcerts

(again, update the url as needed). This returns the entire chain (drop -showcerts from above if you just want leaf), so pick up the level you want (either the leaf, which is the first one, or elastic ce proxy root - you should not typically need master).

Copy/paste the appropriate cert into a file (say, proxy.pem)

• Find java's default truststore. It should be in $JAVA_HOME/jre/libs/security/cacerts, and copy it somewhere: cp $JAVA_HOME/jre/libs/security/cacerts .

• Now, run keytool to add new cert to the trust store: $JAVA_HOME/bin/keytool -keystore cacerts -storepass changeit -noprompt -importcert -file proxy.pem -alias cloud-proxy

• Create a bundle: zip cacerts.zip cacerts and upload it to a location where it is publicly readable (do not use google drive for this - the shareable links it creates are actually to a stupid html page, not direct download). If you use dropbox, replace www.dropbox.com in the link with dl.dropboxuercontent.com for the same reason.

Add bundle to the cluster (from which the indexing will be done)

In Advanced Editor, add user_bundles section to the plan (inside elasticsearch element):

   "user_bundles": [
      {
          "name": "cacerts",
          "url": "https://dl.dropboxusercontent.com/s/avt13juwgqcaavv/cacerts.zip",
          "elasticsearch_version": "6.8.0"
      }
    ]

(elasticsearch_version must be the same as the cluster version ... for some reason :/)

When plan completes, if you have access to the ES containers, verify that java is running with this options near the thrid quarter of it's options string: -Djavax.net.ssl.trustStore=/app/config/cacerts -Djavax.net.ssl.trustStorePassword=changeit (after -Des.cgroups.hierarchy.override=/ and before -Des.path.home=/elasticsearch)

And that there is a cacerts file in the /app/config directory (it would not start if there wasn't). You can inspect the contents of that file using keytool. E.g.: keytool -keystore /app/config/cacerts -noprompt -storepass changeit -exportcert -alias cloud-proxy -rfc (this should show the certificate that was added earlier).

Try reindexing from remote again

POST _reindex
{
  "source": {
    "remote": {
      "host": "https://4794ff5877d647abaf1f435a7090bbcc.192.168.44.10.ip.es.io:9243",
      "username": "elastic",
      "password": "aembsVCUrMAoXv2WRdqXEpZW"
    },
    "index": "logstash-0"
  },
  "dest": {
    "index": "logstash-0"
  }
}

On CE-AWS, I now get this: "Host name '4794ff5877d647abaf1f435a7090bbcc.192.168.44.10.ip.es.io' does not match the certificate subject provided by the peer (CN=elastic ce proxy 192.168.44.10)" That is because the CN in the cert I imported is wrong (should be *.ip.es.io or something like that). Apparently, our default proxy certs in ECE aren't really supposed to be used, we expect people to replace them with their own (the certs in ESS are actually correct, and not self-signed, so you could reindex from there without importing anything).

Generate a correct proxy cert

So, the fact that I am getting a different error now than I was before is an indication, that my custom bundle is working. However, for completeness, I can generate the correct proxy certificate from my ce-aws, set it on the proxy, and then add it to my bundle and reimport to see that reindex can actually be made to work this way.

So, I generate a new self-signed cert with correct CN. This needs to be done in a very specific way to keep both openssl and ece happy:

CONFIG="
[req]
distinguished_name=dn
[ dn ]
[ ext ]
basicConstraints=CA:TRUE,pathlen:0"

openssl genrsa -out CA.pem 2048 -nodes  # generate CA private key
openssl req -new -key CA.pem -out CA-cert.pem -subj '/CN=proxy-master' -config <(echo "$CONFIG") -extensions ext # CA cert MUST include basicConstraints

openssl genrsa -out proxy.pem 2048 -nodes # private key for the proxy
openssl req -new -key proxy.pem -out proxy.csr -nodes -subj '/CN=*.192.168.44.10.ip.es.io' # CSR for signing
openssl x509 -req -in proxy.csr -CA CA-cert.pem -CAkey CA.pem  -CAcreateserial -out proxy-cert.pem -days 1825 -sha256 # signed proxy cert

cat proxy.pem proxy-cert.pem CA-cert.pem > chain.pem  # cert chain to upload

Now, uploading chain.pem via admin console, will make proxy use the new cert with correct CN. But my original "general SSL error" is back, of course: I have to update my cacers with my new CA (or proxy cert), upload the bundle, and reallocate (not just restart!) the cluster.

And ... Voiila  Now _reindex works!