apache_solr-lets_encrypt_ssl.txt

===============================================================================
APPLICATION OF LET'S ENCRYPT SSL CERTIFICATE TO APACHE SOLR
===============================================================================

This solution / summary is in two parts.

  1. Failed installation of LE certificate via OpenSSL approach (SO #41592427).

  2. Successful installation via blog entry.


==============================================================================
[failed installation] APACHE SOLR WITH LET'S ENCRYPT SSL CERTIFICATE
==============================================================================
2022-08-30

* Based on:

  [2017-01] Letsencypt Solr SSL JVM
  https://stackoverflow.com/questions/41592427/letsencypt-solr-ssl-jvm/41594395#41594395

* [2022-08-30 19:02:42 -0700 (PST)] This APPEARED to go without issue; however,
   the Opera and Brave browsers again complained with a "net::ERR_CERT_AUTHORITY_INVALID" error (Firefox was OK).

* Although more convoluted, I reverted back to the procedures based on the following blog entry, which gave
  a fully-functioning Let's Encrypt certificate applied to Solr, functioning in all browsers { Firefox | Opera | Brave }.

    [2019-03] Building an Ubuntu VM in Azure running Solr, with a trusted LetsEncrypt certificate - Part 3
    https://robearlam.com/blog/building-an-ubuntu-vm-in-azure-running-solr-with-a-trusted-letsencrypt-certificate-part-3

-------------------------------------------------------------------------------

[user]$ dpl

  2022-08-30 17:36:42 -0700 (PST)
  /home/me/apps/solr-8.11.1/server/etc

  ...
  -rw-r--r-- 1 root root 6.0K Aug 30 16:14 solrKeystore.p12
  -rw-r--r-- 1 solr solr 1.4K Aug 18 13:45 solr-ssl-cert
  -rw-r--r-- 1 solr solr 2.8K Aug 18 13:45 solr-ssl.keystore.p12
  -rw-r--r-- 1 solr solr 2.8K Aug 12 19:48 solr-ssl.keystore.p12-2022.08.12
  ...

[user]$ sudo mv solrKeystore.p12 solrKeystore.p12.2022-08-30

[user]$ ls -l

  ...
  -rw-r--r-- 1 root root  6127 Aug 30 16:14 solrKeystore.p12.2022-08-30
  -rw-r--r-- 1 solr solr  1379 Aug 18 13:45 solr-ssl-cert
  -rw-r--r-- 1 solr solr  2797 Aug 18 13:45 solr-ssl.keystore.p12
  -rw-r--r-- 1 solr solr  2797 Aug 12 19:48 solr-ssl.keystore.p12-2022.08.12
  ...

[user]$ sudo ls -l /etc/letsencrypt/live/www.example.com

  total 44
  lrwxrwxrwx 1 root root    40 Jul 27 13:10 cert.pem -> ...
  lrwxrwxrwx 1 root root    41 Jul 27 13:10 chain.pem -> ...
  -rw-r--r-- 1 root root 12938 Aug 30 14:06 fullcert.pem
  lrwxrwxrwx 1 root root    45 Jul 27 13:10 fullchain.pem -> ...
  -rw-r--r-- 1 root root  9877 Aug 30 14:07 fullchain.pkcs12
  lrwxrwxrwx 1 root root    43 Jul 27 13:10 privkey.pem -> ...
  -rw-r--r-- 1 root root   692 Jul 26 14:25 README

[user]$ sudo openssl pkcs12  -export  -in /etc/letsencrypt/live/www.example.com/fullchain.pem  -inkey /etc/letsencrypt/live/www.example.com/privkey.pem  -out /home/me/apps/solr-8.11.1/server/etc/solr-ssl-le.pkcs12

  Enter Export Password:
  Verifying - Enter Export Password:

[user]$ dpl

  2022-08-30 17:51:53 -0700 (PST)
  /home/me/apps/solr-8.11.1/server/etc

  ...
  -rw-r--r-- 1 root root 6.0K Aug 30 16:14 solrKeystore.p12.2022-08-30
  -rw-r--r-- 1 solr solr 1.4K Aug 18 13:45 solr-ssl-cert
  -rw-r--r-- 1 solr solr 2.8K Aug 18 13:45 solr-ssl.keystore.p12
  -rw-r--r-- 1 solr solr 2.8K Aug 12 19:48 solr-ssl.keystore.p12-2022.08.12
  -rw-r--r-- 1 root root 5.6K Aug 30 17:51 solr-ssl-le.pkcs12                   ## << new file
  ...

[user]$ sudo keytool -v -importkeystore  -srckeystore solr-ssl-le.pkcs12  -destkeystore  solr-ssl-le.p12  -deststoretype PKCS12

  Importing keystore solr-ssl-le.pkcs12 to solr-ssl-le.p12...

  Enter destination keystore password:
  Re-enter new password:
  Enter source keystore password:
  Entry for alias 1 successfully imported.

  Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
  [Storing solr-ssl-le.p12]

[user]$ pl

  /home/me/apps/solr-8.11.1/server/etc

  ...
  -rw-r--r-- 1 root root 6.0K Aug 30 16:14 solrKeystore.p12.2022-08-30
  -rw-r--r-- 1 solr solr 1.4K Aug 18 13:45 solr-ssl-cert
  -rw-r--r-- 1 solr solr 2.8K Aug 18 13:45 solr-ssl.keystore.p12
  -rw-r--r-- 1 solr solr 2.8K Aug 12 19:48 solr-ssl.keystore.p12-2022.08.12
  -rw-r--r-- 1 root root 6.0K Aug 30 17:55 solr-ssl-le.p12                      ## << new keystore
  -rw-r--r-- 1 root root 5.6K Aug 30 17:51 solr-ssl-le.pkcs12
  ...

[user]$ sudo systemctl restart httpd

[user]$ sudo systemctl restart sshd

[user]$ su - solr -c "/home/me/apps/solr-8.11.1/bin/solr restart ; sleep  3; /home/me/apps/solr-8.11.1/bin/solr status"

  Password:

  Sending stop command to Solr running on port 8983 ... waiting up to 30 seconds to allow Jetty process 8313 to stop gracefully.
  Waiting up to 30 seconds to see Solr running on port 8983 [-]
  Started Solr server on port 8983 (pid=8799). Happy searching!

  Found 1 Solr nodes:

  Solr process 8799 running on port 8983
  INFO  - 2022-08-30 17:08:35.494; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
  INFO  - 2022-08-30 17:08:35.498; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
  {
    "solr_home":"/home/me/apps/solr-8.11.1/server/solr",
    "version":"8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55",
    "startTime":"2022-08-31T01:08:10.119Z",
    "uptime":"0 days, 0 hours, 0 minutes, 28 seconds",
    "memory":"133.5 MB (%26.1) of 512 MB"}

[user]$ sudo keytool -list -cacerts | grep solr-ssl

  Enter keystore password:                  ## your password: keytool default p/w is: changeit
  solr-ssl, Aug 29, 2022, trustedCertEntry,
  solr-ssl-le, Aug 30, 2022, trustedCertEntry,

[user]$ dpl

  2022-08-30 18:18:24 -0700 (PST)
  /home/me/apps/solr-8.11.1/server/etc

  ...
  -rw-r--r-- 1 root root 6.0K Aug 30 16:14 solrKeystore.p12.2022-08-30
  -rw-r--r-- 1 solr solr 1.4K Aug 18 13:45 solr-ssl-cert
  -rw-r--r-- 1 solr solr 2.8K Aug 18 13:45 solr-ssl.keystore.p12
  -rw-r--r-- 1 solr solr 2.8K Aug 12 19:48 solr-ssl.keystore.p12-2022.08.12
  -rw-r--r-- 1 solr solr 6.0K Aug 30 17:55 solr-ssl-le.p12
  -rw-r--r-- 1 solr solr 5.6K Aug 30 17:51 solr-ssl-le.pkcs12
  ...

[user]$ sudo keytool -exportcert -rfc -keystore solr-ssl-le.p12  -alias solr-ssl  -file solr-ssl-le-cert

  Enter keystore password:
  keytool error: java.lang.Exception: Alias <solr-ssl> does not exist

[user]$ keytool -list -v -keystore "solr-ssl-le.p12" -storepass <your_password>  | grep -i alias

  Alias name: 1

[user]$ sudo keytool -exportcert -rfc -keystore solr-ssl-le.p12  -alias 1  -file solr-ssl-le-cert

  Enter keystore password:
  Certificate stored in file <solr-ssl-le-cert>

[user]$ cat solr-ssl-le-cert

  -----BEGIN CERTIFICATE-----
    [ ... snip ... ]
  -----END CERTIFICATE-----

[user]$ sudo keytool -import -trustcacerts -cacerts -storepass <password> -noprompt -alias solr-ssl-le -file /home/me/apps/solr-8.11.1/server/etc/solr-ssl-le-cert

  keytool error: java.lang.Exception: Certificate not imported, alias <solr-ssl-le> already exists


[user]$ sudo keytool -delete -trustcacerts -cacerts -storepass <password> -noprompt -alias solr-ssl-le -file /home/me/apps/solr-8.11.1/server/etc/solr-ssl-le-cert

[user]$ sudo keytool -import -trustcacerts -cacerts -storepass <password> -noprompt -alias solr-ssl-le -file /home/me/apps/solr-8.11.1/server/etc/solr-ssl-le-cert

  Certificate was added to keystore

[user]$ sudo keytool -list -cacerts | grep solr-ssl

  Enter keystore password:                  ## your password: keytool default p/w is: changeit
  solr-ssl, Aug 29, 2022, trustedCertEntry,
  solr-ssl-le, Aug 30, 2022, trustedCertEntry,

[user]$

==============================================================================
[end] [failed installation] APACHE SOLR WITH LET'S ENCRYPT SSL CERTIFICATE
==============================================================================


===============================================================================
[re-installation] SOLR WITH LET'S ENCRYPT SIGNED SSL CERTIFICATE
===============================================================================
2022-08-30 | updated 2022-08-31

========================================
NOTES
========================================

1.  The following is largely based on the following tutorial.

      [2019-03] Building an Ubuntu VM in Azure running Solr, with a trusted LetsEncrypt certificate - Part 3
      https://robearlam.com/blog/building-an-ubuntu-vm-in-azure-running-solr-with-a-trusted-letsencrypt-certificate-part-3

2.  A "Let's Encrypt" discussion group member pointed out this StackOverflow
    reference

      [2017-01] Letsencypt Solr SSL JVM
      https://stackoverflow.com/questions/41592427/letsencypt-solr-ssl-jvm/41594395#41594395

    I implemented that approach,

      "[failed installation] APACHE SOLR WITH LET'S ENCRYPT SSL CERTIFICATE"

    above. While it seemed to go well, Opera and Brave [web browsers] threw a

      "net::ERR_CERT_AUTHORITY_INVALID" error (Firefox was OK)

3. For privacy, following I replaced my domain name with "example", and my home directory with "me".

4. "d", "dp", "dpl" ... are my ~/.bashrc aliases for date (d), pwd (p), ls -l (l), and compound variations thereof.

========================================
[end] NOTES
========================================


===============================================================================
CONVERTING THE Let's Encrypt CERTIFICATE TO THE CORRECT FORMAT FOR APACHE SOLR
===============================================================================
2022-08-31

-------------------------------------------------------------------------------

* [2019-03] Building an Ubuntu VM in Azure running Solr, with a trusted LetsEncrypt certificate - Part 3\
  https://robearlam.com/blog/building-an-ubuntu-vm-in-azure-running-solr-with-a-trusted-letsencrypt-certificate-part-3

    ...
    echo "cat /etc/letsencrypt/live/XXXXX.australiaeast.cloudapp.azure.com/*.pem > fullcert.pem" | \
    sudo sh sudo openssl pkcs12 -export -out fullchain.pkcs12 -in fullcert.pem
    ...

    Now what we're doing here is firstly concatenating the certificate chain generated by LetsEncrypt into a single
    PEM format certificate, we're then taking that newly concatenated certificate and using openssl to generate a new 
    certificate in the PKCS12 format that we need."
    ...

 ** CRITIQUE:
 
    [2022-08-30] Apply Let’s Encrypt certificate to Apache Solr
    https://community.letsencrypt.org/t/apply-lets-encrypt-certificate-to-apache-solr/183686/9

      ...
      Yeah, that does not make sense.
      
      In the /etc/letsencrypt/live/ folder are 4 files.
      
      One is named fullchain.pem and already is the full concatenated chain.
      
      By doing that cat you are also placing your privkey.pem and two extra,
      unneeded cert fragments in that file.
      
      It's possible you are now sending out your private key to every client.
      
      That's a huge no-no and if true is cause for your cert to be revoked
      (I don't know solr well enough to say for sure).
      
      In short, that blog is a bad example. ...

-------------------------------------------------------------------------------

[user]$ sudo -s

[root]# date
  Wed Aug 31 09:03:00 PDT 2022

[root]# cd /etc/letsencrypt/live/www.example.com

[root]# ls -l

  ...
  lrwxrwxrwx 1 root root    40 Jul 27 13:10 cert.pem -> ...
  lrwxrwxrwx 1 root root    41 Jul 27 13:10 chain.pem -> ...
  lrwxrwxrwx 1 root root    45 Jul 27 13:10 fullchain.pem -> ...
  lrwxrwxrwx 1 root root    43 Jul 27 13:10 privkey.pem -> ...
  ...

[root]# openssl pkcs12 -export -out fullchain.pkcs12 -in fullchain.pem

  unable to load private key
  140075780192144:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:707:Expecting: ANY PRIVATE KEY

[root]#

## https://stackoverflow.com/questions/31630544/cant-get-private-key-with-openssl-no-start-linepem-lib-c703expecting-any-p

[root]# openssl pkcs12 -export -out fullchain.pkcs12 -in fullchain.pem -inkey privkey.pem

  Enter Export Password:
  Verifying - Enter Export Password:

[root]# pwd; ls -l

  /etc/letsencrypt/live/www.example.com

  ...
  lrwxrwxrwx 1 root root    40 Jul 27 13:10 cert.pem -> ...
  lrwxrwxrwx 1 root root    41 Jul 27 13:10 chain.pem -> ...
  lrwxrwxrwx 1 root root    45 Jul 27 13:10 fullchain.pem -> ...
  -rw-r--r-- 1 root root  5717 Aug 31 09:08 fullchain.pkcs12                    ## << created
  lrwxrwxrwx 1 root root    43 Jul 27 13:10 privkey.pem -> ...
  ...

[root]#

[root]# keytool -v -importkeystore -srckeystore fullchain.pkcs12 -destkeystore  solrKeystore.p12 -deststoretype PKCS12

  Importing keystore fullchain.pkcs12 to solrKeystore.p12...

  Enter destination keystore password:
  Re-enter new password:
  Enter source keystore password:

  Entry for alias 1 successfully imported.
  Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
  [Storing solrKeystore.p12]

[root]# pwd; ls -l

  /etc/letsencrypt/live/www.example.com

  ...
  lrwxrwxrwx 1 root root    40 Jul 27 13:10 cert.pem -> ...
  lrwxrwxrwx 1 root root    41 Jul 27 13:10 chain.pem -> ...
  lrwxrwxrwx 1 root root    45 Jul 27 13:10 fullchain.pem -> ...
  -rw-r--r-- 1 root root  5717 Aug 31 09:08 fullchain.pkcs12
  lrwxrwxrwx 1 root root    43 Jul 27 13:10 privkey.pem -> ...
  -rw-r--r-- 1 root root  6127 Aug 31 09:12 solrKeystore.p12                    ## << created
  ...

[root]# mv /home/me/apps/solr-8.11.1/server/etc/solrKeystore.p12 /home/me/apps/solr-8.11.1/server/etc/solrKeystore.p12-2022-08-30

[root]# ## ls -l /home/me/apps/solr-8.11.1/server/etc/

[root]# cp /etc/letsencrypt/live/www.example.com/solrKeystore.p12 /home/me/apps/solr-8.11.1/server/etc/solrKeystore.p12

[root]# cp /home/me/apps/solr-8.11.1/server/etc/solrKeystore.p12 /home/me/apps/solr-8.11.1/server/etc/solrKeystore.p12-2022-08-31

[root]# ls -l /home/me/apps/solr-8.11.1/server/etc/

  ...
  -rw-r--r-- 1 root root  6127 Aug 31 09:16 solrKeystore.p12                    ## << created
  ...

[root]# exit
  exit

[user]$

----------------------------------------

[user]$ soho

[user]$ cd server/etc/

[user]$ dpl

  2022-08-31 09:21:22 -0700 (PST)
  /home/me/apps/solr-8.11.1/server/etc

  ...
  -rw-r--r-- 1 root root 6.0K Aug 31 09:16 solrKeystore.p12                     ## << wrong owner
  ...

[user]$ sudo chown solr:solr solrKeystore.p12

[user]$ ls -l solrKeystore.p12

  -rw-r--r-- 1 solr solr 6127 Aug 31 09:16 solrKeystore.p12


[user]$ sudo cat /etc/default/solr.in.sh | egrep -w  \
      '^SOLR_SSL_KEY_STORE|^SOLR_SSL_KEY_STORE_TYPE|^SOLR_SSL_TRUST_STORE|^SOLR_SSL_TRUST_STORE_TYPE'

  SOLR_SSL_KEY_STORE=/home/me/apps/solr-8.11.1/server/etc/solrKeystore.p12
  SOLR_SSL_KEY_STORE_TYPE=PKCS12
  SOLR_SSL_TRUST_STORE=/home/me/apps/solr-8.11.1/server/etc/solrKeystore.p12
  SOLR_SSL_TRUST_STORE_TYPE=PKCS12

[user]$ su - solr -c "/home/me/apps/solr-8.11.1/bin/solr restart ; sleep  3; /home/me/apps/solr-8.11.1/bin/solr status"

  Password:

  Sending stop command to Solr running on port 8983 ... waiting up to 30 seconds to allow Jetty process 11589 to stop gracefully.
  Waiting up to 30 seconds to see Solr running on port 8983 [-]
  Started Solr server on port 8983 (pid=14507). Happy searching!


  Found 1 Solr nodes:

  Solr process 14507 running on port 8983
  INFO  - 2022-08-31 08:26:29.026; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
  INFO  - 2022-08-31 08:26:29.030; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
  {
    "solr_home":"/home/me/apps/solr-8.11.1/server/solr",
    "version":"8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 - janhoy - 2021-12-14 13:50:55",
    "startTime":"2022-08-31T16:26:14.202Z",
    "uptime":"0 days, 0 hours, 0 minutes, 17 seconds",
    "memory":"137 MB (%26.8) of 512 MB"}

[user]$ sudo systemctl restart httpd
[user]$ sudo systemctl restart sshd

[user]$


## SUCCESS!   Let's Encrypt-secured Solr operational in all three browsers:
##            { Firefox | Brave | Opera }!   🥳

===============================================================================
[end] CONVERTING THE Let's Encrypt CERTIFICATE TO THE CORRECT FORMAT FOR APACHE SOLR
===============================================================================

===============================================================================
[end] [re-installation] SOLR WITH LET'S ENCRYPT SIGNED SSL CERTIFICATE
===============================================================================

===============================================================================
[end] APPLICATION OF LET'S ENCRYPT SSL CERTIFICATE TO APACHE SOLR
===============================================================================